Graphic processor cluster resource scheduling method and equipment

Abstract

The invention discloses a graphics processor cluster resource scheduling method and device, and relates to the technical field of computers, and the method comprises the steps: obtaining the real-time load and static load of each graphics processor in a graphics processor cluster; for any graphic processor, comparing the real-time load with the static load of the graphic processor to obtain the load state of the graphic processor; and performing internal scheduling on the target model in the graphics processor cluster according to each load state. According to the method and the device, GPU resource allocation is changed into dynamic flow from fixed state, the dynamism of resource scheduling is remarkably improved, meanwhile, through a comparison mechanism of a real-time load and a static load, the system can adaptively recognize and respond to load changes at different time points and under different task pressures, and the system reliability is improved. The adaptability of the system to different operation scenes is enhanced, and finally the effects of optimizing the overall resource utilization rate of the cluster and avoiding local overload or idling are achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and device for scheduling resources in a graphics processor cluster. Background Technology

[0002] As deep learning models become increasingly complex and scalable, efficiently utilizing the computing resources of GPU (Graphics Processing Unit) clusters has become a key challenge. Existing solutions typically focus on the initial allocation and static scheduling of GPU resources in the cluster environment. That is, by analyzing the physical connection topology of GPUs, high-performance connection combinations are prioritized during container scheduling to optimize GPU data exchange efficiency. However, these methods mainly start from the hardware interconnect level, and resource allocation is essentially a static allocation method. Virtual GPU resources are fixed after being mounted to a specific container, which has significant shortcomings in terms of the dynamism and adaptability of GPU resource allocation. Summary of the Invention

[0003] The main purpose of this application is to provide a method and device for scheduling graphics processor cluster resources, which aims to solve the technical problem of insufficient dynamism and adaptability in current GPU resource allocation.

[0004] To achieve the above objectives, this application proposes a graphics processing unit (GPU) cluster resource scheduling method, which includes: Obtain the real-time and static loads of each graphics processor in the graphics processor cluster; For any graphics processor, the real-time load of the graphics processor is compared with the static load to obtain the load status of the graphics processor. The target model is internally scheduled within the graphics processor cluster according to the respective load states.

[0005] In one embodiment, the static load includes a first load threshold, and the load state includes a first load state and a second load state, wherein the first load state is obtained when the real-time load is less than the first load threshold, and the second load state is obtained when the real-time load is greater than or equal to the first load threshold; The step of internally scheduling the target model in the graphics processor cluster according to each of the load states includes: Iterate through each of the graphics processors described; The graphics processor with the first load state is marked as an available graphics processor, wherein the computing node corresponding to the available graphics processor is used to deploy new models; The target model corresponding to the graphics processor with the second load state is migrated to the computing node corresponding to the target graphics processor in each of the available graphics processors.

[0006] In one embodiment, the static load includes a first load threshold, and the load state includes a first load state and a second load state, wherein the first load state is obtained when the real-time load is less than the first load threshold, and the second load state is obtained when the real-time load is greater than or equal to the first load threshold; The step of marking the graphics processor with the first load state as a usable graphics processor includes: The graphics processors whose load state is the first load sub-state are marked as the first type of available graphics processors, wherein the computing nodes corresponding to the first type of available graphics processors are used to deploy new models; The graphics processors whose load state is the second load sub-state are marked as second type of available graphics processors, wherein the computing nodes corresponding to the second type of available graphics processors are used to receive new model deployments in the absence of the first type of available graphics processors.

[0007] In one embodiment, the step of migrating the target model corresponding to the graphics processor with the second load state to the computing node corresponding to the target graphics processor among the available graphics processors includes: Obtain the set of models deployed on the computing node corresponding to the graphics processor whose load state is the second load state; Determine the target model in the model set, and the target graphics processor corresponding to the target model in each of the available graphics processors; The target model is migrated to the computing node corresponding to the target graphics processor.

[0008] In one embodiment, the step of determining the target model in the model set, and the target graphics processor corresponding to the target model among the available graphics processors, includes: The target model among the models is determined based on the load occupancy of each model in the model set, wherein the target model is the model with the smallest load occupancy among the models. Based on the target model, query the models deployed in the computing nodes corresponding to each available graphics processor. If there is a computing node corresponding to an available graphics processor that has a model with the same model as the target model deployed thereon, then the available graphics processor corresponding to the computing node that has a model with the same model as the target model is taken as the target graphics processor.

[0009] In one embodiment, the step of migrating the target model to the computing node corresponding to the target graphics processor includes: The target model is deployed on the computing node corresponding to the target graphics processor, and the routing configuration corresponding to the target model is updated, wherein the routing configuration is used to indicate the graphics processor corresponding to the model call request; Monitor the processing status of model call requests in the computing nodes before migration of the target model; If the processing status is "processing complete", the target model is removed from the computing node before migration.

[0010] In one embodiment, the static load further includes a third load threshold, and the second load state includes a third load sub-state and a fourth load sub-state, wherein the third load threshold is greater than the first load threshold, the third load sub-state is obtained when the real-time load is greater than or equal to the first load threshold and less than the third load threshold, and the fourth load sub-state is obtained when the real-time load is greater than or equal to the third load threshold; After determining the target model in the model set and the target graphics processor corresponding to the target model in each of the available graphics processors, the method further includes: If the second load state is the third load sub-state and the available graphics processor does not exist, the step of migrating the target model to the computing node corresponding to the target graphics processor is cancelled. If the second load state is the fourth load sub-state and the available graphics processor does not exist, the step of migrating the target model to the computing node corresponding to the target graphics processor is canceled, and the application is circuit-broken according to the priority of the application running on the graphics processor.

[0011] In one embodiment, after the step of circuit breaking the application based on the priority of the application running on the graphics processor, the method further includes: The applications that have been circuit-broken are prioritized according to a preset application recovery cycle. The applications to be recovered among the circuit-broken applications and the load of the applications to be recovered are determined. The applications to be recovered are the applications with the highest priority among the circuit-broken applications. Based on the load of the application to be restored and the predicted load of the graphics processor as assessed by the real-time load, the application to be restored is restored if the load state corresponding to the predicted load is the first load state.

[0012] In one embodiment, the step of obtaining the real-time load and static load of each graphics processor in the graphics processor cluster includes: For any graphics processor in the graphics processor cluster, obtain the static load of the graphics processor and the real-time load at the current sampling point; The dynamic load average is determined based on the real-time load and the real-time load of the graphics processor at each historical sampling point. A new static load is determined based on the static load and its corresponding first weight, and the dynamic load average and its corresponding second weight. The first weight is positively correlated with the monitoring duration of the real-time load, and the initial value of the second weight is positively correlated with the initial load deviation between the static load and the dynamic load average during the monitoring period.

[0013] In one embodiment, before the step of determining the new static load based on the static load and its corresponding first weight, and the dynamic load average and its corresponding second weight, the method further includes: The static load is compared with the average dynamic load to obtain the load deviation value of the graphic processing. If the load deviation value is greater than a preset deviation threshold, the first weight and the second weight are adjusted. Specifically, if the load deviation value increases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is decreased and the second weight is increased; if the load deviation value decreases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is increased and the second weight is decreased.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the graphics processor cluster resource scheduling method as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application firstly, by continuously monitoring the real-time load and comparing it with the static load, can perceive the deviation between the dynamic load (i.e., real-time load) and the static load of each GPU in real time, thereby quantifying the abstract load situation into a specific load state. This provides a dynamic and accurate evaluation basis for resource scheduling decisions, breaking through the limitation that static allocation cannot perceive runtime changes. Secondly, based on this dynamically evaluated load state, the scheduling of the target model is triggered and executed, which means that resource scheduling decisions are no longer static behaviors of initialization, but can be dynamically responded to and actively adjusted according to the real-time running status of the GPU cluster.

[0016] In summary, this application transforms GPU resource allocation from fixed to dynamic, significantly improving the dynamism of resource scheduling. Furthermore, through the comparison mechanism between real-time and static loads, the system can adaptively identify and respond to load changes at different times and under different task pressures, enhancing the system's adaptability to different operating scenarios. Ultimately, this achieves the effect of optimizing the overall resource utilization of the cluster and avoiding local overload or idleness. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the graphics processor cluster resource scheduling method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the graphics processor cluster resource scheduling method of this application. Figure 3 This is a schematic diagram of the system framework of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 4 This is a schematic diagram of the framework flow of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the state determination process of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 6 This is a schematic diagram of the resource scheduling process of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 7This is a schematic diagram of the static load adjustment process of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 8 This is a schematic diagram of the application circuit breaker process for the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 9 This is a schematic diagram of the application recovery process of the graphics processor cluster resource scheduling method provided in Embodiment 2 of this application; Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the graphics processor cluster resource scheduling method in this application embodiment. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is: to obtain the real-time load and static load of each graphics processor in the graphics processor cluster; for any graphics processor, to compare the real-time load and static load of the graphics processor to obtain the load status of the graphics processor; and to perform internal scheduling of the target model in the graphics processor cluster according to the load status.

[0023] Existing solutions typically focus on the initial allocation and static scheduling of GPU resources in a cluster environment. That is, by analyzing the physical connection topology of GPUs, high-performance connection combinations are prioritized during container scheduling to optimize GPU data exchange efficiency. However, these methods mainly start from the hardware interconnect level, and resource allocation is essentially a static allocation method. Once virtual GPU resources are mounted to a specific container, they are fixed, which has obvious shortcomings in terms of the dynamism and adaptability of GPU resource allocation.

[0024] This application provides a solution that transforms GPU resource allocation from fixed to dynamic, significantly improving the dynamism of resource scheduling. At the same time, through a mechanism that compares real-time load with static load, the system can adaptively identify and respond to load changes at different times and under different task pressures, enhancing the system's adaptability to different operating scenarios. Ultimately, this achieves the effect of optimizing the overall resource utilization of the cluster and avoiding local overload or idleness.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or graphics processor cluster resource scheduling system capable of performing the above functions. The following description uses a graphics processor cluster resource scheduling system as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide a method for scheduling graphics processor cluster resources, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the graphics processor cluster resource scheduling method of this application.

[0027] In this embodiment, the graphics processor cluster resource scheduling method includes steps S10~S30: Step S10: Obtain the real-time load and static load of each graphics processor in the graphics processor cluster; It should be noted that real-time load refers to the actual workload carried by the graphics processing unit (GPU) at a specific moment, obtained through monitoring. This data directly reflects the status of various computing resources currently being consumed by the GPU, including but not limited to dynamic indicators such as real-time computing power utilization, memory usage, temperature, or power consumption. Monitoring real-time load is the starting point of the dynamic scheduling process, aiming to continuously capture the instantaneous operating status of each GPU in the cluster, providing accurate, time-varying load data input for subsequent comparisons and decisions.

[0028] Static load is a pre-determined or pre-defined load reference value for each graphics processor. This value is typically determined based on the GPU's hardware capabilities, the pre-defined resource requirements of the deployed model, or historical operating experience, representing the load level that the GPU should withstand under standard, expected, or typical operating conditions. The core function of determining static load is to provide a comparative anchor for real-time load. By comparing the real-time load with this benchmark, it objectively assesses whether the current operating state deviates from expectations, thus laying the foundation for generating load status.

[0029] Step S20: For any graphics processor, compare the real-time load of the graphics processor with the static load to obtain the load status of the graphics processor. It's important to note that load status is a status indicator representing the current load condition of the GPU, obtained by comparing the real-time load of the graphics processor with its static load. This comparison process essentially calculates the degree or relationship between the real-time load and the static load, transforming the specific load value into a higher-level state classification for decision-making (e.g., indicating high, low, or moderate load). As an evaluation conclusion, load status abstracts and simplifies complex load data, providing a direct and clear basis for triggering and guiding model scheduling.

[0030] Understandably, this approach requires comparing the constantly changing actual load of the graphics processing unit (GPU) during real-time operation with a value reflecting its expected or baseline load level. This allows for a quantitative and decision-making assessment of its current operating status. By pre-determining a static load as a comparison benchmark for each GPU and directly comparing the real-time load obtained from real-time monitoring with this static load, the problem of inaccurate and quantifiable assessment of whether the GPU is currently under light, overloaded, or in a normal state due to the lack of a dynamic comparison mechanism between runtime load and benchmark values ​​is avoided. This prevents blind or delayed scheduling decisions. The approach achieves dynamic and accurate assessment of the operating status of each GPU in the cluster, transforming complex real-time load data into clear load status indicators, providing accurate and reliable triggering basis and decision-making foundation for subsequent resource scheduling actions.

[0031] Step S30: Perform internal scheduling of the target model in the graphics processor cluster according to the load status of each load. It should be noted that the target model specifically refers to the software model instance currently deployed on a specific compute node of a particular GPU within the GPU cluster, and selected as the target for migration. This model can be a computational model performing AI inference or training tasks. During the scheduling process, the system selects this type of model as the migration target from the GPUs whose load status indicates the need for adjustment, based on the load status of each GPU. By redeploying this model to compute nodes corresponding to other GPUs, the system achieves load rebalancing and optimized resource utilization within the cluster.

[0032] Understandably, in order to proactively and purposefully adjust the distribution of computing tasks (models) in the cluster based on the load status assessed in the preceding steps, which reflects the load conditions of each graphics processor, the system directly triggers and selects target models from the GPUs that need adjustment based on the load status assessment results, and migrates them to the computing nodes corresponding to other GPUs. This avoids the problem in existing static resource allocation schemes where resources are fixed once a model is deployed, and cannot be dynamically rebalanced according to real-time load fluctuations in the cluster, resulting in some GPUs being overloaded while others are idle, leading to rigid resource utilization. This system achieves dynamic scheduling and elastic allocation of GPU cluster resources, and can automatically and timely adjust the model deployment location according to the real-time load status, thereby optimizing overall load balancing, improving resource utilization efficiency, and preventing local overload.

[0033] For example, the system periodically collects real-time load data for each GPU in the GPU cluster, which can be the current computing power and memory utilization. Simultaneously, the system maintains a preset static load value for each GPU, based on its hardware specifications and typical task load. Next, for any GPU, the system compares its collected real-time load value with the GPU's static load value. If the real-time load is greater than the static load, a "high load state" flag is generated; otherwise, a "low load state" flag is generated. Then, the system makes scheduling decisions based on the generated load state flags: when a GPU is detected as "high load," the system automatically selects the model with the largest load contribution from among the multiple models currently deployed on its compute node as the target model. Simultaneously, the system searches the cluster for a compute node corresponding to a GPU currently marked as "low load" as the target migration destination. Finally, the system executes a migration operation, migrating the selected target model from the compute node of the high-load source GPU to the compute node of the low-load target GPU, thus completing a dynamic resource scheduling operation.

[0034] This embodiment provides a method for scheduling graphics processor cluster resources, which transforms GPU resource allocation from fixed to dynamic, significantly improving the dynamism of resource scheduling. At the same time, through the comparison mechanism between real-time load and static load, the system can adaptively identify and respond to load changes at different times and under different task pressures, enhancing the system's adaptability to different operating scenarios, and ultimately achieving the effect of optimizing the overall resource utilization of the cluster and avoiding local overload or idleness.

[0035] In one feasible implementation, the static load includes a first load threshold, and the load state includes a first load state and a second load state, wherein the first load state is obtained when the real-time load is less than the first load threshold, and the second load state is obtained when the real-time load is greater than or equal to the first load threshold. It should be noted that the first load state refers to a specific load state obtained by comparing the GPU's real-time load with a first load threshold included in the static load. The determination condition is that the real-time load is less than the first load threshold. This state indicates that the GPU's current real-time load is below a preset threshold, meaning that it has sufficient remaining processing power or is operating under light load. In the scheduling logic, GPUs determined to be in the first load state are considered potential resource providers.

[0036] The second load state refers to a specific load state obtained by comparing the GPU's real-time load with a first load threshold included in the static load. The determination condition is that the real-time load is greater than or equal to the first load threshold. This state indicates that the GPU's current real-time load has reached or exceeded a preset threshold, meaning its processing capacity is strained or operating under high load. In the scheduling logic, GPUs determined to be in the second load state are considered objects requiring load adjustment or resource release.

[0037] Step S30 may include steps S31 to S33: Step S31: Traverse each of the graphics processors; Step S32: Mark the graphics processor with the first load state as an available graphics processor, wherein the computing node corresponding to the available graphics processor is used to deploy a new model; It's important to note that "available GPUs" specifically refers to one or more GPUs marked as "first load state" after traversing the GPU cluster and performing load status assessment. This marking is a scheduling decision based directly on the processor's real-time load being less than the first load threshold. Once marked as an available GPU, it is formally incorporated into the system's available resource pool. Its core purpose is to prepare its corresponding compute nodes to receive and deploy new models, thereby providing additional computing power to the cluster.

[0038] In specific implementation, the static load also includes a second load threshold. The first load state includes a first load sub-state and a second load sub-state. The second load threshold is less than the first load threshold. The first load sub-state is obtained when the real-time load is less than the second load threshold. The second load sub-state is obtained when the real-time load is greater than or equal to the second load threshold and less than the first load threshold. It should be noted that the first load sub-state refers to a more specific load state obtained by comparing the GPU's real-time load with the second load threshold included in the static load. The determination condition is that the real-time load is less than the second load threshold. This state indicates that the GPU's current real-time load is far below a preset core threshold, meaning it is operating in a highly idle or extremely low-load condition. In the scheduling logic, GPUs determined to be in the first load sub-state are considered the highest quality and most prioritized computing resources.

[0039] The second load sub-state refers to a more specific load state obtained by comparing the GPU's real-time load with the second and first load thresholds included in the static load. The determination condition is that the real-time load is greater than or equal to the second load threshold and less than the first load threshold. This state indicates that the GPU's current real-time load has exceeded a lower warning threshold but has not yet reached a higher core threshold, meaning it is operating under light load with considerable capacity remaining. In the scheduling logic, GPUs determined to be in the second load sub-state are considered second-priority available resources.

[0040] Step S32 may also include steps S321 to S322: Step S321: Mark the graphics processor with the load state of the first load sub-state as the first type of available graphics processor, wherein the computing node corresponding to the first type of available graphics processor is used to deploy the new model. It should be noted that the first category of available GPUs refers to those GPUs marked as "first load sub-state" after further refinement of load status judgment. This marking is based on a stricter criterion that the real-time load is less than the second load threshold. Once marked as a first-category available GPU, the processor is included in the highest priority available resource pool, and its core purpose is to serve as the primary target, with its corresponding compute nodes used to deploy new models.

[0041] Step S322: Mark the graphics processor with the load state of the second load sub-state as the second type of available graphics processor, wherein the computing node corresponding to the second type of available graphics processor is used to receive new model deployments when the first type of available graphics processor does not exist.

[0042] It should be noted that the second category of available GPUs refers to those GPUs marked as "second load sub-state" after further refining the load status assessment. This marking is based on the criterion that the real-time load is between the second load threshold and the first load threshold. After being marked as a second-category available GPU, the processor is included in the secondary priority available resource pool. Its core purpose is to serve as a backup target for receiving new model deployments when the first category of available GPUs is unavailable, i.e., when there are no highly idle GPUs in the cluster.

[0043] Understandably, since the idle levels of available GPUs still vary significantly, treating them equally without distinction could lead to GPUs already under light load being prematurely assigned new tasks and quickly entering a high-load state, thereby weakening the scheduling system's buffering capacity and long-term stability. Therefore, this implementation adds a lower second load threshold to the static load, further subdividing the "first load state" into "first load sub-state" and "second load sub-state," and accordingly marking GPUs as "first-class available graphics processors" with higher priority and "second-class available graphics processors" with lower priority. Furthermore, new models are set to be deployed first-class first, and second-class only when there are no first-class available. This avoids the problems that may arise from coarse-grained management of available resources, such as uneconomical resource allocation and failure to prioritize the use of the most idle resources, which could shorten the overall system load buffer window. This achieves fine-grained classification and priority scheduling of available GPU resources, ensuring that new tasks are preferentially carried by the most idle GPUs, thereby establishing a better load gradient within the cluster, improving the rationality of resource allocation, the system's buffering capacity to cope with load fluctuations, and long-term stability.

[0044] Step S33: Migrate the target model corresponding to the graphics processor with the second load state to the computing node corresponding to the target graphics processor among the available graphics processors.

[0045] In the specific implementation process, step S33 may include steps S301 to S303: Step S301: Obtain the set of models deployed on the computing node corresponding to the graphics processor whose load state is the second load state; Step S302: Determine the target model in the model set, and the target graphics processor corresponding to the target model in each of the available graphics processors; Step S303: Migrate the target model to the computing node corresponding to the target graphics processor.

[0046] Understandably, since the scheduling instruction to migrate the target model requires a clear source and destination of the operation object, and the model, as a software entity, is not directly deployed on the GPU hardware, but rather on the computing node containing the GPU, the precise selection of the source and target of the model migration operation is achieved by explicitly obtaining the set of models deployed on the computing node corresponding to the graphics processor in the second load state during migration execution, and then migrating the model to the computing node corresponding to the target graphics processor. This allows the scheduling instruction to be accurately mapped to the computing node entity where the model is actually deployed.

[0047] As an example, the static load further includes a third load threshold, and the second load state includes a third load sub-state and a fourth load sub-state, wherein the third load threshold is greater than the first load threshold, the third load sub-state is obtained when the real-time load is greater than or equal to the first load threshold and less than the third load threshold, and the fourth load sub-state is obtained when the real-time load is greater than or equal to the third load threshold; It's important to note that the third load sub-state refers to a specific load state obtained by comparing the GPU's real-time load with the first and third load thresholds included in the static load. The determination condition is that the real-time load is greater than or equal to the first load threshold and less than the third load threshold. This state indicates that the GPU's real-time load has exceeded the first load threshold that classifies it as "needing adjustment," but has not yet reached the higher third load threshold, which represents an extremely stressful state. This means it is in a clearly heavy-load operating condition but has not yet reached a dangerous critical point. In the scheduling logic, GPUs determined to be in the third load sub-state are the core objects that trigger model migration operations to actively alleviate their load.

[0048] The fourth load sub-state refers to a specific load state obtained by comparing the GPU's real-time load with the third load threshold included in the static load. The determination condition is that the real-time load is greater than or equal to this third load threshold. This state indicates that the GPU's real-time load has reached or exceeded a preset, higher danger threshold, meaning it is in a dangerous operating condition of extreme stress, nearing overload, or already overloaded. In the scheduling logic, GPUs determined to be in the fourth load sub-state are key targets for triggering more urgent and stronger intervention measures (such as model migration or even application circuit breaking).

[0049] After step S302, steps S331 to S332 may also be included: Step S331: If the second load state is the third load sub-state and the available graphics processor does not exist, cancel the step of migrating the target model to the computing node corresponding to the target graphics processor. Step S332: If the second load state is the fourth load sub-state and the available graphics processor does not exist, cancel the step of migrating the target model to the computing node corresponding to the target graphics processor, and circuit breaker the application according to the priority of the application running on the graphics processor.

[0050] Understandably, within the second load state, which is also "requiring adjustment," the severity and urgency of the load vary significantly. Using the same processing strategy might lead to insufficient response to GPUs already on the verge of danger, causing service overload, or unnecessary excessive intervention for GPUs not yet at a dangerous level. Therefore, by adding a higher third load threshold to the static load, the "second load state" is further subdivided into "third load sub-states" and "fourth load sub-states," with differentiated processing strategies for each sub-state: for the third load sub-state, migration is only performed if available GPUs exist; otherwise, no processing is required. For the fourth load sub-state, if no available GPUs are available, priority-based application circuit breaking is triggered directly. This avoids the problems that might arise from using a uniform migration strategy for high load conditions, such as system lag and inability to quickly prevent overload crashes under extreme resource constraints. It achieves refined hierarchical management and differentiated emergency response for high-load GPUs, enabling the system to automatically select appropriate control intensity based on the severity of the overload, thereby significantly enhancing the system's self-protection capability and overall stability under high pressure.

[0051] In this implementation, to clearly determine which GPUs are eligible to receive new models and where the migration should originate and proceed, a first load threshold is preset in the static load as the core judgment benchmark. By traversing the cluster, the real-time load of each GPU is compared with this threshold, thereby specifically distinguishing their load states into "first load state" and "second load state". Based on this, GPUs in the first load state are marked as "available GPUs" for deploying new models, while the target models on GPUs in the second load state are migrated to these available GPUs. This avoids the problems of ambiguous migration decisions and low matching efficiency between resource providers and demanders caused by the lack of a unified and quantitative load state judgment standard and resource pool marking mechanism, which may lead to invalid migration or migration delays. It achieves clear and rapid classification and marking of GPU resource supply and demand states, so that the resource pool of "available GPUs" is clearly established. The migration process becomes a targeted operation from a clear "second load state" provider to a clear "available GPU" recipient, thereby significantly improving the accuracy, executability and overall efficiency of scheduling decisions.

[0052] In one feasible implementation, step S302, which involves determining the target model in the model set and the target graphics processor corresponding to the target model among the available graphics processors, may further include steps S331-S332: Step S301: Determine the target model among the models based on the load occupancy of each model in the model set, wherein the target model is the model with the smallest load occupancy among the models. It should be noted that load consumption refers to the actual amount of hardware resources consumed by the target model when it is deployed and running on its current GPU. This quantitative metric can include, but is not limited to, the amount of GPU memory continuously used for model inference or training tasks, as well as the computing power utilized, such as GPU core utilization and other key resources. Its value can be derived from dynamic monitoring data during model runtime or estimated based on its static resource requirements. In the migration decision-making process, the model with the lowest load consumption is compared and selected as the target model. The aim is to prioritize migrating models with low resource consumption, minimal impact on overall system performance during migration, and short migration operation time, thereby achieving the optimization goal of load balancing with the lowest scheduling cost.

[0053] Step S302: Based on the target model, query the models deployed in the computing nodes corresponding to each available graphics processor. If there is a computing node corresponding to an available graphics processor that has a model with the same model as the target model deployed thereon, then the available graphics processor corresponding to the computing node that has a model with the same model as the target model is taken as the target graphics processor.

[0054] Understandably, when multiple models are deployed on a single GPU's corresponding compute node, blindly migrating without clearly matching the model selection to the target GPU may result in choosing a model with excessive load, leading to high migration costs and long migration times, or choosing a target GPU that cannot effectively share resources, resulting in poor overall resource utilization. Therefore, this implementation method prioritizes selecting the model with the lowest load based on the load of each model when determining the target model to reduce migration overhead and business impact. When determining the target graphics processor, it prioritizes selecting available GPU compute nodes that have already deployed the same model to achieve instance reuse and save GPU memory. This avoids the problems that may be caused by improper selection of migration objects and targets, such as excessive consumption and long migration times, significant disruption to business, and insufficient utilization of existing model instances leading to redundant GPU memory usage. It achieves efficient load rebalancing by minimizing migration costs and resource consumption, improves the efficiency, success rate, and economic benefits of migration operations, and optimizes the overall resource utilization of the cluster.

[0055] Furthermore, it should be noted that when no model identical to the target model is deployed on any of the available graphics processors' corresponding computing nodes, a target graphics processor is selected from among the available graphics processors according to predetermined selection rules (such as the lightest load), and a new instance of the target model is deployed on its computing node. Specifically, the complete runtime environment and parameters of the target model are deployed to the computing node of the selected available graphics processor, making it a new, independently running model instance, thus completing the migration. This operation ensures the universality and robustness of the migration strategy. Even under optimal conditions where no model instance can be reused, the system can still enforce load balancing decisions by creating a new instance, thereby guaranteeing the necessary executability of the scheduling mechanism and the overall load adjustment capability of the system, avoiding scheduling deadlock or failure due to imperfect target conditions.

[0056] For example, the system first periodically collects the real-time memory utilization and computing power utilization of all running models on each compute node through a monitoring agent deployed on each node. The two metrics are then summed according to preset weights (e.g., memory weight 0.6, computing power weight 0.4) to calculate the current load of each model. When a compute node corresponding to a GPU is determined to require model migration, the system compares the load utilization of all models on that compute node and selects the model with the smallest value as the target model. Subsequently, the system queries the model deployment registry of all compute nodes corresponding to GPUs marked as "available GPUs" to find compute nodes that have already deployed instances with the same model identifier (e.g., the same algorithm version and parameter hash value) as the target model. If found, the system identifies the GPU corresponding to the compute node as the "target graphics processor" and triggers the migration process: a new instance of the target model is started on the compute node corresponding to the target graphics processor (or an existing instance is reused to carry more requests), the routing configuration of the load balancer is updated synchronously, and subsequent requests for the model are directed to the compute node of the target graphics processor. After the old instance on the compute node corresponding to the original GPU finishes processing the existing requests, it is safely unloaded, thereby completing the load adjustment in a way that minimizes resource consumption and migration time.

[0057] In this implementation, when triggering model migration, the model with the smallest load occupancy in the model set is selected as the target model. When determining the target graphics processor, available graphics processors that have already deployed the same model as the target model are queried and selected first. This effectively avoids the problems that may occur during load balancing, such as excessive overhead and long time consumption of the migration operation itself, significant delay and disturbance to business request processing, and duplicate occupation of additional video memory resources due to failure to reuse existing model instances. It achieves efficient scheduling by minimizing the resource and time costs of migration operations. By prioritizing the migration of lightweight models, the impact on system performance is reduced. By prioritizing the selection of GPUs with the same instance, video memory sharing is maximized and redundant occupation is reduced, thereby significantly improving migration efficiency, success rate and overall cluster resource performance.

[0058] In one feasible implementation, step S303 may further include steps S3031 to S3033: Step S3031: Deploy the target model on the computing node corresponding to the target graphics processor, and update the routing configuration corresponding to the target model, wherein the routing configuration is used to indicate the graphics processor corresponding to the model call request; It's important to note that routing configuration is a set of network or software rules used to accurately direct externally initiated model call requests to the specific GPU instance in the cluster currently responsible for handling those requests. This routing configuration typically exists, but is not limited to, in the form of routing tables, load balancing strategies, or service discovery mechanisms, explicitly recording the mapping between each available model instance and the network address or identifier of its current host GPU. Updating this configuration is a critical step during model migration, ensuring that after the target model is deployed to the new compute node corresponding to the target GPU, all newly arriving requests for that model are correctly and seamlessly forwarded to the new instance, thus achieving a smooth service switchover.

[0059] A model invocation request refers to an external instruction or data packet that requires a model in the GPU cluster to perform a computational task. This request may, but is not limited to, contain the input data required by the model (e.g., images, text, or feature vectors) and the expected model output. The request may be initiated by an upper-layer application, algorithm service, or other system component and sent to the GPU cluster over the network. The system distributes the request to the specific model instance running on the corresponding GPU for processing based on the routing configuration.

[0060] Step S304: Monitor the processing status of the target model's model call request in the computing node before migration; It's important to note that the processing status refers to the monitoring and assessment during model migration of the original target model instance still running on the source GPU (i.e., the graphics processor before migration) to determine whether it is still processing received but incomplete model call requests. This data focuses on whether the old instance is in a specific state such as "actively processing" (corresponding to requests still being computed), "idle" (corresponding to no pending requests), or "has completed all scheduled tasks." The purpose of monitoring this status is to serve as a basis for decisions regarding the safe removal of old instances, ensuring that all received requests are properly processed during the migration process, avoiding request loss or service interruption.

[0061] Step S305: If the processing status is "processing complete", remove the target model from the computing node before migration.

[0062] Understandably, since the migration operation itself may directly lead to service interruption or request loss, this implementation further deploys a new instance of the target model on the compute node corresponding to the target graphics processor during migration and immediately updates the routing configuration to direct new requests to the new instance. At the same time, it continuously monitors the processing status of the old model instance on the compute node before migration for existing requests, and removes it only after confirming that it has been processed. This effectively avoids the problems of lost processing model call requests, temporary service unavailability, or client request failure caused by untimely request routing switching or premature termination of old instances in the traditional "stop before migrate" or simple copy process. It achieves uninterrupted migration, ensuring that all model call requests can be processed continuously and without interruption during load redistribution, thereby achieving a smooth service switch that is imperceptible to users and greatly improving the service availability and reliability of the system during dynamic scheduling.

[0063] For example, when the system decides to migrate the target model from the source GPU's corresponding compute node (i.e., the compute node before migration) to the target GPU's corresponding compute node, it first loads and starts a brand new instance of the target model on the target GPU's corresponding compute node. Next, the system dynamically updates its routing configuration by calling the cluster load balancer's management API, adding the address of the new instance on the target GPU's corresponding compute node to the list of backend servers for the virtual service endpoint of the target model. Optionally, it sets the weight of the old instance on the source GPU's corresponding compute node to zero or removes it from the list, ensuring that all newly arriving model call requests are directly distributed to the new instance. Simultaneously, a monitoring agent deployed on the source GPU's corresponding compute node continuously monitors the request queue length and the number of active threads of the old model instance to determine its processing status. When the monitoring agent detects that the instance's request queue is empty and all active compute threads have completed (i.e., the processing status is completed), the system triggers a safety cleanup process. First, it stops receiving any potential requests flowing to the old instance, then it unloads its model and releases the occupied GPU memory, finally completing the migration.

[0064] Furthermore, it should be noted that when the processing status is "incomplete," the system will perform continuous monitoring and waiting, i.e., postpone or suspend the final step of removing the target model from the pre-migration graphics processor. Specifically, the system will continue to keep the old model instance on the pre-migration compute node running, allowing it to continue processing its received but not yet completed model call requests. Simultaneously, the system will continue to ensure, through updated routing configurations, that all new model call requests are only distributed to the new model instance on the corresponding compute node of the target graphics processor, and no new requests are sent to the old instance. The system will continuously and periodically check metrics such as the old instance's request queue, active threads, or connection count until its processing status changes to "processing complete," at which point the final cleanup operation of removing the old instance will be triggered and executed. This operation ensures that no processing request is forcibly interrupted or dropped during the entire model migration window, thus strictly guaranteeing a seamless service switchover and the integrity of data processing.

[0065] In this implementation, by first deploying the target model on the computing node corresponding to the target graphics processor and updating the routing configuration to guide new requests, and continuously monitoring the processing status of the original target model on the computing node before migration for existing requests, the original target model is removed only when the processing status is completed. This avoids the problems of lost model call requests, temporary service interruptions, or client request failures caused by the asynchronous start-up and shutdown of model instances and request route switching during dynamic load adjustment. It achieves seamless connection of business request processing during the model migration process. Through the parallel operation of old and new instances and the smooth switching of requests, it ensures that all received and subsequent model call requests can be continuously processed. Thus, while realizing dynamic resource scheduling, it ensures the continuous availability of services and data consistency, significantly improving the reliability of the system.

[0066] In one feasible implementation, after step S332, which involves circuit breaking the application based on its priority on the graphics processor, steps S333-S334 may be included: Step S333: Prioritize the circuit-broken applications according to the preset application recovery cycle, determine the applications to be recovered among the circuit-broken applications, and the load of the applications to be recovered, wherein the applications to be recovered are the applications with the highest priority among the circuit-broken applications. It should be noted that the preset application recovery cycle is a pre-configured, fixed, or dynamically adjustable time interval parameter that controls the frequency at which the system periodically executes the recovery detection process. This cycle determines how often the system proactively checks whether applications previously suspended due to the circuit breaker mechanism are ready to resume operation (e.g., every 30 seconds or every minute). The purpose of setting this cycle is to achieve a manageable balance between avoiding excessive system resource consumption from overly frequent detection operations and ensuring timely service recovery after resource release to improve business continuity.

[0067] The application to be recovered refers to the specific application that is prioritized for recovery when the recovery detection process is triggered during a preset application recovery cycle. This application is selected from all applications currently in a circuit breaker state, sorted according to their preset priorities. The criterion for this is the application with the highest priority among those that have experienced circuit breakers. Listing the highest-priority circuit-breaker application as the application to be recovered reflects a scheduling strategy that prioritizes ensuring the continuity of more important business operations when resources allow, and is the core manifestation of a refined circuit breaker and recovery mechanism.

[0068] Step S334: Based on the load occupancy of the application to be restored and the real-time load, assess the predicted load of the graphics processor. If the load state corresponding to the predicted load is the first load state, restore the application to be restored.

[0069] It's important to note that predicted load refers to an estimated load value pre-calculated by the system to assess the potential load impact on the graphics processor if the application to be restored is restarted. This value is calculated by superimposing or combining the current real-time load of the graphics processor with the load consumption of the application to be restored (i.e., the typical or historical resource consumption required for the application to run normally). Predicted load is used to proactively determine whether a recovery operation will immediately cause the GPU's load state to re-enter a high-risk area requiring another circuit breaker, and it is a key basis for making safe recovery decisions.

[0070] Understandably, since circuit breaking is an emergency braking measure, without a systematic automatic recovery mechanism, the broken applications will be stagnant for a long time, the resource window cannot be effectively utilized, and the system cannot intelligently restart services after the load decreases. Therefore, this implementation further sets a preset application recovery cycle to trigger detection at regular intervals. In each cycle, the broken applications are sorted by priority and the highest priority application is selected as the application to be recovered. The predicted load is calculated by superimposing its real-time load and the load occupancy of the application to be recovered. Recovery is only performed when the load state corresponding to the predicted load is a safe first load state. This avoids the problems of improper recovery timing, repeated system load fluctuations, and indefinite blocking of low-priority applications that may be caused by relying on manual intervention or simple timed recovery after circuit breaking. It realizes a complete closed loop and intelligence of the circuit breaking and recovery mechanism, ensuring that the system recovers services in an orderly and safe manner according to business importance when resources allow. This not only improves resource utilization efficiency but also ensures the continuity of core business. At the same time, the forward-looking assessment of predicted load effectively prevents secondary overload caused by recovery operations and enhances the system's self-regulation and long-term stable operation capabilities.

[0071] For example, the system triggers a recovery detection process periodically according to a preset application recovery cycle (e.g., every 60 seconds). First, it sorts the applications that have been suspended in descending order based on their preset business priorities, identifies the application with the highest priority as the application to be recovered, and queries its historical running data to obtain its typical load usage. Next, the system obtains the current real-time load of the GPU and adds this real-time load to the load usage of the application to be recovered to calculate the predicted load. Subsequently, the system compares the predicted load with a first load threshold in the static load. If the predicted load is less than the threshold, that is, the load state corresponding to the predicted load is the first load state, it is determined that it is safe to recover the application, and the system immediately performs the recovery operation, reallocates resources to the application, and starts its request processing process. If the predicted load exceeds the threshold, the system abandons the recovery of the application in the current cycle and waits for the next cycle to detect it again, thereby achieving safe and orderly automatic recovery.

[0072] In this embodiment, a technical solution is adopted to set a preset application recovery cycle to trigger detection at regular intervals. Within each cycle, applications that have been circuit-broken are prioritized and the application with the highest priority is selected as the application to be recovered. The predicted load is calculated by superimposing the real-time load of the GPU with the load occupancy of the application to be recovered. Recovery is only performed when the load state corresponding to the predicted load is a safe first load state. This avoids the problems that may arise after the implementation of the circuit breaker mechanism due to the lack of an automatic and safe recovery strategy, such as low efficiency caused by manual intervention, repeated system load fluctuations caused by blind recovery, and insufficient resource utilization and business unfairness caused by low-priority applications being blocked indefinitely. It realizes the automated closed-loop management of the circuit breaker and recovery process, and can restart services in an orderly and forward-looking manner according to business importance. It not only effectively prevents secondary overload that may be caused by recovery operations through the assessment of predicted load, ensuring system stability, but also significantly improves the technical effect of resource utilization efficiency and core business continuity.

[0073] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of obtaining the real-time load and static load of each graphics processor in the graphics processor cluster in step S10 may further include steps S11 to S13: Step S11: For any graphics processor in the graphics processor cluster, obtain the static load of the graphics processor and the real-time load at the current sampling point. It should be noted that the current sampling point refers to the latest load monitoring moment corresponding to the system's execution of the static load determination step. The real-time GPU load data collected at this moment is the real-time load under the current sampling point. This load represents the actual load of the GPU at the most recent monitoring instant, and is the latest and most immediate data point when calculating the dynamic load average, as well as the direct basis for triggering this static load adjustment calculation.

[0074] Step S12: Determine the dynamic load average based on the real-time load and the real-time load of the graphics processor at each historical sampling point; Historical sampling points refer to the various time points recorded by the system over several preset monitoring periods prior to the current sampling point. Each historical sampling point corresponds to a real-time GPU load data point collected and stored at that time. These historical data points together constitute a time series, used to analyze the long-term trend of the load and calculate the average value to smooth out the impact of short-term fluctuations.

[0075] The dynamic load average is an average value calculated using specific mathematical methods (such as arithmetic mean, weighted average, or exponential smoothing) based on real-time GPU load data collected at the current sampling point and a series of historical sampling points. This value comprehensively reflects the average load level of the GPU within the most recent monitoring time window, aiming to eliminate instantaneous fluctuations and more stably represent the dynamic operating characteristics of the GPU, serving as an important input when adjusting static load.

[0076] Step S13: Determine a new static load based on the static load and the first weight corresponding to the static load, and the dynamic load average and the second weight corresponding to the dynamic load average. The first weight is positively correlated with the monitoring duration of the real-time load, and the initial value of the second weight is positively correlated with the initial load deviation between the static load and the dynamic load average during the monitoring period.

[0077] It should be noted that the monitoring duration refers to the accumulated time from when the system begins monitoring the load on a specific GPU to the current moment, or the length of the historical time window considered when calculating the dynamic load average. This duration is a key factor in determining the magnitude of the first weight (α), and it is positively correlated with the first weight. This means that the longer the monitoring time, the richer the historical data accumulated by the system, and the more inclined it is to trust the historical benchmark when adjusting the static load (corresponding to giving a higher weight α to historical static demands), thereby enhancing the long-term stability of the system.

[0078] The initial load deviation refers to the difference between the observed GPU static load and the calculated average dynamic load during an initial period (or a specific calibration phase) after system load monitoring is initiated. This deviation is used to initialize the second weight (β), which is positively correlated with the initial value of the second weight. This means that the larger the initial deviation, the further the preset static load deviates from the actual operating average. In this case, the system needs to rely more on dynamic feedback in the initial stage (corresponding to giving a higher initial weight β to the average dynamic stress) to quickly correct the baseline and make the prediction closer to reality.

[0079] Additionally, it should be noted that during the determination and adjustment of the first weight α and the second weight β, the constraint condition of α+β=1 must be ensured; otherwise, it will lead to the divergence of the static load value.

[0080] It is understandable that the preset static load may gradually deviate from the actual needs due to hardware differences, long-term operational drift, or changes in business models. If a fixed benchmark is used for comparison for a long time, the load status assessment will be distorted, which will affect the long-term accuracy of scheduling decisions. Therefore, this embodiment adopts a scheme that obtains the static load of the GPU, combines the real-time load of the current sampling point and the historical sampling point to calculate the dynamic load average, and introduces a first weight positively correlated with the monitoring duration and a second weight positively correlated with the initial load deviation value. The static load is periodically calculated and updated through weighted fusion. This avoids the problem that the load status assessment gradually becomes inaccurate and the scheduling decision quality deteriorates over time due to the static load benchmark remaining unchanged for a long time and failing to reflect the actual operating trend of the system. It realizes the adaptive dynamic calibration and optimization of the static load benchmark, which can intelligently adjust according to the actual operating load mode of the GPU, thereby ensuring the long-term accuracy and reliability of the load status assessment. This provides a solid and self-evolving decision foundation for the entire dynamic scheduling system and significantly improves the adaptability and stability of the system in long-term operation.

[0081] For example, the system first acquires the GPU's preset initial static load value and periodically (e.g., every second) collects its real-time load as the current sampling point data. Simultaneously, it retrieves the real-time load data from the N most recent historical sampling points (e.g., the first 100 sampling points) from storage. Next, the system calculates the arithmetic mean of these current and historical sampling point data to obtain the dynamic load average. Then, the system determines a first weight α based on the accumulated monitoring duration since the current monitoring operation occurred (α = 0.8 if the duration is long enough), and determines the initial value of a second weight β based on the initial load deviation observed at the beginning of system startup (β initial value = 0.2 if the initial load deviation is large). Finally, the system calculates the new static load using the formula "new static load = α × historical static load (i.e., the previous static load value) + β × dynamic load average," and updates the result to the new static load of the GPU for subsequent load status evaluation, thereby achieving adaptive adjustment of the static load benchmark.

[0082] In this embodiment, by acquiring the static load of the graphics processor and combining it with the real-time load of current and historical sampling points, the dynamic load average is calculated. A first weight positively correlated with the monitoring duration and a second weight positively correlated with the initial load deviation value are introduced. The static load is periodically calculated and updated through a weighted fusion formula. This effectively avoids the problem of gradually distorted load status assessment and decreased long-term accuracy and reliability of scheduling decisions caused by the long-term fixed static load benchmark, which cannot adapt to changes in hardware performance, long-term drift of business load, or environmental differences. It realizes the automated and intelligent dynamic calibration of the static load benchmark, enabling it to continuously track and conform to the actual operating load mode of the GPU, thereby ensuring the timeliness and accuracy of the load assessment benchmark. This provides an adaptive and self-optimizing decision basis for the entire dynamic scheduling system, significantly improving the resource prediction accuracy, scheduling adaptability, and overall stability of the system during long-term operation.

[0083] In one feasible implementation, steps S101-S102 may be included before step S13: Step S101: Compare the static load with the average dynamic load to obtain the load deviation value of the graphic processing. It's important to note that the load deviation value refers to a quantitative difference metric obtained during dynamic feedback adjustment. This is achieved by comparing the GPU's static load with the average dynamic load calculated based on the real-time load (either by calculating the absolute value or relative proportion of the difference between the two). This value directly reflects the degree of deviation between the preset static load benchmark and the GPU's recent actual average operating load. The load deviation value is a key input signal that triggers and guides the adaptive adjustment of the first and second weights: when the value is large, it indicates that the current static load does not match the actual operating conditions, and the system will make the new static load calculation more dependent on dynamic feedback by decreasing the first weight and increasing the second weight; conversely, the opposite adjustment will be made. Therefore, the load deviation value is the core feedback variable for achieving continuous and accurate calibration of the static load benchmark.

[0084] Step S102: When the load deviation value is greater than a preset deviation threshold, the first weight and the second weight are adjusted. Specifically, when the load deviation value increases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is decreased and the second weight is increased; when the load deviation value decreases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is increased and the second weight is decreased.

[0085] Understandably, relatively fixed weights lack timeliness and precision in adjusting to complex and ever-changing real-world conditions, failing to respond sensitively to instantaneous deviations between static and actual loads. Therefore, this implementation method further compares the static load with the average dynamic load before weighted fusion to obtain a real-time load deviation value. This value serves as the core feedback signal, dynamically adjusting the first and second weights. Specifically, when the deviation value increases compared to the previous value, α decreases and β increases; when it decreases, α increases and β decreases. This avoids the shortcomings of rigid weight adjustment mechanisms that cannot accurately respond to short-term load fluctuations or trend changes, leading to delayed or over-adjusted static load updates and affecting the accuracy of subsequent load status assessments. This achieves adaptive closed-loop fine-tuning of weights, enabling the static load calculation process to sensitively and appropriately correct real-time performance deviations. Consequently, it significantly improves the dynamic accuracy and response speed of the static load benchmark in tracking actual load changes, further enhancing the real-time accuracy and robustness of the entire scheduling system's resource prediction and decision-making.

[0086] For example, before performing weighted fusion, the system first calculates the absolute value of the difference between the static load and the average dynamic load to obtain the current load deviation value. This load deviation value is then compared with a preset deviation threshold (e.g., 15%). If the deviation exceeds the threshold, a weight adjustment is triggered: the system retrieves the load deviation value recorded during the last weight adjustment and compares it with the current deviation value. If the current deviation value is larger than the previous one, the first weight α0.05 is decreased by a preset step size (e.g., 0.05), and the second weight β0.05 is increased accordingly. If the current deviation value is smaller than the previous one, α0.05 is increased by a preset step size, and β0.05 is decreased. The adjustment magnitude is proportional to the change in the deviation value, but does not exceed the preset maximum adjustment step size (e.g., 0.1). After the adjustment is completed, the system uses the updated α and β to calculate the new static load according to the formula "new static load = α × historical static load (i.e., the previous static load value) + β × average dynamic load". Furthermore, the adjustment step size can be the aforementioned preset step size or k * load deviation value, where k is a preset coefficient. This embodiment does not impose any specific limitations on this.

[0087] In this embodiment, by comparing the average static load with the average dynamic load before performing weighted fusion to obtain the load deviation value, and using this deviation value as the core feedback signal to dynamically adjust the first and second weights, the problem of inaccurate load status assessment caused by the fixed weight adjustment mechanism, which cannot respond sensitively and appropriately to real-time fluctuations or trend changes in the load, is avoided. This avoids the problem of delayed static load updates, over-adjustment, or under-adjustment, which leads to inaccurate load status assessment. The embodiment achieves intelligent, closed-loop adaptive fine-tuning of weight parameters, enabling the static load calculation process to track the dynamic changes of the actual load in real time and accurately. This significantly improves the real-time accuracy and adaptive capability of the static load benchmark, providing a more reliable and agile decision-making basis for the entire scheduling system, and further enhancing the dynamic performance and long-term stability of system resource prediction and scheduling optimization.

[0088] For example, to help understand the implementation flow of the graphics processor cluster resource scheduling method obtained in this embodiment combined with the above embodiment one, please refer to... Figure 3 , Figure 3 A system architecture diagram of a graphics processing unit cluster resource scheduling method is provided, specifically: The application layer, algorithm layer, and model layer constitute the top-down path for the transmission and transformation of resource requirements: business requirements are parsed into algorithm calls at the application layer, further transformed into requests for specific model instances by the algorithm layer, and finally, the model layer requests computing resources from the lower layer. This layered structure decouples and manages business characteristics, algorithm dependencies, and model resource requirements. The GPU cluster provides distributed physical computing power. The computing power management module, as the system's central hub, continuously monitors the real-time load of the GPU cluster, receives resource requests and dynamic pressure feedback from the model layer, and makes scheduling decisions (such as model migration, application circuit breaking / recovery) based on multi-layered requirement information and cluster status, thereby driving the GPU cluster to execute specific resource allocation and adjustment actions. This diagram fully illustrates the closed-loop process from "software requirement definition" to "hardware resource execution," demonstrating the collaboration and data flow relationships between various technical modules during dynamic scheduling.

[0089] Furthermore, referring to Figure 4The application layer, comprised of business scenario management, is responsible for parsing business configurations, constructing time-based computing power requirements, and managing application priorities and circuit breaker control. The algorithm scheduling layer, corresponding to the algorithm layer, is responsible for converting business requests into specific model inference requests and managing dependencies between algorithms. The bottom inference foundation layer, corresponding to the model layer, includes a static requirement system and a dynamic stress statistics module, responsible for providing model resource benchmarks and real-time operational data. The core computing power scheduling management module acts as a computing power management service, receiving monitoring and requirement information from each layer, making decisions based on resource allocation strategies (such as model splitting and deployment) in its scheduling decision center, and issuing control commands. The diagram illustrates the requirement transmission path from top to bottom, where business requests are converted into inference requests, and the closed-loop control flow of monitoring data feedback from bottom to top and scheduling commands being executed across layers, fully presenting the dynamic process from business triggering to underlying scheduling execution.

[0090] Furthermore, referring to Figure 5 This demonstrates the execution process of the two core phases of the technical solution: "resource monitoring" and "load assessment." In the resource monitoring phase, the system periodically acquires two types of data: model service runtime data and GPU real-time load data. This involves monitoring the real-time load of each GPU in the graphics processor cluster and calculating the dynamic load value for each GPU based on the latter. In the load assessment phase, the system compares the calculated dynamic load with the static load to obtain the deviation value and checks whether the utilization rate of a single resource (such as memory or computing power) exceeds 85% (corresponding to the third load threshold). If it does, the GPU state is directly determined to be "dangerous" (corresponding to the fourth load sub-state). Based on this, the system classifies the GPU state sequentially as "idle" (corresponding to the first load sub-state), "lightly loaded" (corresponding to the second load sub-state), and "heavily loaded" (corresponding to the third load sub-state) according to different threshold ranges for the dynamic load value (i.e., real-time load) (less than 20% (corresponding to the second load threshold), less than 60% (corresponding to the first load threshold), and less than 85%), regardless of whether it is through a single check or the dynamic load value exceeds 85%. Ultimately, this will lead to a "dangerous" state. This series of threshold judgments and state divisions specifically implements the subdivision of load states through multiple load thresholds, providing accurate state basis for subsequent scheduling decisions (such as model migration and circuit breaking).

[0091] Furthermore, referring to Figure 6This demonstrates the complete process of scheduling decisions and execution based on load assessment results in the technical solution. The process begins with determining the GPU's status: if the status is "idle," it is directly marked and added to the idle GPU pool to accept new model deployments, that is, GPUs with the first load state (or the first load sub-state) are marked as available graphics processors (or the first type of available graphics processors). If the status is "lightly loaded," it is marked and added to the lightly loaded GPU pool, and it is set to accept new model deployments only when the idle GPU pool is empty, corresponding to the secondary scheduling logic of the second type of available graphics processors. For GPUs with a "heavy load" status, the process enters the critical model migration decision stage: the system first selects a transferable model, that is, determines the target model based on factors such as model load occupancy; then it calculates the optimal target GPU and triggers the migration process. For GPUs with a "dangerous" status, the process first attempts to offload the model; if this fails, a circuit breaker strategy is triggered: applications are sorted by priority and circuit breakers are triggered starting from the lowest priority. After each application is circuit breaker triggered, the load is reassessed until it drops below a safe threshold, corresponding to a mechanism for circuit breaking and subsequent evaluation based on priority. Finally, once the migration decision takes effect (the migration flag is true), the process enters the model migration execution phase: the target GPU is selected according to the rule of "idle card priority", the new model instance is deployed, the routing configuration is updated and new requests are directed to the new instance, while the old instance is removed after it has finished processing the existing requests.

[0092] Furthermore, referring to Figure 7 This demonstrates the closed-loop workflow of the dynamic feedback adjustment mechanism in the technical solution. The process begins with initial static demand calculation, followed by dynamic stress detection and periodic collection of runtime data to obtain dynamic stress indicators, and then calculates the dynamic stress average, i.e., the dynamic load average. The core step is to calculate the prediction error, which is the load deviation value used to trigger adjustments. If the error exceeds a threshold, adaptive parameter adjustment is triggered: the system dynamically increases the β weight, decreases the α weight, or performs the opposite operation depending on whether the error continues to increase or decrease, and the adjustment range is limited to the maximum step size. This precisely realizes the technical solution of dynamically adjusting the first and second weights based on changes in the load deviation value. Subsequently, the process uses the adjusted α and β to calculate the new static demand according to the formula: Adjusted static demand = α * Historical static demand + β * Dynamic stress average. This formula is the core embodiment of the method for determining the new static load. Finally, when the feedback cycle is reached, the system writes the new static demand into the configuration to update the baseline; otherwise, monitoring continues. This completes the fully closed-loop adaptive process from error monitoring and parameter adjustment to baseline update, ensuring that the static load can continuously track and conform to the actual operating pressure.

[0093] Furthermore, referring to Figure 8This demonstrates the triggering conditions and execution process of the circuit breaker mechanism in the technical solution. The process begins with monitoring the dynamic load of the GPU and first determining whether its state is "dangerous". Upon confirmation of a dangerous state, the process does not immediately trigger the circuit breaker; instead, two key constraints are introduced for triggering: first, monitoring whether the duration of the dangerous state exceeds a preset threshold; and second, checking if there are available GPU resources for model offloading. The circuit breaker process is only truly triggered when both conditions are met simultaneously (i.e., the duration exceeds the threshold and there are no available resources for offloading). This logic corresponds to the situation where, in the case where the second load state is the fourth load sub-state and the available graphics processor is absent, the application running on the graphics processor is circuit-broken according to its priority. After entering the circuit breaker process, the system first collects the priorities of all applications on the GPU and sorts them in descending order. Then, it enters a cyclical decision-making process: if the GPU load is still greater than or equal to 80%, the application with the lowest priority in the current list is circuit-broken, and the GPU load is reassessed after each circuit breaker until the load drops below 80%. This process, which prioritizes circuit breakers from low to high and iteratively evaluates each one, fully implements a refined, priority-based circuit breaker strategy. The process concludes by logging the event, notifying the administrator, and initiating recovery checks, leading to the subsequent recovery mechanism. The entire process embodies the principle of using circuit breaking as a last resort and prioritizing the protection of core business continuity.

[0094] Furthermore, referring to Figure 9 This demonstrates the complete closed-loop process of the recovery detection mechanism after a circuit breaker failure in the technical solution. The process begins by setting a recovery detection cycle, corresponding to a preset application recovery cycle that periodically triggers recovery detection. Within each detection cycle, the system first checks if a list of applications that have failed circuit breakers exists. If so, these applications are sorted in ascending order of priority to determine the steps for recovering the applications among those that have failed circuit breakers. The application with the highest priority is the candidate to be checked. Subsequently, the process enters the crucial resource detection stage to determine whether the current system resources are sufficient to accommodate the application to be checked. Here, "sufficient resources" does not simply mean judging whether there are currently idle resources, but rather requires assessing the predicted load by combining the real-time load of the GPU and the load occupancy of the application, and determining whether the predicted load will put the GPU in a safe first load state. If resources are sufficient, the operation to recover the application is executed, and its status is updated and events are recorded; if resources are insufficient, its circuit breaker state is maintained. Regardless of whether recovery is successful, the system waits for the next cycle to execute again after completing this detection, thus forming a continuous, condition-driven automatic recovery loop. This flowchart clearly reveals how the recovery mechanism, based on three elements—cycle, priority, and security prediction—achieves an orderly and secure restart of circuit-broken applications, preventing recovery operations from causing secondary system overload.

[0095] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the graphics processor cluster resource scheduling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0096] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the graphics processor cluster resource scheduling method in Embodiment 1 above.

[0097] The following is for reference. Figure 10 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0098] like Figure 10As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0099] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0100] The electronic device provided in this application, employing the graphics processor cluster resource scheduling method in the above embodiments, can solve the technical problem of insufficient dynamism and adaptability in GPU resource allocation. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the graphics processor cluster resource scheduling method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0101] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0103] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for scheduling resources in a graphics processing unit cluster, characterized in that, The graphics processor cluster resource scheduling method includes: Obtain the real-time and static loads of each graphics processor in the graphics processor cluster; For any graphics processor, the real-time load of the graphics processor is compared with the static load to obtain the load status of the graphics processor. The target model is internally scheduled within the graphics processor cluster according to the respective load states.

2. The graphics processor cluster resource scheduling method as described in claim 1, characterized in that, The static load includes a first load threshold, and the load state includes a first load state and a second load state, wherein the first load state is obtained when the real-time load is less than the first load threshold, and the second load state is obtained when the real-time load is greater than or equal to the first load threshold; The step of internally scheduling the target model in the graphics processor cluster according to each of the load states includes: Iterate through each of the aforementioned graphics processors; The graphics processor with the first load state is marked as an available graphics processor, wherein the computing node corresponding to the available graphics processor is used to deploy new models; The target model corresponding to the graphics processor with the second load state is migrated to the computing node corresponding to the target graphics processor in each of the available graphics processors.

3. The graphics processor cluster resource scheduling method as described in claim 2, characterized in that, The static load further includes a second load threshold. The first load state includes a first load sub-state and a second load sub-state. The second load threshold is less than the first load threshold. The first load sub-state is obtained when the real-time load is less than the second load threshold. The second load sub-state is obtained when the real-time load is greater than or equal to the second load threshold and less than the first load threshold. The step of marking the graphics processor with the first load state as a usable graphics processor includes: The graphics processors whose load state is the first load sub-state are marked as the first type of available graphics processors, wherein the computing nodes corresponding to the first type of available graphics processors are used to deploy new models; The graphics processors whose load state is the second load sub-state are marked as second type of available graphics processors, wherein the computing nodes corresponding to the second type of available graphics processors are used to receive new model deployments in the absence of the first type of available graphics processors.

4. The graphics processing unit cluster resource scheduling method as described in claim 2, characterized in that, The step of migrating the target model corresponding to the graphics processor with the second load state to the computing node corresponding to the target graphics processor in each of the available graphics processors includes: Obtain the set of models deployed on the computing node corresponding to the graphics processor whose load state is the second load state; Determine the target model in the model set, and the target graphics processor corresponding to the target model in each of the available graphics processors; The target model is migrated to the computing node corresponding to the target graphics processor.

5. The graphics processor cluster resource scheduling method as described in claim 4, characterized in that, The step of determining the target model in the model set, and the target graphics processor corresponding to the target model in each of the available graphics processors, includes: The target model among the models is determined based on the load occupancy of each model in the model set, wherein the target model is the model with the smallest load occupancy among the models. Based on the target model, query the models deployed in the computing nodes corresponding to each available graphics processor. If there is a computing node corresponding to an available graphics processor that has a model with the same model as the target model deployed thereon, then the available graphics processor corresponding to the computing node that has a model with the same model as the target model is taken as the target graphics processor.

6. The graphics processor cluster resource scheduling method as described in claim 4, characterized in that, The step of migrating the target model to the computing node corresponding to the target graphics processor includes: The target model is deployed on the computing node corresponding to the target graphics processor, and the routing configuration corresponding to the target model is updated, wherein the routing configuration is used to indicate the graphics processor corresponding to the model call request; Monitor the processing status of model call requests in the computing nodes before migration of the target model; If the processing status is "processing complete", the target model is removed from the computing node before migration.

7. The graphics processor cluster resource scheduling method as described in claim 4, characterized in that, The static load also includes a third load threshold, and the second load state includes a third load sub-state and a fourth load sub-state, wherein the third load threshold is greater than the first load threshold, the third load sub-state is obtained when the real-time load is greater than or equal to the first load threshold and less than the third load threshold, and the fourth load sub-state is obtained when the real-time load is greater than or equal to the third load threshold; After determining the target model in the model set and the target graphics processor corresponding to the target model in each of the available graphics processors, the method further includes: If the second load state is the third load sub-state and the available graphics processor does not exist, the step of migrating the target model to the computing node corresponding to the target graphics processor is cancelled. If the second load state is the fourth load sub-state and the available graphics processor does not exist, the step of migrating the target model to the computing node corresponding to the target graphics processor is canceled, and the application is circuit-broken according to the priority of the application running on the graphics processor.

8. The graphics processor cluster resource scheduling method as described in claim 7, characterized in that, After the step of circuit breaking the application based on the priority of the application running on the graphics processor, the method further includes: The applications that have been circuit-broken are prioritized according to a preset application recovery cycle. The applications to be recovered among the circuit-broken applications and the load of the applications to be recovered are determined. The applications to be recovered are the applications with the highest priority among the circuit-broken applications. Based on the load of the application to be restored and the predicted load of the graphics processor as assessed by the real-time load, the application to be restored is restored if the load state corresponding to the predicted load is the first load state.

9. The graphics processing unit cluster resource scheduling method as described in claim 1, characterized in that, The steps for obtaining the real-time and static loads of each graphics processor in the graphics processor cluster include: For any graphics processor in the graphics processor cluster, obtain the static load of the graphics processor and the real-time load at the current sampling point; The dynamic load average is determined based on the real-time load and the real-time load of the graphics processor at each historical sampling point. A new static load is determined based on the static load and its corresponding first weight, and the dynamic load average and its corresponding second weight. The first weight is positively correlated with the monitoring duration of the real-time load, and the initial value of the second weight is positively correlated with the initial load deviation between the static load and the dynamic load average during the monitoring period.

10. The graphics processor cluster resource scheduling method as described in claim 9, characterized in that, Before the step of determining the new static load based on the static load and its corresponding first weight, and the dynamic load average and its corresponding second weight, the method further includes: The static load is compared with the average dynamic load to obtain the load deviation value of the graphic processing. If the load deviation value is greater than a preset deviation threshold, the first weight and the second weight are adjusted. Specifically, if the load deviation value increases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is decreased and the second weight is increased; if the load deviation value decreases compared to the load deviation value corresponding to the previous weight adjustment, the first weight is increased and the second weight is decreased.

11. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the graphics processor cluster resource scheduling method as described in any one of claims 1 to 10.

Images