A GPU resource efficient scheduling method based on multi-scene intelligent interaction

Abstract

The application discloses a GPU resource efficient scheduling method based on multi-scene intelligent interaction, comprising the following steps: step one, generating an interaction request scheduling descriptor based on an intelligent interaction request; step two, constructing a candidate resource action set, and screening an executable candidate resource action set according to a display memory capacity constraint and a concurrency capacity constraint; step three, generating a feature vector; step four, inputting the feature vector into a pre-trained improved NGBoost model to generate a complete time delay condition distribution for the executable candidate resource action set; step five, calculating a default probability; step six, screening the candidate resource action according to an allowed default upper limit, and determining a to-be-executed resource action according to the default probability; and step seven, executing GPU resource configuration and updating tail distribution parameters by using an execution result. The application realizes controllable GPU scheduling risk and improved resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the fields of multi-program control and machine learning technology, and in particular to a method for efficient scheduling of GPU resources based on multi-scenario intelligent interaction. Background Technology

[0002] With the widespread adoption of large-scale AI model inference services, cloud computing platforms, and multi-tenant computing clusters, GPUs have gradually become the core computing resources supporting intelligent interactive services. In practical deployments, the same GPU often needs to simultaneously handle inference requests, training tasks, and interactive computing loads from multiple scenarios. These requests differ significantly in model size, input length, output size, real-time constraints, and interaction stages. Therefore, how to dynamically allocate and schedule GPU resources while meeting latency constraints has become an important research direction in the field of computing systems. In existing technologies, GPU scheduling schemes are typically based on static partitioning rules, fixed concurrency limits, or heuristic load balancing strategies. Some schemes introduce machine learning models to predict task execution latency and select resource allocation methods accordingly to improve overall throughput or reduce timeout risks.

[0003] In multi-scenario intelligent interaction environments, GPU execution latency distribution exhibits a significant long-tail characteristic. Especially under conditions of high-concurrency inference, severe memory fragmentation, and frequent partition reconstruction or time-slice switching, a small number of extreme execution samples often dominate the overall service quality. Existing scheduling methods based on mean or variance estimation are insufficient to characterize such low-frequency, high-impact events. Even when using probabilistic regression models for latency prediction, they typically focus on fitting the distribution within the main intervals, lacking the ability to characterize the risks in high-quantile intervals. This leads to a systematic underestimation of some resource allocation schemes during the scheduling phase, resulting in problems such as sudden queuing amplification, preemption delays at stage boundaries, and interaction deadline violations during actual operation.

[0004] Therefore, how to provide an efficient GPU resource scheduling method based on multi-scenario intelligent interaction is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an efficient GPU resource scheduling method based on multi-scenario intelligent interaction. This invention constructs a unified scheduling modeling framework by integrating interaction request features, GPU status, and resource action parameters. An extreme tail distribution extrapolation layer is introduced inside the NGBoost model to separate and characterize the main interval of execution latency and long-tail risk. Based on the complete conditional distribution, the default probability is calculated to drive resource screening and sorting. At the same time, the tail distribution parameters are continuously updated using execution feedback, so as to achieve controllable GPU scheduling risk and simultaneous improvement of resource utilization efficiency in multi-scenario mixed deployment environments.

[0006] An efficient GPU resource scheduling method based on multi-scenario intelligent interaction according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Generate an interaction request scheduling descriptor based on the intelligent interaction request, and extract scene configuration parameters from the scene configuration table;

[0008] Step 2: Construct a set of candidate resource actions based on the interaction request scheduling descriptor and GPU status information, and filter the set of executable candidate resource actions according to the memory capacity constraint and the concurrency capacity constraint;

[0009] Step 3: Generate a feature vector based on the interaction request scheduling descriptor, GPU status information, and set of executable candidate resource actions;

[0010] Step 4: Input the feature vector into the pre-trained improved NGBoost model. The improved NGBoost model has an extremum tail distribution extrapolation layer to generate a complete delay conditional distribution for the set of executable candidate resource actions.

[0011] Step 5: Calculate the default probability of the corresponding set of executable candidate resource actions based on the complete delay condition distribution;

[0012] Step 6: Compare the default probability with the allowed default limit in the scenario configuration parameters one by one, delete the candidate resource actions whose default probability exceeds the corresponding allowed default limit, and sort the candidate resource actions that have not been deleted according to the default probability, and select the candidate resource action at the top of the sorted sequence as the resource action to be executed.

[0013] Step 7: Configure GPU resources according to the resource actions to be executed, collect the corresponding execution results to form new training samples, and update the extreme value tail distribution extrapolation layer in the improved NGBoost model.

[0014] Optionally, step one is as follows:

[0015] The system receives a request message for intelligent interaction and parses the scene identifier, target model identifier, input size, output limit, interaction deadline, and interaction stage identifier from the request message. The input size is the input data length parameter, the output limit is the output data quantity limit parameter, and the interaction stage identifier is limited to the inference pre-filling stage, the inference decoding stage, or the training iteration stage.

[0016] Encode the scene identifier and the target model identifier, and use a preset mapping table to perform a lookup operation to obtain the scene index and the model index;

[0017] Discrete bucketing is performed on the input size and the output upper limit. The input size and the output upper limit are compared with the preset bucket boundary set one by one, and the bucket number that satisfies the bucket boundary interval is determined as the size index.

[0018] Convert the interaction deadline into a deadline parameter with a uniform time unit, and combine the deadline parameter with the interaction stage identifier to form a stage deadline parameter pair;

[0019] The scene index, model index, scale index and phase deadline parameter are combined according to the preset field order to form an interactive request scheduling descriptor.

[0020] The scenario configuration parameter set is retrieved from the scenario configuration table using the scenario index as the key. The scenario configuration parameter set includes the allowed default upper limit parameter, the risk quantile set parameter, and the preemption boundary type parameter. The preemption boundary type parameter is limited to a predefined boundary type enumeration, which includes the pre-filled end boundary, the decoding step boundary, and the iteration end boundary.

[0021] Optionally, step two is as follows:

[0022] Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, number of concurrent execution requests, computing unit utilization ratio, partition specification identifier, and time slice configuration parameters;

[0023] Read the scene index field, model index field, scale index field and phase deadline parameter field from the interactive request scheduling descriptor, and combine the running status parameters to form a resource construction input record;

[0024] Based on the resource construction input record, enumerate the partition specification candidates, concurrent flow quantity candidates, computing unit quota ratio candidates, time slice length candidates, and preemption boundary type candidates, and arrange each candidate by Cartesian product to generate resource action combinations, forming a candidate resource action set;

[0025] For each combination of resource actions, the corresponding video memory requirement is calculated. The video memory requirement is obtained by adding the unit video memory consumption parameter corresponding to the model index to the cache increment parameter corresponding to the scale index.

[0026] Compare the video memory requirement value with the video memory capacity corresponding to the partition specification identifier one by one, and delete the resource action combination whose video memory requirement value exceeds the corresponding video memory capacity.

[0027] For resource action combinations that have not been deleted, calculate the corresponding concurrent capacity requirement value. The concurrent capacity requirement value is obtained by adding the candidate number of concurrent streams to the current number of concurrent execution requests.

[0028] The concurrent capacity requirement is compared with the preset maximum concurrent threshold one by one. Resource action combinations whose concurrent capacity requirement exceeds the maximum concurrent threshold are deleted, and an executable candidate resource action set is generated.

[0029] Optional, step three is as follows:

[0030] Extract the scenario index field, model index field, scale index field, and phase deadline parameter field based on the interactive request scheduling descriptor;

[0031] Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, video memory fragmentation ratio, number of concurrent execution requests, computing unit occupancy ratio, partition specification identifier, and current time slice configuration parameters. The video memory fragmentation ratio is calculated by the ratio of the largest contiguous available video memory block to the available video memory capacity.

[0032] For each candidate resource action in the candidate resource action set, extract the partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, candidate time slice length parameters, and preemption boundary type parameters;

[0033] The scene index field, model index field, scale index field, stage deadline parameter field, running status parameter, and candidate resource action parameter are concatenated to generate a feature vector according to the preset field order.

[0034] The feature vectors are written to the feature cache and stored using timestamps and GPU identifiers as index keys.

[0035] Optional, step four is as follows:

[0036] The improved NGBoost model includes a gradient boosting regression tree sequence module, a main distribution parameter generation module, and an extreme value tail distribution extrapolation layer;

[0037] The gradient boosting regression tree sequence module performs residual regression fitting on the input feature vectors sequentially and generates intermediate regression representation vectors in each iteration.

[0038] The main body distribution parameter generation module performs parameter mapping processing based on the intermediate regression representation vector to generate a parameter set of the main body time delay distribution, which includes location parameters and scale parameters;

[0039] The extreme value tail distribution extrapolation layer includes a threshold determination unit, an over-threshold sample buffer unit, a tail parameter estimation unit, and a segmented distribution construction unit;

[0040] The threshold determination unit performs quantile inverse calculation based on the main body time delay distribution and determines the time delay value corresponding to the preset high quantile value as the tail distribution threshold.

[0041] The over-threshold sample caching unit selects latency samples that are greater than the tail distribution threshold from the historical training cache, and performs threshold shifting on the selected samples to generate a tail residual sample set.

[0042] The tail parameter estimation unit performs extreme value distribution parameter estimation processing based on the tail residual sample set, generates shape parameters and scale parameters, and writes the generated parameters into the tail distribution parameter buffer.

[0043] The segmented distribution construction unit constructs a segmented probability distribution based on the main delay distribution and the extreme value distribution parameters in the tail distribution parameter buffer. The main delay distribution is used in the interval below the tail distribution threshold, and the extreme value distribution is used in the interval above the tail distribution threshold.

[0044] The segmented distribution construction unit performs cumulative distribution generation processing on the segmented probability distribution, and generates a cumulative distribution function for the feature vector corresponding to each executable candidate resource action. Each cumulative distribution function is written into the scheduling prediction buffer according to the candidate resource action identifier, forming a complete set of delay conditional distributions that corresponds one-to-one with the set of executable candidate resource actions.

[0045] Optional, step five is as follows:

[0046] For each candidate resource action in the set of executable candidate resource actions, read the complete delay conditional distribution generated by the improved NGBoost model for the corresponding feature vector;

[0047] Extract the interaction deadline from the interaction request scheduling descriptor and convert the interaction deadline into a time unit consistent with the full delay condition distribution;

[0048] Locate the segmented interval containing the interaction deadline parameter in the cumulative distribution function corresponding to the complete time delay conditional distribution, and calculate the cumulative probability value corresponding to the interaction deadline parameter using piecewise linear interpolation.

[0049] Subtract one from the cumulative probability value in the unit probability space to obtain the remaining probability that exceeds the interaction deadline parameter, and determine the remaining probability as the default probability corresponding to the candidate resource action.

[0050] Optional, step six specifically includes:

[0051] For each candidate resource action, read the corresponding default probability, and read the allowed default upper limit parameter that matches the interaction request scheduling descriptor from the scenario configuration table;

[0052] Compare the default probability with the allowed default upper limit parameter one by one, and remove the candidate resource action with the default probability greater than the allowed default upper limit parameter from the set of executable candidate resource actions to generate a set of candidate resource actions after constraint filtering.

[0053] After constraint screening, the candidate resource action set is sorted in ascending order according to the default probability value to generate a sorted sequence;

[0054] When there are multiple candidate resource actions with the same default probability in the sorted sequence, parallel resolution processing is performed according to the preset order of the candidate resource action identifiers;

[0055] Select the corresponding candidate resource action from the first position of the sorted sequence and determine it as the resource action to be executed.

[0056] Optionally, step seven is as follows:

[0057] Based on the resource action to be executed, read the corresponding partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, time slice length parameters, and preemption boundary type parameters from the candidate resource action set;

[0058] The GPU partition control interface is called according to the partition specification parameters to perform instance partitioning or instance switching operations, and the concurrent flow limit configuration parameters are set according to the concurrent flow quantity parameters, and the computing unit quota configuration parameters are set according to the computing unit quota ratio parameters.

[0059] The time slice length of the time slice scheduler is set according to the time slice length parameter, and resource switching is triggered at the interaction phase boundary according to the preemption boundary type parameter and the interaction phase identifier.

[0060] After the GPU completes resource configuration, record the timestamp when the configuration takes effect and the timestamp when the interaction request is completed, and calculate the difference between the two to obtain the actual latency;

[0061] The actual latency is combined with the identifier of the resource action to be executed, the GPU identifier, and the scene index field to generate new training samples, and the new training samples are written to the training cache.

[0062] The tail distribution parameter buffer of the extreme tail distribution extrapolation layer in the improved NGBoost model is updated based on the newly added training samples accumulated in the training buffer.

[0063] The beneficial effects of this invention are:

[0064] This invention constructs a unified feature modeling framework that integrates interactive request scheduling descriptors, GPU runtime state parameters, and candidate resource action parameters. Combining an improved NGBoost model with a collaborative design of an extreme tail distribution extrapolation layer, it addresses the issues of long-tailed GPU execution latency distribution, difficulty in characterizing extreme loads, and insufficient assessment of sudden default risk in multi-scenario intelligent interactive environments. It proposes a latency prediction mechanism based on segmented modeling of the main distribution and tail distribution, achieving refined characterization of the risk of different resource configuration schemes in high-quantile intervals. In the resource decision-making stage, it introduces a default probability calculation and constraint screening strategy based on the complete latency condition distribution, comparing allowable... By setting a default cap and ranking the execution probability, a unified risk assessment scale is established among multiple candidate resource actions, effectively suppressing sudden defaults in high-concurrency and frequent reconfiguration scenarios. In the execution feedback phase, a closed-loop update process is constructed with GPU partition control interface, time slice scheduler and stage boundary switching mechanism as the core, continuously feeding back the actual execution latency to the tail distribution parameter buffer, realizing online adaptive capability for extreme risk characterization. Thus, in complex computing environments with cross-scenario mixed deployment, dynamic partitioning and frequent switching of interaction stages, scheduling stability and risk controllability are maintained at the same time, significantly improving GPU resource utilization efficiency and service quality assurance capability of multi-scenario intelligent interaction services. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0066] Figure 1 This is a schematic diagram of the overall process of a GPU resource high-efficiency scheduling method based on multi-scenario intelligent interaction proposed in this invention;

[0067] Figure 2 This is a schematic diagram of the improved NGBoost model and the extreme tail distribution extrapolation layer structure in the GPU resource high-efficiency scheduling method based on multi-scenario intelligent interaction proposed in this invention.

[0068] Figure 3 This is a schematic diagram of the default probability calculation and resource action sorting process in the GPU resource efficient scheduling method based on multi-scenario intelligent interaction proposed in this invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0070] refer to Figures 1-3 A method for efficient GPU resource scheduling based on multi-scenario intelligent interaction includes the following steps:

[0071] Step 1: Generate an interaction request scheduling descriptor based on the intelligent interaction request, and extract scene configuration parameters from the scene configuration table;

[0072] Step 2: Construct a set of candidate resource actions based on the interactive request scheduling descriptor and GPU state information, and filter the executable candidate resource actions according to the memory capacity constraints and concurrency capacity constraints.

[0073] Step 3: Generate feature vectors based on the interactive request scheduling descriptor, GPU state information, and the set of executable candidate resource actions;

[0074] Step 4: Input the feature vector into the pre-trained improved NGBoost model. The improved NGBoost model has an extremum tail distribution extrapolation layer to generate a complete delay conditional distribution for the set of executable candidate resource actions.

[0075] Step 5: Calculate the default probability of the corresponding set of executable candidate resource actions based on the complete delay condition distribution;

[0076] Step 6: Compare the default probability with the allowed default limit in the scenario configuration parameters one by one, delete the candidate resource actions whose default probability exceeds the corresponding allowed default limit, and sort the candidate resource actions that have not been deleted according to the default probability, and select the candidate resource action at the top of the sorted sequence as the resource action to be executed.

[0077] Step 7: Configure GPU resources according to the resource actions to be executed, collect the corresponding execution results to form new training samples, and update the extreme value tail distribution extrapolation layer in the improved NGBoost model.

[0078] In this embodiment, step one specifically includes:

[0079] The system receives a request message for intelligent interaction and parses the scene identifier, target model identifier, input size, output limit, interaction deadline, and interaction stage identifier from the request message. The input size is the input data length parameter, the output limit is the output data quantity limit parameter, and the interaction stage identifier is limited to the inference pre-filling stage, the inference decoding stage, or the training iteration stage.

[0080] Encode the scene identifier and the target model identifier, and use a preset mapping table to perform a lookup operation to obtain the scene index and the model index;

[0081] Discrete bucketing is performed on the input size and the output upper limit. The input size and the output upper limit are compared with the preset bucket boundary set one by one, and the bucket number that satisfies the bucket boundary interval is determined as the size index.

[0082] Convert the interaction deadline into a deadline parameter with a uniform time unit, and combine the deadline parameter with the interaction stage identifier to form a stage deadline parameter pair;

[0083] The scene index, model index, scale index and phase deadline parameter are combined according to the preset field order to form an interactive request scheduling descriptor.

[0084] The scenario configuration parameter set is retrieved from the scenario configuration table using the scenario index as the key. The scenario configuration parameter set includes the allowed default upper limit parameter, the risk quantile set parameter, and the preemption boundary type parameter. The preemption boundary type parameter is limited to a predefined boundary type enumeration, which includes the pre-filled end boundary, the decoding step boundary, and the iteration end boundary.

[0085] In this implementation, the scene index and model index are generated through a preset mapping table, which is stored in the configuration database using integer numbers. The input size bucket boundaries adopt a fixed interval division method, set into four levels according to the input length range: 0-512, 513-2048, 2049-8192, and above 8193. The output upper limit bucket adopts three levels: 0-128, 129-512, and 513-2048. The interaction deadline is uniformly converted to milliseconds. The allowed default limit adopts three discrete values: 0.01, 0.03, and 0.05. The risk quantile set adopts 0.95 and 0.99. The preemption boundary type adopts enumeration values ​​1, 2, and 3, corresponding to the pre-filled end boundary, decoding step boundary, and iteration end boundary, respectively. The scene configuration table is stored in an in-memory key-value database and accessed by the scene index.

[0086] In this embodiment, step two specifically includes:

[0087] Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, number of concurrent execution requests, computing unit utilization ratio, partition specification identifier, and time slice configuration parameters;

[0088] Read the scene index field, model index field, scale index field and phase deadline parameter field from the interactive request scheduling descriptor, and combine the running status parameters to form a resource construction input record;

[0089] Based on the resource construction input record, enumerate the partition specification candidates, concurrent flow quantity candidates, computing unit quota ratio candidates, time slice length candidates, and preemption boundary type candidates, and arrange each candidate by Cartesian product to generate resource action combinations, forming a candidate resource action set;

[0090] For each combination of resource actions, the corresponding video memory requirement is calculated. The video memory requirement is obtained by adding the unit video memory consumption parameter corresponding to the model index to the cache increment parameter corresponding to the scale index.

[0091] Compare the video memory requirement value with the video memory capacity corresponding to the partition specification identifier one by one, and delete the resource action combination whose video memory requirement value exceeds the corresponding video memory capacity.

[0092] For resource action combinations that have not been deleted, calculate the corresponding concurrent capacity requirement value. The concurrent capacity requirement value is obtained by adding the candidate number of concurrent streams to the current number of concurrent execution requests.

[0093] The concurrent capacity requirement is compared with the preset maximum concurrent threshold one by one. Resource action combinations whose concurrent capacity requirement exceeds the maximum concurrent threshold are deleted, and an executable candidate resource action set is generated.

[0094] In this implementation, the partition specification candidates are derived from the GPU physical partitioning configuration table or the multi-instance GPU configuration table; the concurrent stream quantity candidates and the computing unit quota ratio candidates are derived from the scheduling policy parameter table; and the time slice length candidates and the preemption boundary type candidates are derived from the scene configuration parameter set. The unit computing memory consumption parameter establishes the correspondence between the model index and the single-step computing memory usage through offline benchmark testing. The cache increment parameter is obtained by statistically analyzing the growth curves of key-value cache or active cache under different index sizes. The preset maximum concurrency threshold is jointly determined by the GPU architecture specification and driver concurrency limits and stored in the resource constraint table, which is periodically refreshed by the scheduler during operation.

[0095] In this embodiment, step three specifically includes:

[0096] Extract the scenario index field, model index field, scale index field, and phase deadline parameter field based on the interactive request scheduling descriptor;

[0097] Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, video memory fragmentation ratio, number of concurrent execution requests, computing unit occupancy ratio, partition specification identifier, and current time slice configuration parameters. The video memory fragmentation ratio is calculated by the ratio of the largest contiguous available video memory block to the available video memory capacity.

[0098] For each candidate resource action in the candidate resource action set, extract the partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, candidate time slice length parameters, and preemption boundary type parameters;

[0099] The scene index field, model index field, scale index field, stage deadline parameter field, running status parameter, and candidate resource action parameter are concatenated to generate a feature vector according to the preset field order.

[0100] The feature vectors are written to the feature cache and stored using timestamps and GPU identifiers as index keys.

[0101] In this implementation, the preset field order is fixed as follows: scene index field, model index field, scale index field, stage deadline parameter field, allocated video memory capacity field, available video memory capacity field, video memory fragmentation ratio field, number of concurrent execution requests field, computing unit occupancy ratio field, partition specification parameter field, number of concurrent streams parameter field, computing unit quota ratio parameter field, time slice length parameter field, and preemption boundary type parameter field. Each field is uniformly stored in integer or floating-point encoding form. The partition specification parameter is mapped to continuous values ​​through a lookup table. The number of concurrent streams parameter and computing unit quota ratio parameter are normalized. The stage deadline parameter is converted into the remaining time. The feature cache adopts a circular buffer structure and is stored in partitions according to GPU identifier.

[0102] In this embodiment, step four specifically includes:

[0103] The improved NGBoost model includes a gradient boosting regression tree sequence module, a main distribution parameter generation module, and an extreme value tail distribution extrapolation layer;

[0104] The gradient boosting regression tree sequence module performs residual regression fitting on the input feature vectors sequentially and generates intermediate regression representation vectors in each iteration.

[0105] The main body distribution parameter generation module performs parameter mapping processing based on the intermediate regression representation vector to generate a parameter set of the main body time delay distribution, which includes location parameters and scale parameters;

[0106] The extreme value tail distribution extrapolation layer includes a threshold determination unit, an over-threshold sample buffer unit, a tail parameter estimation unit, and a segmented distribution construction unit;

[0107] The threshold determination unit performs quantile inverse calculation based on the main body time delay distribution and determines the time delay value corresponding to the preset high quantile value as the tail distribution threshold.

[0108] The over-threshold sample caching unit selects latency samples that are greater than the tail distribution threshold from the historical training cache, and performs threshold shifting on the selected samples to generate a tail residual sample set.

[0109] The tail parameter estimation unit performs extreme value distribution parameter estimation processing based on the tail residual sample set, generates shape parameters and scale parameters, and writes the generated parameters into the tail distribution parameter buffer.

[0110] The segmented distribution construction unit constructs a segmented probability distribution based on the main delay distribution and the extreme value distribution parameters in the tail distribution parameter buffer. The main delay distribution is used in the interval below the tail distribution threshold, and the extreme value distribution is used in the interval above the tail distribution threshold.

[0111] The segmented distribution construction unit performs cumulative distribution generation processing on the segmented probability distribution, and generates a cumulative distribution function for the feature vector corresponding to each executable candidate resource action. Each cumulative distribution function is written into the scheduling prediction buffer according to the candidate resource action identifier, forming a complete set of delay conditional distributions that corresponds one-to-one with the set of executable candidate resource actions.

[0112] In this implementation, the high quantile threshold is determined using a fixed quantile ratio. The corresponding delay value is obtained by performing an inverse operation on the cumulative distribution function of the main distribution of the improved NGBoost model. The quantile ratio is set to 0.90, 0.95, or 0.97 according to the scene type and stored in the scene configuration table for the scheduling module to call. The tail distribution parameters are updated periodically using a sliding time window method. The window length is set to the most recent 3000 to 20000 completed request records. When the cumulative number of newly added over-threshold samples reaches the preset update threshold, the parameter re-estimation process is triggered. The re-estimation process is only performed on the shape and scale parameters and does not change the gradient boosting regression tree structure, thereby maintaining the stability of the main model and simultaneously correcting the ability to characterize extreme risks when the load changes.

[0113] In GPU scheduling environments with multi-scenario intelligent interaction, request load and execution latency exhibit significant long-tail distribution characteristics. When high concurrency is coupled with memory fragmentation, context switching, and partition adjustments, a small number of extreme samples often determine whether a service level violation is triggered. The conventional NGBoost model obtains the main latency distribution parameters by performing a uniform probability fitting on the overall sample distribution. This approach tends to have high-frequency, low-latency samples dominate parameter updates when the number of low-frequency extreme samples is limited, resulting in a systematic underestimation in the high quantile range. This leads to an underestimation of the risk of some resource actions during the scheduling phase, resulting in phenomena such as sudden queuing amplification, preemption chain reactions, and deadline violations.

[0114] The improved NGBoost model retains the original natural gradient boosting idea in its overall framework, uses gradient boosting regression tree sequences as base learners, generates intermediate regression representations through round-by-round residual fitting, and outputs the subject time delay distribution parameters in the form of parameterized probability distribution. At the same time, it maintains support for mixed inputs of continuous and discrete features and the training process of log-likelihood scoring criteria, so that the subject interval modeling ability and convergence characteristics are consistent with the conventional NGBoost model.

[0115] This invention introduces an extreme tail distribution extrapolation layer within the NGBoost model, separating the main distribution modeling from the tail distribution modeling. By establishing independent extreme distribution parameters for samples exceeding the threshold in the range above the high quantile threshold, the model maintains the stability of the main interval fitting while specifically characterizing sudden latency spikes in high-concurrency scenarios. This structure avoids the problem of extreme samples being masked by the smoothing of the main distribution, ensuring that each candidate resource action obtains a complete conditional distribution including extreme risks during the prediction stage. This allows for the early elimination of allocation schemes with potentially high default probabilities during the action selection stage. In actual operation, this improvement maintains stable tail risk estimation even under scenarios with sudden load increases, mixed inference, and frequent switching of dynamic partitions, significantly reducing the proportion of sudden defaults during cross-scenario scheduling and achieving a scheduling security improvement effect that exceeds the expectations of conventional probabilistic regression models.

[0116] In this embodiment, step five specifically includes:

[0117] For each candidate resource action in the set of executable candidate resource actions, read the complete delay conditional distribution generated by the improved NGBoost model for the corresponding feature vector;

[0118] Extract the interaction deadline from the interaction request scheduling descriptor and convert the interaction deadline into a time unit consistent with the full delay condition distribution;

[0119] Locate the segmented interval containing the interaction deadline parameter in the cumulative distribution function corresponding to the complete time delay conditional distribution, and calculate the cumulative probability value corresponding to the interaction deadline parameter using piecewise linear interpolation.

[0120] Subtract one from the cumulative probability value in the unit probability space to obtain the remaining probability that exceeds the interaction deadline parameter, and determine the remaining probability as the default probability corresponding to the candidate resource action.

[0121] In this implementation, the cumulative distribution function is stored in the scheduling prediction buffer as a discrete sampling point array. Each sampling point corresponds to a cumulative probability value at a different time scale. The time scale resolution is set to the range of 1 millisecond to 10 milliseconds according to the scene type. Piecewise linear interpolation is performed between two adjacent sampling points, with the time scale difference as the interpolation interval width. The interaction deadline parameter is synchronized and corrected with the GPU driver time base before entering the calculation. The risk assessment table adopts a columnar storage structure and uses the GPU identifier, candidate resource action identifier, and scene index as a joint index key to support parallel access and fast sorting in high-frequency scheduling scenarios.

[0122] In this embodiment, step six specifically includes:

[0123] For each candidate resource action, read the corresponding default probability, and read the allowed default upper limit parameter that matches the interaction request scheduling descriptor from the scenario configuration table;

[0124] Compare the default probability with the allowed default upper limit parameter one by one, and remove the candidate resource action with the default probability greater than the allowed default upper limit parameter from the set of executable candidate resource actions to generate a set of candidate resource actions after constraint filtering.

[0125] After constraint screening, the candidate resource action set is sorted in ascending order according to the default probability value to generate a sorted sequence;

[0126] When there are multiple candidate resource actions with the same default probability in the sorted sequence, parallel resolution processing is performed according to the preset order of the candidate resource action identifiers;

[0127] Select the corresponding candidate resource action from the first position of the sorted sequence and determine it as the resource action to be executed.

[0128] In this implementation, the upper limit of default is allowed to be stored in the scenario configuration table in a discrete configuration manner, and is set to three levels of values: 0.01, 0.03, and 0.05 according to the scenario index; the default probability is uniformly represented as a floating-point number in the range of 0 to 1 and retains 6 decimal places; the sorting sequence is implemented using the quicksort algorithm; when the difference in default probability is less than 0.0001, it is judged as the same value and parallel resolution is triggered; the candidate resource action identifier is encoded with a 32-bit integer; the scheduling decision record table adopts a relational storage structure and uses the GPU identifier, scenario index, and timestamp as a joint index key to support atomic writes and fast queries in high-concurrency scheduling scenarios.

[0129] In this embodiment, step seven specifically includes:

[0130] Based on the resource action to be executed, read the corresponding partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, time slice length parameters, and preemption boundary type parameters from the candidate resource action set;

[0131] The GPU partition control interface is called according to the partition specification parameters to perform instance partitioning or instance switching operations, and the concurrent flow limit configuration parameters are set according to the concurrent flow quantity parameters, and the computing unit quota configuration parameters are set according to the computing unit quota ratio parameters.

[0132] The time slice length of the time slice scheduler is set according to the time slice length parameter, and resource switching is triggered at the interaction phase boundary according to the preemption boundary type parameter and the interaction phase identifier.

[0133] After the GPU completes resource configuration, record the timestamp when the configuration takes effect and the timestamp when the interaction request is completed, and calculate the difference between the two to obtain the actual latency;

[0134] The actual latency is combined with the identifier of the resource action to be executed, the GPU identifier, and the scene index field to generate new training samples, and the new training samples are written to the training cache.

[0135] The tail distribution parameter buffer of the extreme tail distribution extrapolation layer in the improved NGBoost model is updated based on the newly added training samples accumulated in the training buffer.

[0136] In this implementation, the GPU partition control interface calls the multi-instance partitioning command through the driver layer management module to complete instance reconstruction. The concurrency flow limit and computing unit quota parameters are written through the device control register or runtime configuration table. The time slice scheduler is triggered periodically by the kernel-level scheduling thread, and the time slice length is set in milliseconds. The stage boundary signal is generated by the inference engine when pre-filling ends or the decoding step is completed. The training cache is stored in a log structure with GPU partitions and written in timestamp order. The tail distribution parameter buffer is maintained in an independent memory area and refreshed using a sliding window mechanism.

[0137] Example 1: To verify the feasibility and technical effectiveness of this invention in a real-world engineering environment, the proposed efficient GPU resource scheduling method based on multi-scenario intelligent interaction was deployed in a large-scale inference service cluster of a cloud computing center. This cluster consists of 64 data center-grade GPUs, each with 80GB of video memory. It supports multi-instance GPU (MIG) partitioning and time-slice scheduling mechanisms. The cluster simultaneously runs multiple intelligent interaction service scenarios, including long text dialogue generation, code completion, document question answering, and offline fine-tuning training tasks. Each service differs significantly in input length, output scale, interaction stage, and latency constraints. Specifically, the dialogue generation service requires 99% of requests to return the first token within 800ms, the code completion service allows 95% of requests to complete within 1200ms, and the training task uses a more lenient deadline but exhibits significant fluctuations in video memory usage.

[0138] Before adopting the method of this invention, the cluster used a scheduling method based on a fixed concurrency limit and static MIG splitting rules, selecting resource configuration schemes only based on the current GPU utilization and queue length. When the system load is high or multiple scenarios are deployed in a mixed manner, some GPUs frequently undergo instance reconstruction and time slice switching, resulting in abnormally long execution times for a small number of requests. Especially during the decoding stage and when the memory fragmentation rate exceeds 20%, the actual tail latency is significantly higher than the scheduling system's prediction, thus causing sudden deadline defaults.

[0139] In this embodiment, the scheduling system is connected to the method module of the present invention. When an intelligent interaction request arrives, it first parses the request message to generate an interaction request scheduling descriptor, maps the scene type to an integer index, numbers the target model as a model index, and discretizes the input length and output upper limit according to a preset bucket interval. At the same time, it reads the corresponding allowed default upper limit, risk quantile parameter, and preemption boundary type from the scene configuration table. Subsequently, the scheduling system collects the current GPU memory usage, fragmentation ratio, number of concurrent execution requests, computing unit utilization, and time slice status. It enumerates different MIG specifications, concurrent stream counts, and time slice length combinations to form a resource action set, and filters out the executable candidate resource action set under memory capacity constraints and concurrency threshold constraints.

[0140] For each candidate resource action, the system constructs a feature vector containing a scene index, model index, scale index, stage deadline parameter, GPU state parameter, and resource action parameter, and inputs the feature vector into a pre-trained improved NGBoost model. Based on the main distribution prediction, this model independently models historical out-of-threshold latency samples through an internal extreme tail distribution extrapolation layer, thereby generating a complete latency conditional distribution for each candidate resource action. The scheduling system further performs interpolation operations in the cumulative distribution function based on the interaction deadline, calculates the remaining probability exceeding the deadline, and writes this probability as the default probability into the risk assessment table. Subsequently, resource actions with excessively high default probabilities are filtered one by one according to the allowed default upper limit set in the scene configuration, and the remaining actions are sorted in ascending order of default probability. The action with the highest default probability is selected to perform GPU partition adjustment, concurrent flow configuration, and time slice setting. After the request is executed, the system records the actual execution latency and writes it back to the training cache, only updating the tail distribution parameters in the extreme tail distribution extrapolation layer, thus continuously adapting to sudden load changes without compromising the stability of the main prediction.

[0141] To verify the beneficial effects of this invention, it was run continuously for 30 days under real production load. The results were compared with the original scheduling strategy and a scheduling strategy using only the conventional NGBoost probabilistic prediction model. The tail default rate, average GPU utilization, and P99 latency performance of the three schemes under mixed-load scenarios were statistically analyzed. Approximately 84 million request records were collected during the experiment, with high-concurrency windows accounting for about 27%. The experimental results are shown in Table 1.

[0142] Table 1. Performance Comparison of Different Scheduling Strategies under Mixed Load in Multiple Scenarios

[0143] Comparison indicators Static rule scheduling Conventional NGBoost scheduling Method of the present invention Deadline default rate (%) 3.84 2.17 0.62 P99 latency (ms) 2140 1560 980 Average GPU utilization (%) 71.2 74.9 82.6 Instance refactoring frequency (times / hour) 18.6 14.2 7.4 Average memory fragmentation rate (%) 24.3 21.5 15.1

[0144] As shown in Table 1, under high-load mixed scenarios, the method of this invention reduces the overall deadline default rate from 3.84% to 0.62%, a decrease of approximately 71% compared to the conventional NGBoost solution alone. Simultaneously, P99 latency decreases from 1560ms to 980ms, with significant convergence of extreme tail latency. Regarding GPU utilization, the method of this invention, while strictly controlling risk, still improves the average utilization to 82.6%, significantly higher than the baseline solution. Because the scheduling system can identify high-risk resource actions in advance and avoid frequent instance reconstructions, the method of this invention reduces the number of GPU partition reconstructions per unit time by approximately 60%, and the memory fragmentation rate remains at a low level. Further analysis shows that in a stress test with a 2x increase in sudden concurrency, the default rate of the original scheduling system instantly rises to 6.3%, the conventional NGBoost solution rises to 3.9%, while the method of this invention consistently remains below 1%, verifying the effectiveness of the extreme tail distribution extrapolation layer in characterizing extreme execution latency.

[0145] As can be seen from the above embodiments, the present invention can significantly improve the efficiency of GPU resource utilization while ensuring service level constraints in a real cloud multi-scenario intelligent interactive environment, and maintain a stable predictive ability for long-tail risks through a continuous feedback mechanism, thus having good engineering feasibility and significant practical application value.

[0146] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for efficient GPU resource scheduling based on multi-scenario intelligent interaction, characterized in that, Includes the following steps: Step 1: Generate an interaction request scheduling descriptor based on the intelligent interaction request, and extract scene configuration parameters from the scene configuration table; Step 2: Construct a set of candidate resource actions based on the interaction request scheduling descriptor and GPU status information, and filter the set of executable candidate resource actions according to the memory capacity constraint and the concurrency capacity constraint; Step 3: Generate a feature vector based on the interaction request scheduling descriptor, GPU status information, and set of executable candidate resource actions; Step 4: Input the feature vector into the pre-trained improved NGBoost model. The improved NGBoost model has an extremum tail distribution extrapolation layer to generate a complete delay conditional distribution for the set of executable candidate resource actions. Step 5: Calculate the default probability of the corresponding set of executable candidate resource actions based on the complete delay condition distribution; Step 6: Compare the default probability with the allowed default limit in the scenario configuration parameters one by one, delete the candidate resource actions whose default probability exceeds the corresponding allowed default limit, and sort the candidate resource actions that have not been deleted according to the default probability, and select the candidate resource action at the top of the sorted sequence as the resource action to be executed. Step 7: Configure GPU resources according to the resource actions to be executed, collect the corresponding execution results to form new training samples, and update the tail distribution parameter buffer of the extreme tail distribution extrapolation layer in the improved NGBoost model.

2. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step one specifically involves: The system receives a request message for intelligent interaction and parses the scene identifier, target model identifier, input size, output limit, interaction deadline, and interaction stage identifier from the request message. The input size is the input data length parameter, the output limit is the output data quantity limit parameter, and the interaction stage identifier is limited to the inference pre-filling stage, the inference decoding stage, or the training iteration stage. Encode the scene identifier and the target model identifier, and use a preset mapping table to perform a lookup operation to obtain the scene index and the model index; Discrete bucketing is performed on the input size and the output upper limit. The input size and the output upper limit are compared with the preset bucket boundary set one by one, and the bucket number that satisfies the bucket boundary interval is determined as the size index. Convert the interaction deadline into a deadline parameter with a uniform time unit, and combine the deadline parameter with the interaction stage identifier to form a stage deadline parameter pair; The scene index, model index, scale index and phase deadline parameter are combined according to the preset field order to form an interactive request scheduling descriptor. The scenario configuration parameter set is retrieved from the scenario configuration table using the scenario index as the key. The scenario configuration parameter set includes the allowed default upper limit parameter, the risk quantile set parameter, and the preemption boundary type parameter. The preemption boundary type parameter is limited to a predefined boundary type enumeration, which includes the pre-filled end boundary, the decoding step boundary, and the iteration end boundary.

3. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step two specifically involves: Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, number of concurrent execution requests, computing unit utilization ratio, partition specification identifier, and time slice configuration parameters; Read the scene index field, model index field, scale index field and phase deadline parameter field from the interactive request scheduling descriptor, and combine the running status parameters to form a resource construction input record; Based on the resource construction input record, enumerate the partition specification candidates, concurrent flow quantity candidates, computing unit quota ratio candidates, time slice length candidates, and preemption boundary type candidates, and arrange each candidate by Cartesian product to generate resource action combinations, forming a candidate resource action set; For each combination of resource actions, the corresponding video memory requirement is calculated. The video memory requirement is obtained by adding the unit video memory consumption parameter corresponding to the model index to the cache increment parameter corresponding to the scale index. Compare the video memory requirement value with the video memory capacity corresponding to the partition specification identifier one by one, and delete the resource action combination whose video memory requirement value exceeds the corresponding video memory capacity. For resource action combinations that have not been deleted, calculate the corresponding concurrent capacity requirement value. The concurrent capacity requirement value is obtained by adding the candidate number of concurrent streams to the current number of concurrent execution requests. The concurrent capacity requirement is compared with the preset maximum concurrent threshold one by one. Resource action combinations whose concurrent capacity requirement exceeds the maximum concurrent threshold are deleted, and an executable candidate resource action set is generated.

4. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step three specifically involves: Extract the scenario index field, model index field, scale index field, and phase deadline parameter field based on the interactive request scheduling descriptor; Collect the current running status parameters of the GPU, including allocated video memory capacity, available video memory capacity, video memory fragmentation ratio, number of concurrent execution requests, computing unit occupancy ratio, partition specification identifier, and current time slice configuration parameters. The video memory fragmentation ratio is calculated by the ratio of the largest contiguous available video memory block to the available video memory capacity. For each candidate resource action in the candidate resource action set, extract the partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, candidate time slice length parameters, and preemption boundary type parameters; The scene index field, model index field, scale index field, stage deadline parameter field, running status parameter, and candidate resource action parameter are concatenated to generate a feature vector according to the preset field order. The feature vectors are written to the feature cache and stored using timestamps and GPU identifiers as index keys.

5. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step four specifically involves: The improved NGBoost model includes a gradient boosting regression tree sequence module, a main distribution parameter generation module, and an extreme value tail distribution extrapolation layer; The gradient boosting regression tree sequence module performs residual regression fitting on the input feature vectors sequentially and generates intermediate regression representation vectors in each iteration. The main body distribution parameter generation module performs parameter mapping processing based on the intermediate regression representation vector to generate a parameter set of the main body time delay distribution, which includes location parameters and scale parameters; The extreme value tail distribution extrapolation layer includes a threshold determination unit, an over-threshold sample buffer unit, a tail parameter estimation unit, and a segmented distribution construction unit; The threshold determination unit performs quantile inverse calculation based on the main body time delay distribution and determines the time delay value corresponding to the preset high quantile value as the tail distribution threshold. The over-threshold sample caching unit selects latency samples that are greater than the tail distribution threshold from the historical training cache, and performs threshold shifting on the selected samples to generate a tail residual sample set. The tail parameter estimation unit performs extreme value distribution parameter estimation processing based on the tail residual sample set, generates shape parameters and scale parameters, and writes the generated parameters into the tail distribution parameter buffer. The segmented distribution construction unit constructs a segmented probability distribution based on the main delay distribution and the extreme value distribution parameters in the tail distribution parameter buffer. The main delay distribution is used in the interval below the tail distribution threshold, and the extreme value distribution is used in the interval above the tail distribution threshold. The segmented distribution construction unit performs cumulative distribution generation processing on the segmented probability distribution, and generates a cumulative distribution function for the feature vector corresponding to each executable candidate resource action. Each cumulative distribution function is written into the scheduling prediction buffer according to the candidate resource action identifier, forming a complete set of delay conditional distributions that corresponds one-to-one with the set of executable candidate resource actions.

6. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step five specifically involves: For each candidate resource action in the set of executable candidate resource actions, read the complete delay conditional distribution generated by the improved NGBoost model for the corresponding feature vector; Extract the interaction deadline from the interaction request scheduling descriptor and convert the interaction deadline into a time unit consistent with the full delay condition distribution; Locate the segmented interval containing the interaction deadline parameter in the cumulative distribution function corresponding to the complete time delay conditional distribution, and calculate the cumulative probability value corresponding to the interaction deadline parameter using piecewise linear interpolation. Subtract one from the cumulative probability value in the unit probability space to obtain the remaining probability that exceeds the interaction deadline parameter, and determine the remaining probability as the default probability corresponding to the candidate resource action.

7. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step six specifically involves: For each candidate resource action, read the corresponding default probability, and read the allowed default upper limit parameter that matches the interaction request scheduling descriptor from the scenario configuration table; Compare the default probability with the allowed default upper limit parameter one by one, and remove the candidate resource action with the default probability greater than the allowed default upper limit parameter from the set of executable candidate resource actions to generate a set of candidate resource actions after constraint filtering. After constraint screening, the candidate resource action set is sorted in ascending order according to the default probability value to generate a sorted sequence; When there are multiple candidate resource actions with the same default probability in the sorted sequence, parallel resolution processing is performed according to the preset order of the candidate resource action identifiers; Select the corresponding candidate resource action from the first position of the sorted sequence and determine it as the resource action to be executed.

8. The method for efficient GPU resource scheduling based on multi-scenario intelligent interaction according to claim 1, characterized in that, Step seven specifically involves: Based on the resource action to be executed, read the corresponding partition specification parameters, concurrent flow quantity parameters, computing unit quota ratio parameters, time slice length parameters, and preemption boundary type parameters from the candidate resource action set; The GPU partition control interface is called according to the partition specification parameters to perform instance partitioning or instance switching operations, and the concurrent flow limit configuration parameters are set according to the concurrent flow quantity parameters, and the computing unit quota configuration parameters are set according to the computing unit quota ratio parameters. The time slice length of the time slice scheduler is set according to the time slice length parameter, and resource switching is triggered at the interaction phase boundary according to the preemption boundary type parameter and the interaction phase identifier. After the GPU completes resource configuration, record the timestamp when the configuration takes effect and the timestamp when the interaction request is completed, and calculate the difference between the two to obtain the actual latency; The actual latency is combined with the identifier of the resource action to be executed, the GPU identifier, and the scene index field to generate new training samples, and the new training samples are written to the training cache. The tail distribution parameter buffer of the extreme tail distribution extrapolation layer in the improved NGBoost model is updated based on the newly added training samples accumulated in the training buffer.

Images