Display card resource load balancing control method and system

Abstract

The invention relates to the technical field of graphics card resource scheduling, and discloses a graphics card resource load balancing control method and system. The method comprises the following steps: constructing a sample set containing actual GPU video memory occupation and utilization rate; constructing a GPU video memory prediction function and a GPU utilization rate prediction function; the schedulability of the current task is judged through a soft constraint matching mechanism; gPU target allocation for executing the task; and carrying out dynamic scheduling by adopting a multi-stage KV cache block recovery mechanism. In the prior art, only static configuration or a single index is adopted to carry out GPU load estimation, and especially in the scene of large Token length uncertainty and obvious KV cache dynamic expansion in an LLM class model reasoning stage, accurate video memory resource scheduling control is difficult to realize. Due to the fact that the multivariable prediction model and the hybrid priority scheduling mechanism are introduced, the load stability of the GPU in the large language model reasoning task is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graphics card resource scheduling, and particularly relates to a graphics card resource load balancing control method and system. BACKGROUND

[0002] At present, with the successful application of the Transformer architecture in natural language processing, code generation, automatic question answering and other tasks, large language models (LLM) such as GPT, BERT, OPT, LLaMA and the like are constantly emerging, and the model parameter scale has increased from hundreds of millions to tens of billions or even trillions. At the inference deployment level, in order to meet the service needs of high throughput and low delay, the mainstream deployment form adopts the single-machine multi-GPU parallel inference mode, and improves the graphics card utilization rate through Batch merging, streaming execution and GPU multi-task concurrent scheduling. In actual deployment, user requests often show obvious uncertainty characteristics, especially the Token sequence length fluctuation range is large, the Batch request granularity changes frequently, and the response window is difficult to accurately estimate. However, the large language model inference process itself involves a large KV cache generation and storage overhead, which makes the GPU memory usage not only related to the model parameters, but also sensitive to the input Token length and Batch Size, and changes nonlinearly. Taking LLaMA-13B as an example, its inference memory occupancy can increase from about 12GB of Batch=4, Token=512 to more than 30GB of Batch=16, Token=2048, significantly increasing the resource scheduling difficulty.

[0003] In the prior art, GPU task scheduling often adopts polling, static configuration, and estimated Batch upper limit to allocate resources, and mostly only considers a single factor such as Batch Size for estimation, ignoring the combined effect of Token length changes on memory utilization. Such static or empirical methods can maintain basic usability when small-scale models are running, but in the inference service scenario of large model high-concurrency deployment and frequent task input changes, there are obvious deficiencies. For example, in the LLaMA model running, if the Batch is 8 and the Token length changes from 512 to 2048, the actual memory peak value may jump from 16GB to 28GB, far exceeding the static estimated threshold, resulting in frequent OOM (Out of Memory) or resource vacancy phenomena.

[0004] Therefore, there is an urgent need for a graphics card resource load balancing control method to achieve fine allocation control of graphics card resources in the case of dynamic changes in Token length, uneven Batch requests, and uncertain KV cache occupancy, so as to improve the efficiency and stability of graphics card resource utilization. SUMMARY

[0005] In view of the above technical deficiencies, the purpose of the present application is to provide a control method for load balancing of graphics card resources, aiming to solve the technical problem that in the prior art, only static configuration or a single index is used for GPU load estimation, especially in the scene where Token length uncertainty is large and KV cache dynamic expansion is obvious during LLM model inference stage, it is difficult to realize accurate graphics memory resource scheduling control.

[0006] To solve the above technical problems, the present application adopts the following technical solutions: the present application provides a control method for load balancing of graphics card resources, The control method for load balancing of graphics card resources comprises: Step S10: using NVIDIA-SMI to collect batch inference request input parameters during LLM inference runtime, the batch inference request input parameters including batch size B, Token sequence length , GPU occupancy and GPU utilization ; based on the batch inference request input parameters, a dynamic sample collection and modeling mechanism with supervised error correction is used to generate a triple sample set ; Step S20: based on the triple sample set , a multi-dimensional regression mechanism based on structure perception feature construction is used to perform GPU load prediction task, and GPU memory prediction function and GPU utilization prediction function are outputted; Step S30: based on the GPU memory prediction function and the GPU utilization prediction function , a multi-objective soft constraint matching mechanism is used to perform task schedulability determination task, and a schedulability judgment matrix is outputted ; Step S40: according to the schedulability judgment matrix , a hybrid priority scheduling mechanism based on fluctuation buffer window is used to perform inference task allocation control, and a task allocation target matrix is outputted; Step S50: using the task allocation target matrix, a multi-level KV cache block recycling method is used for dynamic memory management.

[0007] Preferably, in step S10, the step of generating a triple sample set based on the batch inference request input parameters using a dynamic sample collection and modeling mechanism with supervised error correction comprises: First, perform real-time sampling and data archiving task: during LLM inference runtime, based on batch inference request input parameters, batch size B, Token sequence length , GPU occupancy and GPU utilization normalization and structured storage, and output an initial sample data structure set; error feedback and dynamic labeling tasks: after the LLM inference is completed, the prediction sample data structure set output by the pre-deployed GPU prediction lightweight model is obtained, and the weighted mean square error method is used to calculate the sample error set according to the prediction sample data structure set and the initial sample data structure set; and the initial sample data structure set is labeled with the sample error set as the weight to form a dynamic sample set with supervised error feedback; Finally, perform the partition pre-training task: according to the dynamic sample set with supervised error feedback and the Token sequence length Perform partition gradient boosting regression training using the scikit-learn library of Python, output the regression sub-tree set of the dynamic sample set with supervised error feedback; finally, based on the regression sub-tree set, use the joblib library of Python to serialize and save, and generate a triple sample set .

[0008] Preferably, in step S10, the triple sample set The formula is expressed as: n is the total number of samples in the triple sample set, is the batch size of the i-th sample in the triple sample set, is the Token sequence length of the i-th sample in the triple sample set, is the GPU occupancy rate of the i-th sample in the triple sample set , is the GPU utilization rate of the i-th sample in the triple sample set.

[0009] Preferably, in step S20, based on the triple sample set Perform GPU load prediction tasks using a multi-dimensional regression mechanism based on structure-aware features, and output GPU memory prediction function and GPU utilization prediction function The steps include: Step S201: based on the triple sample set Construct a structure-aware feature set F, which includes structure-aware basic features, structure-aware combined features and structure-aware auxiliary features; based on the structure-aware basic features, structure-aware combined features and structure-aware auxiliary features, construct a structure-aware feature matrix The structure-aware feature matrix is used to enhance the sensitivity of the model to changes in the input space structure; Step S202: based on the structure-aware feature matrix The first regression sub-model and the second regression sub-model are respectively constructed for the two prediction targets of GPU occupancy and GPU utilization; wherein, the first regression sub-model is constructed by using ridge regression; and the second regression sub-model is constructed by using Huber regression method; Step S203: performing error verification by using a residual error analysis method based on K-fold cross-validation for the first regression sub-model, constructing and outputting a GPU memory prediction function ; performing generalization verification by using a robustness evaluation method based on residual error distribution consistency for the second regression sub-model, constructing and outputting a GPU utilization prediction function .

[0010] Preferably, in step S20, the perception-based feature includes a batch size feature and a Token sequence length feature; the structure-perception combined feature includes a batch and sequence coupling feature, a unit Token load feature, a batch smooth logarithm feature, and a sequence length logarithm feature; and the structure-perception auxiliary feature includes a request type marker feature.

[0011] Preferably, in step S30, based on the GPU memory prediction function and the GPU utilization prediction function , a multi-objective soft constraint matching mechanism is used to perform a task schedulability judgment task, and a schedulability judgment matrix is output . Specifically, the step includes: Step S301: based on the GPU memory prediction function and the GPU utilization prediction function , expected resource consumption indicators of a current to-be-scheduled inference task on each GPU device are calculated, wherein the expected resource consumption indicators are used to represent the resource occupation trend of the current to-be-scheduled inference task when landing on each GPU; Step S302: for the expected resource consumption indicators, memory constraint matching judgment and utilization constraint matching judgment are sequentially performed through a preset soft constraint matching condition, and a matching result of the resource consumption indicators is output; Step S303: based on the matching result, a Boolean operation method is used to form a final schedulability judgment result, and based on the schedulability judgment result, a schedulability judgment matrix is generated .

[0012] Preferably, in step S40, based on the schedulability judgment matrix , a hybrid priority scheduling mechanism based on a fluctuation buffer window is used to perform inference task allocation control, and a task allocation target matrix is output. Specifically, the step includes: Step S401: based on the schedulability judgment matrix , a memory occupation sequence in a preset period T is obtained, and a memory fluctuation buffer window value is calculated by using a skewness sliding range method according to the memory occupation sequence And based on the video memory fluctuation buffer window value The fluctuation buffer matrix is constituted. Step S402: fuse the fluctuation buffer matrix and the GPU video memory prediction function And the GPU utilization prediction function The comprehensive scheduling priority score function P is constructed in a linear weighting manner. Step S403: screen out all the GPU node set G in the schedulability judgment matrix In the schedulability judgment matrix, the current task set to be scheduled is obtained, and the task allocation target matrix is constructed based on the schedulable GPU node set G, the current task set to be scheduled and the comprehensive scheduling priority score function P using the minimum priority matching strategy.

[0013] The application also provides a graphics card resource load balancing control system, which comprises: The parameter acquisition and sample modeling module is used for collecting the batch inference request input parameters of the LLM inference runtime by using the NVIDIA-SMI, and the batch inference request input parameters include the batch size B, the Token sequence length , GPU occupancy And GPU utilization ; based on the batch inference request input parameters, a dynamic sample collection and modeling mechanism with supervised error correction is used to generate a three-tuple sample set ; The structure perception regression prediction module is used for executing the GPU load prediction task based on the three-tuple sample set Using a multi-dimensional regression mechanism constructed based on structure perception features, the GPU video memory prediction function And the GPU utilization prediction function ; The soft constraint inference schedulability judgment module is used for executing the task schedulability judgment task based on the GPU video memory prediction function And the GPU utilization prediction function , and outputting the schedulability judgment matrix ; The hybrid priority scheduling control module is used for executing the inference task allocation control based on the schedulability judgment matrix Using a hybrid priority scheduling mechanism based on the fluctuation buffer window, the task allocation target matrix is output. The block-type dynamic video memory management module is used for dynamic video memory management in a multi-level KV cache block recycling manner.

[0014] The application further provides a control device for GPU resource load balancing, comprising a memory, a processor and a control program for GPU resource load balancing stored in the memory and executable on the processor, and the control program for GPU resource load balancing realizes the control method for GPU resource load balancing when executed by the processor.

[0015] The application further provides a computer program product comprising the control program for GPU resource load balancing, which realizes the control method for GPU resource load balancing when executed by a processor.

[0016] The application has the beneficial effect that the application first constructs a GPU memory and utilization rate prediction model by taking batch size and Token sequence length as joint input features, which can effectively represent the nonlinear influence of KV cache growth in the inference stage of a Transformer model on memory load. Compared with traditional schemes based on static empirical formulas or single-parameter estimation, the application can realize high-precision prediction of GPU resource usage peaks before inference execution, reducing resource estimation errors.

[0017] The application introduces a memory fluctuation buffer window mechanism and a multi-level KV cache block recycling strategy to reasonably utilize GPU resources under high load conditions while dynamically avoiding the risk of memory overflow caused by KV cache expansion in the inference intermediate state. Combined with soft constraint schedulability judgment and a hybrid priority scoring mechanism, the application can effectively improve the stability and throughput efficiency of inference services in scenarios with large fluctuations in Token length or task-intensive concurrency. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0019] Figure 1 The flowchart of the first embodiment of the control method for GPU resource load balancing.

[0020] Figure 2 The GPU memory utilization fluctuation trend and fluctuation buffer window trigger area diagram of the first embodiment of the control method for GPU resource load balancing.

[0021] Figure 3 The fitting result diagram of the GPU memory occupancy prediction model of the first embodiment of the control method for GPU resource load balancing.

[0022] Figure 4 Figure 1 is a GPU utilization prediction effect schematic diagram of a first embodiment of a graphics card resource load balancing control method of the present application.

[0023] Figure 5 Figure 2 is a device schematic diagram of a graphics card resource load balancing control method of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0025] Embodiment one: as shown in Figure 1, a flow schematic diagram of a first embodiment of a graphics card resource load balancing control method of the present application is proposed. Figure 1

[0026] In the first embodiment, the graphics card resource load balancing control method comprises: Step S10: using NVIDIA-SMI to collect batch inference request input parameters of LLM inference runtime, the batch inference request input parameters including batch size B, Token sequence length L, GPU occupancy rate and GPU utilization rate; Step S20: using a dynamic sample collection and modeling mechanism with supervised error correction to generate a triple sample set based on the batch inference request input parameters; It should be noted that the "dynamic sample collection and modeling mechanism with supervised error correction" refers to the process of real-time collection and closed-loop modeling of batch inference request parameters of large language model (LLM) inference runtime. The collection content includes not only batch size B and Token sequence length L, but also GPU running load closely related memory occupancy rate and GPU computing utilization rate; among them, the memory occupancy rate is used to reflect the actual pressure of KV cache dynamic growth on the graphics card memory, and the GPU utilization rate reflects the degree of algorithm power scheduling and kernel concurrency. This mechanism periodically calls the NVIDIAsmi interface through Python pynvml library, combines task ID and timestamp for data pairing, so as to generate structured sample entries. Each sample entry contains input features, device state and prediction error label, which can be directly used for memory prediction function and GPU utilization rate prediction function training in subsequent steps.

[0027] ​​​​It should be understood that, compared with the method of relying only on empirical formula or linear proportion estimation of GPU load in traditional technology (such as simply setting the video memory occupancy = Batch Size × Token length × constant coefficient), the supervised error correction modeling mechanism of the present application has obvious improvement. The traditional method lacks self-feedback ability to the error source, which leads to the accumulation of video memory prediction error when the Token sequence distribution is skewed and the input sample dispersion is large. The present application introduces a "supervised error label" in the sampling stage, which can statistically label the difference between the predicted result and the actual video memory occupancy in real time, and update the model parameters by weighting, realizing the self-learning and dynamic convergence of the model. For example, in the long sequence reasoning task, the model will automatically increase the weight coefficient of Token length, so that the prediction function is more consistent with the actual KV cache expansion trend, thereby maintaining the prediction stability and high confidence under mixed load of long and short sequences.

[0028] For example, in a LLaMA-13B model inference scenario, three groups of typical task samples were recorded: (1) Batch Size = 4, Token length = 512, video memory occupancy is 11.8 GB, utilization rate is 46%; (2) Batch Size = 8, Token length = 1024, video memory occupancy is 18.6 GB, utilization rate is 63%; (3) Batch Size = 16, Token length = 2048, video memory occupancy is 31.9 GB, utilization rate is 87%. The predicted value of the traditional method using linear proportion estimation is about 27 GB, with an error of 15%.

[0029] And through the supervised error correction mechanism of the present application, after continuously updating 20 batches of samples, the model converges to 31.2 GB for the same input video memory prediction value, with an error of 2.1%. The results show that the dynamic sample collection and feedback modeling process can effectively capture the real law of GPU video memory fluctuation, significantly improve the prediction accuracy and scheduling stability, and provide reliable basic data support for subsequent GPU task allocation.

[0030] Step S20: based on the triple sample set A multi-dimensional regression mechanism based on structure perception features is adopted to perform GPU load prediction tasks, and a GPU video memory prediction function is output And a GPU utilization rate prediction function ; It should be noted that the "multi-dimensional regression mechanism based on structure-aware features" in this step refers to the establishment of a GPU load prediction model, not only batch size B and Token sequence length L as input features, but also further introduce structure-aware feature vectors, including the product item, square item, logarithmic item and cross-influence factor, so as to form a nonlinear mapping relationship in the feature space. This feature construction method can reflect the combined influence of memory and computing power in the KV cache generation, matrix multiplication accumulation and Attention weight calculation process under the Transformer structure. During model training, the supervised error labeled in the triple sample set is used as the loss feedback signal, and the multi-dimensional regression parameters are corrected in batches to form two target functions: GPU memory prediction function and GPU utilization prediction function. The former is used to predict the peak memory occupancy under different input configurations, and the latter is used to infer the concurrent strength and utilization trend of the computing unit, which together form the input basis for subsequent scheduling and threshold control.

[0031] It can be understood that the implementation of this step changes the GPU load prediction process from a single variable linear relationship to a multi-dimensional dynamic nonlinear regression model. By introducing structure-aware features, the synergistic effect of batch size and Token length on memory occupancy, as well as the resource coupling characteristics of different computing stages, can be more accurately described. This mechanism can estimate the memory and computing resource requirements of a task before inference, providing forward-looking data support for subsequent task allocation and memory threshold judgment. Unlike traditional static threshold methods, the present application can predict future GPU peak utilization before the model has executed inference, thereby limiting task concurrency or adjusting allocation strategies in advance, significantly improving stability and task execution efficiency. For example, as shown in Figure 3 The fitting results of the GPU memory occupancy prediction model based on structure-aware features are shown, which are basically consistent with the real label curve, indicating that the constructed multi-dimensional nonlinear regression mechanism has strong modeling ability in capturing the complex relationship between batch size, historical memory residual and Token length and other parameters. As shown in Figure 4 Similarly, the prediction accuracy of GPU core utilization is shown, and it can be seen that the prediction curve is also highly consistent with the real curve.

[0032] For example, taking the LLaMA-13B model in a single four-card environment as an example, by collecting GPU running data under different Batch and Token combinations, a three-tuple sample set containing 3000 samples is constructed. After training using the multi-dimensional regression mechanism described in this step, the GPU memory prediction function predicts that the memory usage is 18.4GB under the condition of Batch=8, Token=1024, and the actual measured value is 18.7GB, with an error of only 1.6%; the GPU utilization prediction function predicts a value of 65.3%, and the actual measured value is 66.1%. Compared with the traditional linear estimation method, the average error of the structure-aware multi-dimensional regression model of the present application is significantly reduced, effectively improving the prediction accuracy of memory and computing power requirements. Through the prediction results, the number of acceptable tasks can be determined before scheduling, preventing problems such as memory overallocation or GPU idling, and achieving more stable and efficient inference load balancing control.

[0033] Step S30: Based on the GPU memory prediction function and the GPU utilization prediction function A multi-objective soft constraint matching mechanism is used to perform task schedulability determination, and a schedulability judgment matrix is output ; It should be noted that the "multi-objective soft constraint matching mechanism" refers to the process of constructing multiple constraint objectives and making comprehensive judgments based on the results of the memory prediction function and the GPU utilization prediction function output in step S20 before task scheduling. This mechanism takes the available GPU memory space, current computing utilization, predicted utilization growth trend, and GPU running stability as inputs to establish a multi-objective decision model. This model calculates the matching degree between each task and each GPU node through a weighted function and outputs the task-GPU schedulability result in matrix form. "Soft constraint" means allowing a certain range of resource fluctuations under the premise of meeting the safe operation boundary, thereby avoiding the problem of task rejection or frequent migration caused by hard threshold judgment. Finally, the task carrying state of each GPU is determined based on the results of the schedulability judgment matrix.

[0034] It should be understood that, compared with the scheme of using a single indicator (such as GPU free memory or average utilization) for task schedulability judgment in traditional technology, the soft constraint matching mechanism of the application has been significantly improved. The traditional method often considers that the memory remaining amount is lower than a certain fixed threshold as "unschedulable", ignores the difference in memory growth rate of different tasks in the subsequent reasoning stage, and causes part of the GPU resources to be over reserved or scheduling failure. The application introduces the predicted value into the constraint condition, so that the schedulability calculation of the GPU node under different task combinations has "dynamic tolerance", and allows short buffer scheduling when the resource is close to the upper limit, thereby improving the overall utilization and throughput of the GPU under the premise of safety. This method is particularly suitable for reasoning scenarios with high long sequence task proportion and frequent memory fluctuation, and can effectively avoid the problems of memory reservation waste and node overload in traditional scheduling algorithms.

[0035] For example, taking a multi-GPU inference cluster as an example, when multiple batch inference tasks arrive in the queue at the same time, first, the memory prediction function is used to estimate the memory demand curve of each task in the inference stage, and the utilization prediction function is used to evaluate the calculation load trend. If the predicted memory occupancy of a GPU node is close to the upper limit, but the calculation utilization is still in the expandable interval, the soft constraint matching mechanism will judge that the task is "conditionally schedulable" according to the task priority and risk weight, and set a delayed start or low priority execution mode for it in the subsequent scheduling stage. In this way, resource contention between tasks can be avoided, and the idle calculation period of the GPU can be fully utilized. Practice shows that under this mechanism, the resource utilization of GPU scheduling fluctuates more smoothly, the inference queue blocking rate is significantly reduced, and the stability and processing efficiency of the overall inference task are improved.

[0036] Step S40: according to the schedulability judgment matrix The hybrid priority scheduling mechanism based on fluctuation buffer window is used to perform inference task allocation control, and a task allocation target matrix is output. It should be noted that the "hybrid priority scheduling mechanism based on fluctuation buffer window" means that when performing inference task allocation control, the short-term resource fluctuation trend of the GPU node and the scheduling priority of the current task are considered at the same time, and a dynamic and controllable task allocation strategy is realized through the introduction of sliding window analysis and multi-level scheduling queue. The "fluctuation buffer window" refers to a sliding time window mechanism constructed on the basis of GPU resource utilization prediction, which is used to dynamically monitor the fluctuation gradient of the memory occupancy and utilization of each GPU node to evaluate the stability of resource load; the "hybrid priority scheduling" is a joint scheduling sorting and adaptive matching based on the static priority of the task itself (such as the type, source or urgency of the inference task) and the dynamic fluctuation risk of the node state, to generate a task allocation target matrix and submit it to the execution control module.

[0037] It can be understood that the mechanism significantly improves the comprehensive scheduling performance of reasoning task allocation between load balancing and resource safety. By introducing the fluctuation buffer window, the trend fluctuation can be perceived before the video memory or computing resources reach the peak, and the task allocation strategy can be responded in advance, so as to avoid the problems such as "critical fullness" of video memory and hard task discard. At the same time, the mixed priority scheduling mechanism not only ensures the timely scheduling of high-priority tasks, but also provides a buffer waiting space and a dynamic insertion channel for low-priority tasks, preventing some tasks from being stranded or starved for a long time due to single priority sorting, and enhancing the overall task throughput efficiency and fairness. For example, as shown in Figure 2 The black solid line represents the change curve of the video memory utilization of a certain GPU node in a continuous reasoning task scheduling period; the black dashed line is the preset "fluctuation buffer threshold", which is usually lower than the physical threshold of the GPU (such as 90%); the gray highlighted area in the figure represents the "intervention period" triggered by the system when it predicts that the video memory utilization will continuously break through the buffer threshold. It can be observed in the figure that the video memory utilization is in a slow growth stage in the early stage, but it shows a significant increasing trend in the middle and late stages, indicating that the reasoning load is increasing or the task queue is piling up; when the utilization rate approaches the buffer threshold shown by the dashed line, the system determines through the "sliding fluctuation window" that the video memory fluctuation trend enters the high-speed rising interval, triggering the dynamic control strategy in the gray intervention area.

[0038] For example, in a multi-task concurrent large language model reasoning scenario, if a certain GPU node frequently carries batch reasoning requests with long Token sequences in a short period of time, although the current resources have not been exhausted, the sliding fluctuation window detects that the video memory utilization of the node shows an accelerated growth trend in the past several scheduling periods. At this time, the "buffer period" can be set for the node according to the fluctuation trend, temporarily limiting the rate at which it accepts new tasks, and preferentially allocating new tasks to other nodes with stable fluctuations. For tasks waiting for scheduling, they can be allocated to the double-buffer scheduling queue according to their static priority and resource adaptation degree, ensuring that high-priority tasks can be quickly allocated, and low-priority tasks can also be fairly scheduled when resources are stable. This strategy not only improves the forward-looking of the scheduling process, but also effectively avoids the "resource contention" phenomenon between task batches, thereby enhancing the stability and throughput of the whole system.

[0039] Step S50: The application task allocation target matrix adopts a multi-level KV cache block recycling method for dynamic video memory management.

[0040] It should be noted that the "multi-level KV cache block recycling" refers to the large amount of Key-Value (KV) cache occupation problem generated by the inference stage of the Transformer type large language model. In combination with the priority of the task allocation target matrix, the Token progress and distribution state, a multi-level memory release priority strategy is constructed, and the KV cache is dynamically managed and selectively recycled in "sub-task granularity". Among them, "multi-level" means that the cache release priority is divided into multiple levels, covering task execution completion release, low priority task recycling, segment level segment cleaning and other strategies; "block" means that during model execution, the KV cache is divided into identifiable memory blocks according to Token position, Batch dimension, etc., so as to realize fine management and recycling control of memory space. This mechanism takes the task allocation target matrix as the core control input to ensure consistency and timing alignment between release operations and task scheduling strategies.

[0041] It can be understood that by performing dynamic KV cache block recycling after task allocation, this step can significantly improve the reuse efficiency and redundant release capability of memory resources in inference. Compared with the traditional one-time and overall cache release method, this step performs differentiated recycling operations based on task execution state and allocation results, which can release unnecessary KV cache segments in advance and alleviate the situation of tight memory in high concurrency scenarios. At the same time, this mechanism has good scalability and supports further integration with page table management, intelligent IO controller and other underlying mechanisms in the future, and is suitable for complex deployment environments such as multi-GPU node and heterogeneous computing resource pool.

[0042] For example, in a multi-path concurrent LLM inference system deployed in practice, when the task allocation target matrix determines that a GPU node will accept a batch of new inference tasks, it will immediately retrieve the KV cache state on the node. If it is found that there are some KV cache segments from low-priority tasks on the current node, and the corresponding inference process has entered the tail segment or has no subsequent expansion Token demand, these cache segments will be released in priority to free up memory space; if not enough, the higher level cache block management strategy can be entered, and the cache on the non-critical path in execution is marked or delayed recycling plan. This multi-level recycling strategy effectively avoids the situation that high-priority tasks are rejected due to insufficient memory, and improves the node's sustainable scheduling capability and overall task completion rate.

[0043] Embodiment two: In addition, the application provides a graphics card resource load balancing control system, which adopts the graphics card resource load balancing control method in the above embodiment, and can solve the technical problem of graphics card resource load balancing control. Compared with the prior art, the graphics card resource load balancing control system provided by the application has the same beneficial effects as the graphics card resource load balancing control method provided by the above embodiment, and other technical features in the graphics card resource load balancing control system are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0044] Embodiment three: The application provides a graphics card resource load balancing control device, please refer to Figure 5A graphics card resource load balancing control device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the graphics card resource load balancing control method of embodiment one. The graphics card resource load balancing control device of the embodiments can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDA (Personal Digital Assistant), PAD (Portable Application Description), PMP (Portable Media Player), car terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The graphics card resource load balancing control device is merely an example and should not limit the functions and use range of the embodiments. The graphics card resource load balancing control device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, or the like) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded into a random access memory 1004 from a storage device 1003. The random access memory 1004 also stores various programs and data required for the operation of the graphics card resource load balancing control device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An I / O interface 1006 is also connected to the bus. In general, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; the storage device 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 1009. The communication device 1009 can allow the graphics card resource load balancing control device to communicate with other devices wirelessly or wired to exchange data. Although the graphics card resource load balancing control device having various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or provided. More or less systems can be alternatively implemented or provided.

[0045] Embodiment four: the application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the control method of the graphic card resource load balancing as described above. The computer program product provided by the application can solve the technical problem of the control of the graphic card resource load balancing. Compared with the prior art, the computer program product provided by the application has the same beneficial effects as the control method of the graphic card resource load balancing provided by the above-mentioned embodiment, which will not be described here.

[0046] In particular, according to the embodiments disclosed by the application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed by the application include a computer program product, which comprises a computer program carried on a computer readable medium, and the computer program comprises program codes for executing the method shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by a processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed by the application are executed.

[0047] It should be understood that various parts of the embodiments disclosed by the application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0048] Obviously, those skilled in the art can make various modifications and variations to the application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the application belong to the scope of the claims of the application and their equivalent technologies, the application also intends to include these modifications and variations.

Claims

1. A method for controlling load balancing of graphic card resources, characterized in that, The method comprises: Step S10: Collecting batch inference request input parameters of the LLM inference runtime by using NVIDIA-SMI, the batch inference request input parameters including batch size B, Token sequence length , GPU occupancy , and GPU utilization ; generating a triple sample set based on the batch inference request input parameters by using a dynamic sample collection and modeling mechanism with supervised error correction ; Step S20: based on the triple sample set A multi-dimensional regression mechanism based on structure perception features is adopted to perform the GPU load prediction task, and a GPU memory prediction function is output The GPU utilization rate prediction function ; Step S30: predicting the function based on GPU memory With GPU utilization prediction function Adopting multi-objective soft constraint matching mechanism to execute task schedulability determination task, output schedulability judgment matrix ; Step S40: judging the matrix according to the schedulability The inference task allocation control is performed by using the hybrid priority scheduling mechanism based on the fluctuation buffer window, and a task allocation target matrix is output. Step S50: applying a task allocation target matrix to perform dynamic video memory management in a multi-level KV cache block recycling manner.

2. The method of claim 1, wherein the method further comprises: In step S10, a dynamic sample collection and modeling mechanism with supervised error correction is used to generate a triple sample set based on the batch inference request input parameters The steps specifically include: First, perform real-time sampling and data archiving tasks: during LLM inference runtime, normalize and structure store data of batch size B, Token sequence length , GPU occupancy , and GPU utilization based on batch inference request input parameters through Python's pynvml library, output initial sample data structure set; Then, error feedback and dynamic labeling tasks are performed: after the LLM inference operation is completed, a prediction sample data structure set output by a pre-deployed GPU prediction lightweight model is obtained, a sample error set is calculated by using a weighted mean square error method according to the prediction sample data structure set and the initial sample data structure set, and the initial sample data structure set is labeled by taking the sample error set as a weight to form a dynamic sample set with supervised error feedback; Finally, execute the partition pre-training task: according to the dynamic sample set with supervised error feedback and the Token sequence length Perform partition gradient boosting regression training using the scikit-learn library of Python, output the regression sub-tree set of the dynamic sample set with supervised error feedback; finally, based on the regression sub-tree set, use the joblib library of Python for serialization and save, and generate a triple sample set .

3. The method for controlling graphics card resource load balancing as described in claim 1, characterized in that, In step S10, the triplet sample set is expressed by the formula: , n is the total number of samples in the triplet sample set, is the batch size of the i-th sample in the triplet sample set, is the Token sequence length of the i-th sample in the triplet sample set, is the GPU occupancy rate of the i-th sample in the triplet sample set , is the GPU utilization rate of the i-th sample in the triplet sample set.

4. The method for controlling graphics card resource load balancing as described in claim 1, characterized in that, In step S20, based on the triple sample set A multi-dimensional regression mechanism based on structure perception features is adopted to perform the GPU load prediction task, and a GPU memory prediction function is output The steps of the GPU utilization rate prediction function , specifically comprising: Step S201: based on the triple sample set The structure-aware feature set F is constructed, and the structure-aware feature set F includes structure-aware base features, structure-aware combination features and structure-aware auxiliary features; a structure-aware feature matrix is constructed based on the structure-aware base features, the structure-aware combination features and the structure-aware auxiliary features The structure-aware feature matrix for enhancing the sensitivity of the model to the change of the input space structure; Step S202: constructing a structure perception feature matrix The first regression sub-model and the second regression sub-model are respectively constructed for the two prediction targets of GPU occupancy and GPU utilization; the first regression sub-model is constructed by using ridge regression; and the second regression sub-model is constructed by using Huber regression. Step S203: performing error verification by using residual analysis method based on K-fold cross-validation for the first regression sub-model, constructing and GPU memory predicting function ; performing generalization verification by using robustness evaluation method based on residual distribution consistency for the second regression sub-model, constructing and outputting GPU utilization predicting function .

5. The method of claim 4, wherein the step of determining the load of the graphic card comprises: determining the load of the graphic card according to the number of the graphic card resources and the number of the graphic card resources in use. In step S20, the perception-based features include batch size features and Token sequence length features; the structure-perception combined features include batch and sequence coupling features, unit Token load features, batch smoothing logarithm features, and sequence length logarithm features; The structure-perception auxiliary features include request type label features.

6. The method of claim 1, wherein the method further comprises: In step S30, the GPU memory prediction function is based on The GPU utilization prediction function is based on The multi-objective soft constraint matching mechanism is used to perform task schedulability determination tasks, and output a schedulability judgment matrix The steps include: Step S301: predicting a function based on GPU memory With GPU utilization prediction function Calculate the expected resource consumption index of the current to-be-scheduled inference task on each GPU device, and the expected resource consumption index is used to represent the resource occupation trend of the current to-be-scheduled inference task when landing on each GPU. Step S302: for the expected resource consumption index, sequentially perform memory constraint matching determination and utilization constraint matching determination through preset soft constraint matching conditions, and output a matching result of the resource consumption index; Step S303: forming a final schedulability decision result based on the matching result by using a Boolean operation method, and generating a schedulability judgment matrix based on the schedulability decision result .

7. The method for controlling graphics card resource load balancing as described in claim 1, characterized in that, In step S40, the schedulability judgment matrix is determined according to the schedulability judgment matrix The step of executing the inference task allocation control by using the hybrid priority scheduling mechanism based on the fluctuation buffer window and outputting the task allocation target matrix comprises the following steps. Step S401: Determine the schedulability matrix Obtain the video memory usage sequence within a preset period T, and calculate the video memory fluctuation buffer window value based on the skewed sliding range method according to the video memory usage sequence. And based on the memory fluctuation buffer window value Construct a fluctuation buffer matrix; Step S402: fuse the fluctuation buffer matrix and the GPU memory prediction function The GPU utilization prediction function The comprehensive scheduling priority score function P is constructed in a linear weighting manner; Step S403: In the schedulability judgment matrix In the schedulability judgment matrix, a set of GPU nodes G in a schedulable state is screened out, and a current set of tasks to be scheduled is obtained; a task allocation target matrix is constructed based on the set of GPU nodes G in a schedulable state, the current set of tasks to be scheduled, and a comprehensive scheduling priority scoring function P by using a minimum priority matching strategy.

8. A control system for graphics card resource load balancing, applied to the control method for graphics card resource load balancing in any one of claims 1 to 7, characterized in that, The control system of the GPU resource load balancing comprises: A parameter collection and sample modeling module is configured to collect batch inference request input parameters of an LLM inference runtime by using NVIDIA-SMI, the batch inference request input parameters including batch size B, Token sequence length , GPU occupancy , and GPU utilization ; generating a triple sample set based on the batch inference request input parameters by using a dynamic sample collection and modeling mechanism with supervised error correction ; a structure-aware regression prediction module configured to predict based on the triple set a multi-dimensional regression mechanism based on structure-aware features to perform the GPU load prediction task, outputting a GPU memory prediction function a GPU utilization prediction function ; The soft constraint reasoning schedulability determination module is configured to determine the schedulability of the task based on the GPU memory prediction function and the GPU utilization prediction function The multi-target soft constraint matching mechanism is adopted to execute the task schedulability determination task, and a schedulability judgment matrix is output ; A hybrid priority scheduling control module is configured to determine a schedulability judgment matrix The inference task allocation control is performed by using a hybrid priority scheduling mechanism based on a fluctuation buffer window, and a task allocation target matrix is output. The block-type dynamic video memory management module is configured to apply a task allocation target matrix to perform dynamic video memory management in a multi-level KV cache block recycling manner.

9. A device for controlling load balancing of graphic card resources, characterized in that, The control device of the GPU resource load balancing comprises a memory, a processor, and a GPU resource load balancing control program stored in the memory and executable on the processor, and the GPU resource load balancing control program implements the GPU resource load balancing control method in any one of claims 1 to 7 when executed by the processor.

10. A computer program product, characterised in that, The computer program product comprises a GPU resource load balancing control program, and the GPU resource load balancing control program implements the GPU resource load balancing control method in any one of claims 1 to 7 when executed by a processor.

Images