Deep learning training and reasoning task dynamic cooperation system based on GPU space-time resource sharing

Abstract

The invention provides a deep learning training and reasoning task dynamic cooperation system based on GPU space-time resource sharing, and the system comprises a GPU resource state perceptron which is used for monitoring a GPU kernel function call sequence and a video memory distribution state of a distributed training task in real time, dynamically capturing a calculation gap and video memory fragments generated by the training task, and transmitting the calculation gap and the video memory fragments to the GPU resource state perceptron; generating a two-dimensional resource spatial-temporal characteristic spectrum, and predicting a GPU calculation idle period caused by communication synchronization based on an LSTM model; the kernel function dynamic scheduling decision maker is used for performing priority division and dynamic resource quota allocation on an online reasoning task and an offline reasoning task by adopting a self-adaptive allocation strategy on the basis of a resource spatial-temporal characteristic spectrum so as to realize spatial-temporal resource decoupling of the training task and the reasoning task; and the kernel function execution arbiter is used for dynamically controlling submission and blockage of the reasoning task kernel function according to the scheduling decision through video memory space multiplexing and a calculation instruction arbitration mechanism. According to the method, the GPU resource utilization rate is remarkably improved, on the premise that stable training task performance is guaranteed, fragmented resources are effectively utilized to support parallel execution of multiple types of reasoning tasks, and efficient resource collaboration of a deep learning task cluster is achieved.

Description

Technical Field

[0001] The present invention relates to deep learning technology, specifically a dynamic collaborative system for deep learning training and reasoning tasks based on GPU spatiotemporal resource sharing. Background Art

[0002] Distributed deep learning training utilizes multiple GPUs for computation through parallelism, effectively improving training speed and throughput. Mainstream distributed training strategies include data parallelism, model parallelism, pipeline parallelism, and hybrid parallelism. These parallel approaches partition data or models across multiple GPUs and leverage explicit communication between them to complete forward and backward propagation of training data. However, due to factors such as irrational resource allocation, high communication overhead, and fault recovery, GPUs in training clusters are often underutilized, resulting in a significant amount of idle GPU resources in terms of computation and memory during distributed deep learning training.

[0003] Communication between GPUs can lead to significant idle time in computing resources. Consequently, in recent years, several research efforts, including torch.DDP, DeepSpeed, and ColossalAI, have attempted to optimize communication overhead. However, these efforts have not been able to effectively address the non-overlap between computation and communication. A significant amount of GPU time is consumed waiting for communication, leading to frequent fluctuations in GPU computational utilization during training tasks due to the non-overlap between computation and communication, reducing computational resource utilization efficiency. Furthermore, GPU memory resources are also fragmented. This is because, to ensure training convergence and maximize GPU performance, the local batch size is typically set to a power of 2, making it difficult to precisely reach the upper limit of the graphics card's memory. This results in a certain amount of memory fragmentation during actual training. This phenomenon of insufficient GPU resource utilization, both in terms of computational resources (the temporal dimension) and memory resources (the spatial dimension), can be collectively referred to as GPU resource fragmentation.

[0004] In recent years, the rapid development of deep learning has brought significant progress in fields such as industrial manufacturing, artistic creation, and online services. The emergence of large language models such as GPT has made significant contributions, leading to widespread attention and in-depth research on large language models. However, the large number of large language model training tasks has led to a surge in demand for GPUs, significantly increasing costs. In this context, GPU resource fragmentation, which significantly reduces cost efficiency, has become an urgent problem that needs to be addressed. Summary of the Invention

[0005] This paper proposes a dynamic collaborative system for deep learning training and reasoning tasks based on GPU spatiotemporal resource sharing.

[0006] The technical solution to achieve the purpose of the present invention is: a dynamic collaborative system for deep learning training and reasoning tasks based on GPU spatiotemporal resource sharing, comprising:

[0007] GPU resource status sensor, kernel function dynamic scheduling decision maker, and kernel function execution arbitrator, including:

[0008] The GPU resource status sensor is used to monitor the GPU kernel function call sequence and memory allocation status of distributed training tasks in real time, dynamically capture the computing gaps and memory fragmentation generated by training tasks, generate a two-dimensional resource spatiotemporal feature map, and predict GPU computing idle cycles caused by communication synchronization based on the LSTM model;

[0009] The kernel function dynamic scheduling decision maker is used to prioritize and dynamically allocate resource quotas for online and offline reasoning tasks based on the spatiotemporal feature map of resources, adopting an adaptive allocation strategy, thereby achieving spatiotemporal resource decoupling of training and reasoning tasks.

[0010] The kernel function execution arbiter is used to dynamically control the submission and blocking of the inference task kernel function according to the scheduling decision through the memory space multiplexing and computing instruction arbitration mechanism.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] This invention solves the problems of low GPU computing utilization and fragmentation in distributed deep learning training systems, achieving more flexible and efficient management and use of computing resources and video memory resources. It also provides additional inference services while effectively improving GPU utilization and maintaining training throughput.

[0013] The present invention utilizes the idle period of computing and memory resources of the training task cluster to execute sufficient inference tasks, thereby achieving efficient utilization of GPU resources.

[0014] The present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of a dynamic collaborative system for deep learning training and inference tasks based on GPU spatiotemporal resource sharing.

[0016] Figure 2 This is an architectural diagram of a dynamic collaborative system for deep learning training and inference tasks based on GPU spatiotemporal resource sharing.

[0017] Figure 3 This is a flowchart of a dynamic collaborative system for deep learning training and inference tasks based on GPU spatiotemporal resource sharing.

[0018] Figure 4 Flowchart for monitoring GPU resource status sensor.

[0019] Figure 5 Adaptive scheduling flow chart for the kernel function dynamic scheduling decision maker.

[0020] Figure 6 Process the inference kernel flow chart for the kernel execution arbitrator component. DETAILED DESCRIPTION

[0021] The present invention mainly constructs a system for dynamic coordination of deep learning training and reasoning tasks based on GPU spatiotemporal resource sharing, which is composed of three key components. Among them, the kernel control layer includes a GPU resource status sensor and a kernel function execution arbitrator: the GPU resource status sensor component is used to monitor and record the GPU idle period of the training task in real time, and is suitable for distributed deep learning frameworks such as DeepSpeed and Megatron; the kernel function execution arbitrator component is responsible for controlling the blocking or forwarding of the kernel function of the reasoning task, and making full use of the GPU idle resources without affecting the training performance. The coordination layer uses a dynamic priority scheduling strategy to adjust the specific resource allocation based on the spatiotemporal characteristics of resources (computing-video memory availability) through the kernel function dynamic scheduling decision maker. At the same time, the kernel function dynamic scheduling decision maker, which serves as the core, designs different scheduling mechanisms and throughput-optimal batch sizes for offline and online reasoning tasks to meet the service quality target requirements, comprehensively optimize GPU resource utilization, and achieve efficient coordination of training and reasoning tasks.

[0022] The GPU resource status sensor in the present invention intercepts the CUDA kernel function of the training task, collects statistics on GPU computing and video memory idle information in real time, and uses the sequence prediction model LSTM to estimate the idle duration, providing an accurate timing judgment basis for executing reasoning tasks.

[0023] The kernel function dynamic scheduling decision maker in the present invention is responsible for identifying GPU idle time for training tasks, relying on the service quality evaluator, combined with T_past and T_pred to jointly determine the size of the available computing resources for the inference task, and at the same time passing the available token size of the inference task to the kernel function execution arbitrator, coordinating the execution strategies of offline and online inference tasks, and realizing efficient dynamic allocation of resources.

[0024] The kernel execution arbiter component in this invention receives information from the kernel dynamic scheduling decider and dynamically controls the CUDA kernels of inference tasks based on the GPU resource requirements of the training and inference tasks, as well as the task's collaborative deployment strategy. It releases or blocks the execution of inference kernels during appropriate idle periods, thereby achieving efficient utilization of GPU resources while avoiding interference with training tasks. Simultaneously, a managed memory mechanism is introduced to expand the available memory for inference, optimize inference batch size, and further improve system throughput.

[0025] This system achieves efficient utilization of GPU resources and seamless coordination of distributed training and inference tasks through the close collaboration of three core components: GPU resource status sensor, kernel function dynamic scheduling decision maker and kernel function execution arbitrator. The overall architecture of this system covers the mutual coordination of the kernel control layer and the kernel coordination layer (such as Figure 2 As shown). Among them, the GPU resource status sensor is located in the kernel control layer. It mainly obtains the CUDA kernel function calls of the training task, counts and predicts the idle resource status of the training task in real time, and sends this information to the kernel function dynamic scheduling decision maker located in the kernel coordination layer to provide basic data for executing inference tasks, so that the kernel function dynamic scheduling decision maker can judge whether the current GPU has the ability to fully handle inference tasks. After receiving the GPU resource idle status information from the GPU resource status sensor, the kernel function dynamic scheduling decision maker combines the evaluation results of the training throughput of the service quality evaluator to adaptively and dynamically determine the execution time of the inference task and the allocated resources (i.e., the number of tokens). The scheduling logic of the kernel function dynamic scheduling decision maker ensures that the filling efficiency of the inference task is maximized without interfering with the training task. As an execution layer component, the kernel function execution arbitrator component is responsible for controlling the actual execution of the inference task according to the scheduling decision of the kernel function dynamic scheduling decision maker. Specifically, for online inference tasks, that is, when the high-priority queue is not empty, the kernel function dynamic scheduling decision maker detects that (T_pred-T_past)>T_exec (where T_exec represents the execution time of a single batch request inference). The kernel function execution arbitrator actively pulls online inference requests from the request queue one by one and executes them immediately, consuming the corresponding number of tokens. For offline inference tasks, that is, when the high-priority queue is empty, the kernel function execution arbitrator receives the token Q_available allocated by the kernel function dynamic scheduling decision maker and controls the submission of CUDA kernel functions for inference tasks. The execution of each CUDA kernel function consumes a corresponding number of tokens. When tokens are insufficient, the kernel function execution arbitrator component blocks the initiation of subsequent inference kernel functions to ensure that there is no conflict with the training task. When the training task resumes, the kernel function dynamic scheduling decision maker reduces the number of tokens to zero, at which point the kernel function execution arbitrator component automatically stops the execution of the inference task.

[0026] In summary, the core structure and processes of the entire system are realized through the coordinated linkage of three core components, which work together to optimize the aggregate throughput of distributed training and inference. Ultimately, this system fully utilizes idle GPU resources while ensuring the stability of training task performance and the service quality of inference tasks, achieving the core goals of system design.

[0027] like Figure 2 As shown, the present invention realizes the efficient capture and rapid utilization of GPU idle resources for distributed training tasks through the close coordination of GPU resource status sensors, kernel function dynamic scheduling decision makers and kernel function execution arbitrator components at different levels. Under the real-time monitoring of the GPU resource status sensor, the kernel function dynamic scheduling decision maker accurately schedules the inference tasks, and the kernel function execution arbitrator component ensures the execution and isolation of the inference tasks through a token mechanism. The entire process not only improves the GPU resource utilization rate, but also ensures that the SLO of online inference tasks is met, as well as the performance stability of training tasks. In addition, Figure 2 While explaining the system architecture, it also points out how this system interacts with distributed training tasks. That is, in the system architecture for dynamic collaboration of deep learning training and inference tasks for GPU spatiotemporal resource sharing, how key components such as the GPU resource status sensor, kernel function dynamic scheduling decider, and kernel function execution arbitrator components distributed in the kernel control layer and kernel coordination layer coordinate with distributed training instances, inference instances, and GPU computing units.

[0028] When executing inference tasks in a distributed training cluster, the system needs to be implemented through the collaborative work of multiple components, including detecting GPU idle resources, dynamically determining the amount of computation required to execute inference tasks, and isolating inference task execution from blocking mechanisms. The system will deploy inference instance execution based on the collaborative work of components, such as Figure 3 As shown:

[0029] (1) The GPU resource status sensor is the first step of the system. It is used to capture the idle resource size of the GPU in distributed training tasks in real time and predict idle cycles. The system uses the statistical information of CUDA kernel function calls in the training cluster to provide real-time feedback on the current GPU status (busy / idle) to the kernel function dynamic scheduling decision maker, which serves as the basis for subsequent inference task scheduling.

[0030] (2) The kernel function dynamic scheduling decision maker adaptively adjusts the timing and duration of inference task execution based on the GPU computing-memory idle state information and predicted idle duration provided by the GPU resource state sensor, and combines the execution status of the training task. It then passes the generated decision information to the kernel function execution arbitrator, thereby achieving inference filling, improving the throughput of inference tasks while avoiding interference with training tasks.

[0031] (3) After receiving the token from the kernel dynamic scheduling decision maker, the kernel execution arbitrator collects the CUDA kernel count information of the inference task, builds the optimal batch size based on the memory fragmentation and the managed memory (CUDA Managed Memory) of the training task gradient part, and makes a comprehensive decision to control the execution of the inference task on the GPU.

[0032] In a further embodiment, the GPU resource status sensor is one of the core components in the system constructed by the present invention, which is responsible for real-time monitoring of the GPU's computing-memory idle state and predicting the idle duration, providing basic data support for filling inference tasks.

[0033] Existing methods typically rely on NVML-based statistical tools to monitor GPU utilization, but these methods suffer from two major issues: first, statistical data collection has significant latency, with sampling periods ranging from 1 to 1 / 6 of a second; second, the fixed interval between data updates (e.g., 200ms) prevents fast and accurate identification of idle bubbles. These shortcomings severely impact the timeliness of speculative population.

[0034] To this end, the GPU resource status sensor introduces a statistical method based on hijacking CUDA kernel function calls to solve the above-mentioned timeliness problem. Specifically, the GPU resource status sensor uses a mounted monitoring library to hijack the CUDA kernel function calls of the training task, records the kernel function call counts, and stores these data in a high-speed updated sliding window (recorded every 2ms). By calculating the call counts within the sliding window, the GPU resource status sensor can quickly identify idle bubbles and send this information to the kernel function dynamic scheduling decision maker. Compared with traditional methods, the hijacking-statistical method of the GPU resource status sensor significantly improves the monitoring frequency and timeliness, ensuring that the GPU idle state can be perceived in real time, providing an accurate time window for the scheduling of subsequent inference tasks; on the other hand, the same hijacking method is used to statistically record all training task and memory allocation APIs to calculate the memory fragmentation of the current graphics card.

[0035] Specifically, each API related to video memory allocation will have a parameter representing the allocation size. By adding these values, we can get the video memory usage of the training task, and then the video memory fragmentation. At the same time, the GPU resource status sensor will build an LSTM timing sequence model to predict the idle period T_pred of the training task, which is used to guide the allocation of tokens for inference tasks.

[0036] The GPU resource status sensor is located in the distributed training instance of the kernel control layer. Its monitoring method mainly hijacks the CUDA kernel function call of the training task, counts the GPU computing-memory fragmentation information in real time, generates a two-dimensional resource spatiotemporal feature map, and sends it to the kernel function dynamic scheduling decision maker to provide basic data for the execution of the inference task. The specific process is as follows: Figure 4 As shown:

[0037] (1) The GPU resource status sensor monitors the CUDA kernel function calls sent to the GPU by the distributed training task through the mounted monitoring library, and tracks the execution status of the CUDA kernel function calls in real time; the GPU resource status sensor records the monitored CUDA kernel function call counts at fixed time intervals, and the CUDA kernel function call count data is stored in a sliding window, which is continuously updated to always keep the latest kernel function call information; the GPU resource status sensor analyzes the continuous zero call counts in the sliding window, that is, the GPU does not receive the CUDA kernel function call cycle within the set time, and the time period of continuous zero calls is regarded as the computing idle time T_past that has occurred;

[0038] (2) The GPU resource status sensor hijacks the kernel function information of all video memory allocation related calls in the training task through the mounted monitoring library, and uses this to count and record the remaining video memory fragmentation information of the current GPU, including the maximum continuous available video memory block size M and the total available video memory F;

[0039] (3) Based on the historical statistics of GPU busy-idle time series (CUDA kernel function call counts, the duration of consecutive zero call cycles, and the kernel function call interval time series obtained from the first two), an LSTM time series prediction model is constructed. After normalization, the data is input into a two-layer LSTM network. The first layer of 64 LSTM units learns short-term call patterns, and the second layer of 32 LSTM units captures long-term dependencies. Finally, the idle probability distribution of the future time window is output through the fully connected layer. The system uses a dynamic threshold method to analyze the output probability and takes the maximum consecutive idle period as the estimated computing idle time. Based on this, the estimated computing idle time T_pred in the current cycle is obtained by inference.

[0040] (4) The GPU resource status sensor analyzes the CUDA kernel function call pattern within the sliding window, and performs spatiotemporal correlation modeling on the actual computing idle time T_past, the predicted idle time T_pred, and the GPU memory fragmentation information (including the maximum continuous available memory block size M and the total available memory F), generating a two-dimensional resource spatiotemporal feature map (where the time dimension represents the temporal changes in computing resource availability, and the spatial dimension depicts the fragmented distribution of memory resources); then, the feature map is sent together with the original analysis data (including T_past, T_pred, and memory fragmentation information) to the kernel function dynamic scheduling decision maker to provide a more comprehensive resource status representation for task scheduling.

[0041] In a further embodiment, the kernel function dynamic scheduling decision maker is the core coordination component in the system constructed by the present invention, which is responsible for dynamically determining the timing of inference task execution and the amount of resource allocation based on the GPU status information provided by the GPU resource status sensor and the predicted GPU idle cycle.

[0042] As the decision center of the system, the kernel function dynamic scheduling decision maker uses an adaptive token allocation algorithm to implement token allocation for reasoning tasks (such as Figure 5 It mainly includes the following stages:

[0043] (1) The kernel function dynamic scheduling decision maker continuously receives monitoring data from the GPU resource status perception module, including: the accumulated computing idle time T_past, the remaining computing idle time T_pred predicted by the LSTM model, and the video memory fragmentation information (the maximum continuous available video memory block size M and the total available video memory F).

[0044] (2) Dynamic resource allocation is achieved through the following steps: Calculation of the total number of tokens, based on the weighted sum of T_past and T_pred monitored in real time, to determine the total number of tokens Q_total that can be allocated in the current cycle. Specifically, Q_total = (1-T_past / T_pred)*SYSTEM_LIMIT, where SYSTEM_LIMIT is a token upper limit value obtained through a large number of task tests, which depends on the computing power of each model of GPU. The significance of this weighted calculation method is that when T_past is small, it means that there is still a long time before the GPU switches from idle to busy, so more tokens can be released for inference tasks; when T_past is large (close to T_pred), that is, the GPU is about to switch from idle to busy, so in order to avoid the impact of asynchronous execution of the kernel function of the inference task on the throughput of the training task, the Q_total that can be allocated at this time must be reduced. Taking the memory constraints into consideration, the final number of tokens that can be allocated is: min(Q_total, max(M / S_online, F / S_offline)), where S_online and S_offline represent the amount of memory required for a single online and offline inference task, respectively. This dual constraint ensures the joint constraints of computing power (time dimension) and memory resources (memory dimension).

[0045] (3) The service quality evaluator monitors the stability of the training task iteration cycle in real time. When performance fluctuations or throughput attenuation exceeding a preset threshold are detected, the expansion factor c is reduced according to the exponential attenuation law of α (α<1). The initial expansion factor c is set to 1, and the new expansion factor c is the original expansion factor multiplied by α, that is, c = c * a. If the throughput returns to the original performance, the new expansion factor c is the expansion divided by α, that is, c = c / a. During system operation, the total number of tokens available for inference tasks is always maintained at Q_available = Q_total * c.

[0046] (4) When there are pending requests in the high-priority request queue of the online reasoning task, the reasoning task type signal S is set to 1, indicating that the total number of tokens Q_available is allotted to the online reasoning task; only when the online reasoning queue is empty, the reasoning task type signal S is set to 0, indicating that the total number of tokens Q_available that can be allocated in the current cycle is allocated to the offline low-priority reasoning task;

[0047] (5) The kernel function dynamic scheduling decision maker sends the token information of the final scheduling decision assigned to the inference task and the inference task type signal S to the kernel function execution arbitrator component, which is responsible for the specific execution.

[0048] In a further embodiment, the kernel function execution arbitrator component is a kernel control layer component in the system constructed by the present invention, which is responsible for specifically managing the execution process of the inference task and realizing isolation and dynamic regulation between the inference task and the training task through the token mechanism. The specific process is as follows Figure 6 As shown:

[0049] (1) Determine the optimal batch size. The kernel function execution arbitrator component manages the memory space occupied by gradient calculation and cross-device communication in the training task by referencing a counter. When the training task completes gradient synchronization and parameter update, the counter returns to zero. At this time, the system will immediately mark these free memory areas as available, update the memory status, and notify the kernel function dynamic scheduling decision maker through a message mechanism. After receiving the notification, the scheduling decision maker will calculate the optimal batch size B based on the current new memory status. opt , the formula is as follows:

[0050]

[0051] Among them, M free The sum of the video memory size temporarily available for inference after the reference count is reset to zero and the remaining allocatable video memory size of the graphics card after the gradient synchronization and parameter update are completed for training; M ovh represents the fixed video memory overhead that cannot be released on the inference side, such as the model weight resident area, CUDA stream, and codec buffer; m is the average video memory usage added during the forward process of processing an inference sample; W is the idle time window reserved for training, specifically T_pred - T_past, and t is the time it takes to complete an inference for a single sample. It is the memory constraint, which indicates the maximum number of samples that can be accommodated in the currently available memory. is a time constraint, indicating how many samples can be inferred in the idle time window. Batch selection: take the smaller value of the two, and then ensure that it is at least 1 to get the optimal batch size B opt .

[0052] The kernel function dynamic scheduling decision maker sends the updated token information assigned to the inference task and the inference task type signal S to the kernel function execution arbitrator component;

[0053] When a new training cycle begins, the video memory resources are gradually reclaimed through the reference counting mechanism;

[0054] The reference counter returns to 1, triggering the video memory recycling process; the execution arbiter suspends the submission of new inference tasks and waits for the ongoing tasks to complete; for long-running offline tasks, the system supports a graceful interrupt mechanism to ensure that the training tasks can obtain the required video memory resources in a timely manner.

[0055] (2) For online inference tasks, when the high-priority queue is not empty, that is, the inference task type signal S is 1, the kernel function dynamic scheduling decision maker detects that (T_pred-T_past)>T_exec (where T_exec represents the execution time of a single batch request inference), and the kernel function execution arbitrator actively pulls online inference requests one by one from the request queue to form the above-mentioned optimal batch size, and then immediately executes inference and consumes the corresponding number of tokens.

[0056] (3) For offline inference tasks, that is, when the high-priority queue is empty and the inference task type signal S is 0, the kernel function execution arbitrator receives the token Q_available allocated by the kernel function dynamic scheduling decision maker, controls the submission of the CUDA kernel function of the inference task, and the execution of each CUDA kernel function consumes a corresponding number of tokens. The batch size will also be optimized in advance. When the tokens are insufficient, the kernel function execution arbitrator component blocks the initiation of subsequent inference kernel functions to ensure that there is no conflict with the training task. When the training task resumes, the kernel function dynamic scheduling decision maker reduces the number of tokens to zero, and the kernel function execution arbitrator component automatically stops the execution of the inference task.

[0057] This paper proposes a dynamic collaborative system for deep learning training and inference tasks based on GPU spatiotemporal resource sharing, allowing inference instances to systematically use the GPU with training instances based on their video memory requirements and GPU idle characteristics. This system increases the aggregate GPU throughput by utilizing idle GPU resources in distributed training.

[0058] The present invention proposes a GPU resource status sensor based on GPU computing and video memory fragmentation hijacking CUDA kernel function-statistical method to detect GPU idle resources in real time and dynamically.

[0059] The present invention proposes an idle cycle prediction period based on the LSTM sequence prediction model in the GPU resource status sensor, which is used to provide decision information for the kernel function dynamic scheduling decision maker.

[0060] The present invention proposes an adaptive kernel function dynamic scheduling decision maker, which comprehensively considers GPU computing-memory fragmentation information and the throughput impact on training tasks to determine the timing and amount of resources to release inference tasks to the kernel function execution arbitrator.

[0061] The present invention proposes a service quality evaluator for real-time monitoring of the stability of the training task iteration cycle and timely preventing the training progress from being disturbed by inference execution.

[0062] The present invention proposes a kernel function execution arbitrator responsible for blocking or forwarding the CUDA kernel function of the inference instance to the GPU, supporting adaptive kernel function scheduling and effectively responding to online and offline inference tasks.

[0063] This paper proposes an inference memory expansion solution that uses managed memory to expand the available memory for inference tasks, thereby optimizing the inference batch size and maximizing the inference throughput.

Claims

1. A dynamic collaborative system for deep learning training and reasoning tasks based on GPU spatiotemporal resource sharing, characterized by: include: GPU resource status sensor, kernel function dynamic scheduling decision maker, and kernel function execution arbitrator, including: The GPU resource status sensor is used to monitor the GPU kernel function call sequence and memory allocation status of distributed training tasks in real time, dynamically capture the computing gaps and memory fragmentation generated by training tasks, generate a two-dimensional resource spatiotemporal feature map, and predict GPU computing idle cycles caused by communication synchronization based on the LSTM model; The kernel function dynamic scheduling decision maker is used to prioritize and dynamically allocate resource quotas for online and offline reasoning tasks based on the spatiotemporal feature map of resources, adopting an adaptive allocation strategy, thereby achieving spatiotemporal resource decoupling of training and reasoning tasks. The kernel function execution arbiter is used to dynamically control the submission and blocking of the inference task kernel function according to the scheduling decision through the memory space multiplexing and computing instruction arbitration mechanism.

2. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 1 is characterized in that: The specific implementation method of the GPU resource status sensor is: (1) The GPU resource status sensor monitors the CUDA kernel function calls sent to the GPU by the distributed training task through the mounted monitoring library, and tracks the execution status of the CUDA kernel function calls in real time; the GPU resource status sensor records the monitored CUDA kernel function call counts at fixed time intervals, and the CUDA kernel function call count data is stored in a sliding window, which is continuously updated to always keep the latest kernel function call information; The GPU resource status sensor analyzes the consecutive zero call counts in the sliding window, that is, the GPU does not receive the call cycle of the CUDA kernel function within the set time, and the time period of consecutive zero calls is regarded as the computing idle time T_past that has occurred; (2) The GPU resource status sensor hijacks the kernel function information of all video memory allocation related calls in the training task through the mounted monitoring library, and thus counts and records the remaining video memory fragmentation information of the current GPU; (3) Based on the historical statistical data of GPU busy-idle, an LSTM time series prediction model is constructed. The historical statistical data is standardized and then input into a two-layer LSTM network. The 64 LSTM units of the first layer LSTM network learn short-term call patterns, and the 32 LSTM units of the second layer LSTM network capture long-term dependencies. Finally, the idle probability distribution of the future time window is output through the fully connected layer. The dynamic threshold method is used to analyze the output probability, and the maximum continuous idle period is used as the expected computing idle time. The expected computing idle time T_pred in the current cycle is obtained by inference. (4) The GPU resource status sensor performs spatiotemporal correlation modeling on the analyzed computing idle time, expected computing idle time, and GPU memory fragmentation information to generate a two-dimensional resource spatiotemporal feature map, in which the time dimension represents the temporal changes in computing resource availability, and the spatial dimension depicts the fragmented distribution of memory resources; the two-dimensional resource spatiotemporal feature map is then sent together with the original analysis data to the kernel function dynamic scheduling decision maker.

3. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 1 is characterized in that: The kernel function dynamic scheduling decision maker uses an adaptive allocation strategy based on the spatiotemporal feature map of resources to prioritize and dynamically allocate resource quotas for online and offline inference tasks. The specific method for achieving spatiotemporal resource decoupling of training and inference tasks is as follows: (1) The kernel function dynamic scheduling decision maker continuously receives monitoring data from the GPU resource status perception module; (2) Dynamic resource allocation: The total number of tokens that can be allocated in the current cycle, Q_total, is determined by weighted summing the accumulated idle time T_past monitored in real time and the remaining idle time T_pred predicted by the LSTM model: Q_total=(1-T_past / T_pred)*SYSTEM_LIMIT Among them, SYSTEM_LIMIT is the token upper limit value; The final number of tokens that can be allocated is: min(Q_total, max(M / S_online, F / S_offline)), where S_online and S_offline represent the amount of video memory required for a single online and offline inference task, respectively, and M_offline and F_offline represent the maximum continuous available video memory block size and the total available video memory, respectively; (3) The service quality evaluator monitors the stability of the training task iteration cycle in real time. When it detects performance fluctuations or throughput attenuation exceeding a preset threshold, it reduces the expansion factor c according to the exponential α attenuation law. The initial expansion factor c is set to 1, and the new expansion factor c is the original expansion factor multiplied by α. If the throughput returns to its original performance, the new expansion factor c is the original expansion factor divided by α. During the operation, the total number of tokens available for the inference task is always maintained at Q_available = Q_total*c. (4) When there are pending requests in the high-priority request queue of the online reasoning task, the reasoning task type signal S is set to 1, indicating that the total number of tokens Q_available is allotted to the online reasoning task; only when the online reasoning queue is empty, the reasoning task type signal S is set to 0, indicating that the total number of tokens Q_available that can be allocated in the current cycle is allocated to the offline low-priority reasoning task; (5) The kernel function dynamic scheduling decision maker sends the token information finally assigned to the inference task and the inference task type signal S to the kernel function execution arbitrator component, which is responsible for the specific execution.

4. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 3 is characterized in that: The monitoring data includes: the accumulated computing idle time T_past, the remaining computing idle time T_pred predicted based on the LSTM model, and video memory fragmentation information.

5. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 1 is characterized in that: The kernel function execution arbiter uses the memory space multiplexing and computing instruction arbitration mechanism to dynamically control the submission and blocking of the inference task kernel function according to the scheduling decision. The specific method is as follows: (1) Determine the optimal batch size: When the training task completes gradient synchronization and parameter update, and the reference counter returns to zero, immediately mark these free memory areas as available, update the memory status, and notify the kernel function dynamic scheduling decision maker through the message mechanism; After receiving the notification, the scheduler calculates the optimal batch size B based on the current new memory status opt , the formula is as follows: Among them, M free The sum of the video memory size temporarily available for inference after the reference count is reset to zero and the remaining allocatable video memory size of the graphics card after the gradient synchronization and parameter update are completed for training; M ovh represents the fixed memory overhead that cannot be released on the inference side, m is the average memory usage added during the forward process of processing an inference sample, and W is the idle time window reserved for training. is the video memory constraint, which indicates the maximum number of samples that can be accommodated in the currently available video memory; is a time constraint, indicating the maximum number of samples that can be reasoned within the idle time window; Batch selection: Take the smaller value of the two, and then ensure that it is at least 1 to get the optimal batch size B opt ; The kernel function dynamic scheduling decision maker assigns the updated final scheduling decision to the token information of the inference task and the inference task type signal S to the kernel function execution arbitrator component; When a new training cycle begins, the video memory resources are gradually reclaimed through the reference counting mechanism; The reference counter is reset to 1, triggering the memory recycling process; the arbitrator suspends the submission of new inference tasks and waits for the ongoing tasks to complete; (2) For online inference tasks, when the high-priority queue is not empty, that is, the inference task type signal S is 1, the kernel function dynamic scheduling decision maker detects that (T_pred-T_past)>T_exec, where T_exec represents the execution time of a single batch request inference. The kernel function execution arbitrator actively pulls online inference requests one by one from the request queue to form the optimal batch size, and then immediately executes inference, consuming the corresponding number of tokens. (3) For offline inference tasks, that is, when the high-priority queue is empty and the inference task type signal S is 0, the kernel function execution arbitrator receives the token Q_available allocated by the kernel function dynamic scheduling decision maker, controls the submission of the CUDA kernel function of the inference task, and the execution of each CUDA kernel function consumes a corresponding number of tokens. The batch size will also be optimized in advance. When the tokens are insufficient, the kernel function execution arbitrator component blocks the initiation of subsequent inference kernel functions to ensure that there is no conflict with the training task. When the training task resumes, the kernel function dynamic scheduling decision maker reduces the number of tokens to zero, and the kernel function execution arbitrator component automatically stops the execution of the inference task.

6. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 1 is characterized in that: The GPU resource status sensor and kernel function execution arbitrator components are arranged in the kernel control layer.

7. The deep learning training and reasoning task dynamic collaboration system based on GPU spatiotemporal resource sharing according to claim 1 is characterized in that: The kernel function dynamic scheduling decision maker is set in the kernel coordination layer.

Images