A heterogeneous computing oriented task partitioning execution and spill recovery system
Patent Information
- Application Number
- CN202610912591.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-24
- Publication Date
- 2026-08-28
AI Technical Summary
[0003]但是现有技术在任务溢出缓解与资源回收方面存在以下不足：1.溢出检测滞后：现有系统通常在任务失败后才触发溢出缓解措施，缺乏提前检测与预防机制
[0065]1、本发明中，通过实时感知硬件资源状态动态调整任务分片策略，让不同架构的GPU和CPU资源得到精准适配与高效利用。系统内置的任务回收机制可在任务遭遇溢出或异常中断时自动保存关键计算状态，结合智能重试策略快速恢复任务执行，降低因单点故障导致全部计算进度丢失的风险，保障长周期复杂任务的连续性与稳定性，即使在硬件资源有限的环境下也能支撑超大模型任务的稳定推进。
Smart Images
Figure CN122653786A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer processing task scheduling technology, specifically a task fragmentation execution and overflow recovery system for heterogeneous computing. Background Technology
[0002] In scenarios such as deep learning, scientific computing, and real-time rendering, GPUs and NPUs, as the core of high-performance computing, suffer from limited resources. Under high load or extremely large task scenarios, this can lead to problems such as memory overflow, task queue backlog, and computation failures. This article describes resource overflow states including memory overflow, task queue overflow, and request timeout. Task queue overflow typically refers to a situation where, during a sudden surge in traffic or when the computing power of a local node is exhausted, a large number of tasks exceeding the system's preset processing capacity accumulate in the scheduling queue, leading to a dangerous state of memory exhaustion or request timeout. To mitigate this crisis, the system employs dynamic load balancing and backpressure mechanisms. By monitoring node load levels in real time, overflowing tasks are quickly routed or migrated to idle nodes. Combined with degradation strategies, request dropping, or token bucket rate limiting, traffic surges are suppressed at the source. After successfully containing the overflow, a resource reclamation mechanism is activated. Through precise scanning and reference counting, redundant memory, GPU memory, and file handles occupied by task backlog are released. Simultaneously, invalid cached objects and zombie threads are cleaned up to reduce the risk of memory leaks.
[0003] However, existing technologies have the following shortcomings in task overflow mitigation and resource reclamation: 1. Lagging overflow detection: Existing systems typically only trigger overflow mitigation measures after a task fails, lacking early detection and prevention mechanisms. 2. Limited task execution strategies: For tasks exceeding resource capacity, there is no way to flexibly adjust execution strategies (such as sharding or phased execution). 3. High task failure rate: Under high load or insufficient resources, tasks are prone to failure due to hardware limitations, lacking task reclamation and retry mechanisms.
[0004] Therefore, a runtime task overflow mitigation and recycling mechanism is needed to improve the system's task execution efficiency and reliability through sliced execution, overflow recycling, and hardware acceleration optimization. Summary of the Invention
[0005] The purpose of this invention is to provide a task fragmentation execution and overflow recovery system for heterogeneous computing in order to solve the problems mentioned above.
[0006] The technical solution adopted in this invention is as follows: a task sharding execution and overflow recovery system for heterogeneous computing, comprising: a resource monitoring module, a task sharding module, an overflow mitigation module, a task recovery module, and a hardware optimization module;
[0007] The real-time status output of the resource monitoring module is connected to the hardware status input of the task sharding module, and the overflow warning output of the resource monitoring module is connected to the trigger signal input of the overflow mitigation module.
[0008] The task sharding module's sharding configuration output is connected to the hardware optimization module's task scheduling input, and the task sharding module's execution progress output is connected to the task recycling module's status tracking input.
[0009] The granularity adjustment instruction output of the overflow mitigation module is connected to the parameter correction input of the task sharding module; the snapshot trigger instruction output of the overflow mitigation module is connected to the snapshot control input of the task recycling module; the resource usage feedback output of the hardware optimization module is connected to the performance data input of the resource monitoring module.
[0010] The task recovery instruction output of the task recycling module is connected to the restart control input of the task sharding module, and the historical interrupt data output of the task recycling module is connected to the training data input of the resource monitoring module.
[0011] In a preferred embodiment, the resource monitoring module collects three core indicators in real time—GPU memory usage, core computing load, and memory bandwidth utilization—every 100 milliseconds. It sets a dynamic adjustment threshold of 85% memory usage and an overflow warning threshold of 90%. When three consecutive samples exceed the overflow warning threshold, a pre-scheduling signal is immediately sent to the overflow mitigation module. When a single sample exceeds the dynamic adjustment threshold, the granularity adjustment command of the task sharding module is directly triggered. The module has a built-in hardware characteristic recognition engine that automatically adapts to the parameter acquisition logic of different GPU architectures such as A100 and RTX4090. For the HBM2 memory architecture, the sampling accuracy is improved to 50 milliseconds, and for the GDDR6 memory architecture, the bandwidth calculation algorithm is optimized, keeping the data acquisition latency within 10 milliseconds, providing other modules with accurate and low-latency hardware status information.
[0012] In a preferred embodiment, the task sharding module is composed of four core units working together: resource awareness, strategy decision-making, execution scheduling, and overhead optimization. The resource awareness unit collects dynamic status data such as hardware memory capacity and computing core load in real time. The strategy decision-making unit calculates the optimal sharding granularity and number of groups based on task type and hardware parameters. The execution scheduling unit is responsible for the loading order, data flow, and priority scheduling of sharded tasks. The overhead optimization unit minimizes the additional overhead between shards through techniques such as time-sharing of memory and multi-level cache preloading, ultimately achieving efficient phased execution of ultra-large tasks under limited hardware resources.
[0013] In a preferred embodiment, the task sharding module first obtains parameters such as the available video memory capacity and computing core frequency of a single GPU (e.g., an A100 GPU) through hardware resource awareness. Combined with the total number of model layers, single-parameter video memory utilization, and runtime cache overhead, it calculates the optimal number of model layer shard groups. In this way, a complete model requiring approximately 75GB of video memory after quantization or compression is split into four shard groups. Each shard group requires only approximately 22GB of video memory to load and execute, enabling large model inference on a single GPU.
[0014] The formula for calculating the number of model layer fragments is as follows:
[0015]
[0016] In the formula:
[0017] G represents the number of model layer slice groups;
[0018] M represents the total number of layers in the model to be executed;
[0019] R represents the available video memory capacity of a single compute node;
[0020] This represents the average video memory usage coefficient for single-model layer parameters and runtime cache.
[0021] In a preferred embodiment, the task sharding module monitors memory usage and computational load changes in real time. It utilizes memory time-sharing technology to allow different shard groups to share a portion of the memory space, while leveraging the GPU's multi-level caching mechanism to reduce data transfer latency between shards. The module dynamically adjusts the timing of data preloading between shards; when the previous shard group has reached the last 10% of its execution progress, it preloads the parameters of the next shard group into the cache's accessible area, reducing the waiting time during shard switching. In experimental scenarios, this optimization strategy can reduce the total overhead of shard execution by approximately 40%, increasing single-card inference throughput to approximately 1.8 times that of the original solution.
[0022] The optimization objective formula for the total overhead of fragmented execution is:
[0023]
[0024] In the formula:
[0025] O(S) represents the total execution overhead under the sharding strategy S;
[0026] S is the set of sharding strategies;
[0027] This is the weighting coefficient for data transmission overhead;
[0028] This represents the total time for cross-shard data transfer under sharding strategy S;
[0029] This represents the total computation context switching time under the sharding strategy S.
[0030] In a preferred embodiment, the task sharding module adaptively adjusts the sharding granularity based on task priority and remaining resources. For high-priority real-time dialogue requests, a smaller sharding granularity is used to reduce single-request latency; for low-priority batch text generation tasks, shard groups are merged to improve overall throughput. Simultaneously, the module dynamically adjusts the weighting coefficients of the sharding strategy based on historical execution data, ensuring optimal system performance under different load scenarios.
[0031] The formula for dynamically adjusting the priority of fragment granularity is:
[0032]
[0033] In the formula:
[0034] Rate the priority of the i-th task request;
[0035] Calculate the weighting coefficients for the complexity of the task;
[0036] Score the relative computational complexity of the i-th task request;
[0037] The weighting coefficient represents the degree of data dependency in the task.
[0038] The relative data dependency score for the i-th task request.
[0039] In a preferred embodiment, the overflow mitigation module uses the memory usage exceeding a warning threshold as a trigger condition. It dynamically adjusts the fragment granularity adjustment coefficient δ based on the overflow level. When the overflow is within 10%, δ is 0.1; between 10% and 20%, it is 0.2; and above 20%, it is 0.3. This is achieved through a formula... The new number of fragment groups is calculated, and a temporary memory release mechanism is initiated, prioritizing the reclamation of intermediate result caches from completed fragments, with a release ratio of no less than 15% of the current memory usage. When five consecutive sampling data show that the overflow state has not been alleviated, the module automatically triggers the status snapshot generation instruction of the task reclamation module. The snapshot compression ratio is set to 0.6 by default and can be dynamically adjusted according to storage overhead and recovery speed, ensuring that the snapshot generation time does not exceed 5% of the fragment execution time, and the retry priority coefficient γ is set to 0.7 to prioritize the rapid recovery of high-completion tasks.
[0040] In a preferred embodiment, after each shard group is executed, the task recycling module automatically saves the current model's hidden layer state, KV cache, necessary attention intermediate states, and prefix cache of the generated sequence to the NVMe high-speed storage medium in the form of incremental snapshots. When a task is at risk of interruption due to memory overflow or hardware failure, the module triggers a state capture mechanism, prioritizing the saving of input tensors of incomplete shards and recoverable intermediate computation results. If a hardware failure has occurred, execution is resumed from the most recent complete snapshot, reducing the risk of losing all computation progress due to interruption. The module achieves a balance between storage overhead and recovery speed by dynamically calculating the compression ratio of snapshot data, ensuring that snapshot generation time does not exceed 5% of the shard execution time.
[0041] The formula for the optimal triggering time of a state snapshot is:
[0042]
[0043] In the formula:
[0044] The optimal snapshot trigger time;
[0045] This is a storage overhead weighting coefficient;
[0046] S(t) represents the storage overhead of generating a snapshot at time t;
[0047] R(t) represents the computational cost of recovering from the snapshot at time t.
[0048] In a preferred embodiment, the task recovery module employs a differentiated intelligent retry strategy: if the interruption is due to memory overflow, it automatically analyzes the peak memory usage at the time of the overflow, adjusts the fragment granularity, and recalculates the optimal number of fragment groups; if the interruption is due to a temporary hardware anomaly, it directly resumes execution from the most recent snapshot, skipping completed calculation steps. Simultaneously, the module uses a lightweight prediction model trained on historical interruption data to identify high-risk task nodes in advance, adding pre-snapshot points before fragments where interruption may occur, further reducing retry costs. In experimental scenarios, this can reduce the average task recovery time to within 30% of the original solution.
[0049] Among them, intelligent retry priority calculation
[0050]
[0051] In the formula:
[0052] Prioritize task retrying;
[0053] This is a weighting coefficient for task completion.
[0054] This represents the percentage of task completion progress.
[0055] Score the degree of task data dependency;
[0056] The formula for adjusting the granularity after overflow is:
[0057]
[0058] In the formula:
[0059] This is the adjusted number of fragment groups;
[0060] This represents the original number of fragment groups;
[0061] This is the overflow adjustment factor;
[0062] This represents the percentage of video memory overflow.
[0063] In a preferred embodiment, the hardware optimization module internally incorporates a multi-level preloading mechanism. Through a video memory prefetch buffer, an asynchronous data transfer channel, and the GPU cache's accessible area, it preloads the computational core data of the three most recently used segments and the input tensors of the next two segments. The preloading trigger is set when the current segment's execution progress reaches 70%. For CPU and GPU collaborative computing scenarios, the module sets a data transfer bandwidth threshold of 10GB / s. When the actual transfer bandwidth falls below this threshold, a data compression transfer algorithm is automatically activated, with the compression ratio dynamically adjusted between 0.4 and 0.8 to balance transfer speed and data volume.
[0064] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0065] 1. In this invention, the task sharding strategy is dynamically adjusted by real-time sensing of hardware resource status, enabling precise adaptation and efficient utilization of GPU and CPU resources of different architectures. The system's built-in task recycling mechanism can automatically save critical computing states when a task encounters overflow or abnormal interruption, and quickly resume task execution in combination with an intelligent retry strategy, reducing the risk of losing all computing progress due to a single point of failure, ensuring the continuity and stability of long-cycle complex tasks, and supporting the stable progress of ultra-large model tasks even in environments with limited hardware resources.
[0066] 2. In this invention, multi-module collaborative operation improves task execution efficiency. The overflow mitigation module provides early warnings and dynamically adjusts the granularity of data sharding, reducing overflow risks at the source. The hardware optimization module customizes caching strategies and data transmission mechanisms for different hardware architectures, further reducing the overhead of computation and data flow. The entire execution process reduces the need for manual intervention in parameter configuration. The system automatically completes resource scheduling, state saving, and fault recovery, reducing the operational complexity of heterogeneous computing tasks. This allows users to focus on the task itself rather than underlying resource management, providing efficient and reliable automated execution support for various heterogeneous computing scenarios. Attached Figure Description
[0067] Figure 1 This is an overall system block diagram of the present invention;
[0068] Figure 2 This is a schematic diagram comparing the performance indicators before and after optimization of the 70B-parameter large language model for online inference in this invention.
[0069] Figure 3 This is a schematic diagram comparing the performance indicators before and after optimization in the multi-task concurrent computational fluid dynamics simulation of this invention;
[0070] Figure 4 This is a schematic diagram comparing the metrics before and after real-time rendering optimization of 8K game scenes in this invention. Detailed Implementation
[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0072] Reference Figure 1 Example 1: A task sharding execution and overflow recovery system for heterogeneous computing, comprising: a resource monitoring module, a task sharding module, an overflow mitigation module, a task recovery module, and a hardware optimization module;
[0073] The real-time status output of the resource monitoring module is connected to the hardware status input of the task sharding module, and the overflow warning output of the resource monitoring module is connected to the trigger signal input of the overflow mitigation module.
[0074] The task sharding module's sharding configuration output is connected to the hardware optimization module's task scheduling input, and the task sharding module's execution progress output is connected to the task recycling module's status tracking input.
[0075] The granularity adjustment instruction output of the overflow mitigation module is connected to the parameter correction input of the task sharding module; the snapshot trigger instruction output of the overflow mitigation module is connected to the snapshot control input of the task recycling module; the resource usage feedback output of the hardware optimization module is connected to the performance data input of the resource monitoring module.
[0076] The task recovery instruction output of the task recycling module is connected to the restart control input of the task sharding module, and the historical interrupt data output of the task recycling module is connected to the training data input of the resource monitoring module.
[0077] The resource monitoring module collects three core indicators in real time: GPU memory usage, core computing load, and memory bandwidth utilization, with a sampling period of 100 milliseconds. It sets memory usage, 85% as a dynamic adjustment threshold, and 90% as an overflow warning threshold. When the sampled data exceeds the overflow warning threshold three times in a row, a pre-scheduling signal is immediately sent to the overflow mitigation module. When the sampled data exceeds the dynamic adjustment threshold in a single instance, the granularity adjustment instruction of the task sharding module is directly triggered.
[0078] The task sharding module is composed of four core units working together: resource awareness, strategy decision-making, execution scheduling, and overhead optimization. The resource awareness unit collects dynamic status data such as hardware memory capacity and computing core load in real time. The strategy decision-making unit calculates the optimal sharding granularity and number of groups based on task type and hardware parameters. The execution scheduling unit is responsible for the loading order, data flow, and priority scheduling of sharded tasks. The overhead optimization unit minimizes the additional overhead between shards through techniques such as time-sharing of memory and multi-level cache preloading, ultimately achieving efficient phased execution of ultra-large tasks under limited hardware resources.
[0079] The task sharding module first obtains parameters such as the available video memory capacity and computing core frequency of a single GPU (e.g., A100) through hardware resource awareness. Combined with the total number of model layers, single-parameter video memory usage and runtime cache overhead, it calculates the optimal number of model layer shard groups. In this way, the complete model, which requires about 75GB of video memory after quantization or compression, is split into 4 shard groups, each of which only requires 22GB of video memory to load and execute, realizing large model inference on a single card.
[0080] The formula for calculating the number of model layer fragments is as follows:
[0081]
[0082] In the formula:
[0083] G represents the number of model layer slice groups;
[0084] M represents the total number of layers in the model to be executed;
[0085] R represents the available video memory capacity of a single compute node;
[0086] This represents the average video memory usage coefficient for single-model layer parameters and runtime cache.
[0087] The task sharding module monitors memory usage and computational load changes in real time. It uses memory time-sharing technology to allow different shard groups to share part of the memory space, and uses the GPU multi-level caching mechanism to reduce data transfer latency between shards. The module dynamically adjusts the timing of data preloading between shards. When the previous shard group has reached the last 10% of its progress, the parameters of the next shard group are preloaded into the cacheable area to reduce the waiting time when switching shards.
[0088] The optimization objective formula for the total overhead of fragmented execution is:
[0089]
[0090] In the formula:
[0091] O(S) represents the total execution overhead under the sharding strategy S;
[0092] S is the set of sharding strategies;
[0093] This is the weighting coefficient for data transmission overhead;
[0094] This represents the total time for cross-shard data transfer under sharding strategy S;
[0095] This represents the total computation context switching time under the sharding strategy S.
[0096] The task sharding module adaptively adjusts the sharding granularity based on task priority and remaining resources. For high-priority real-time dialogue requests, a smaller sharding granularity is used to reduce single-request latency. For low-priority batch text generation tasks, shard groups are merged to improve overall throughput. At the same time, the module dynamically adjusts the weight coefficient of the sharding strategy based on historical execution data, so that the system maintains better execution efficiency under different load scenarios.
[0097] The formula for dynamically adjusting the priority of fragment granularity is:
[0098]
[0099] In the formula:
[0100] Rate the priority of the i-th task request;
[0101] Calculate the weighting coefficients for the complexity of the task;
[0102] Score the relative computational complexity of the i-th task request;
[0103] The weighting coefficient represents the degree of data dependency in the task.
[0104] The relative data dependency score for the i-th task request.
[0105] The overflow mitigation module is triggered when the video memory usage exceeds a warning threshold. It dynamically adjusts the fragment granularity adjustment coefficient δ based on the overflow level. When the overflow is within 10%, δ is 0.1; between 10% and 20%, it is 0.2; and above 20%, it is 0.3. This is achieved through a formula... Calculate the new number of fragment groups and simultaneously activate the temporary memory release mechanism, prioritizing the reclamation of intermediate result caches that have completed fragmentation, with a release ratio of no less than 15% of the current memory usage; when the overflow state is not alleviated after five consecutive sampling data displays, the module automatically triggers the state snapshot generation instruction of the task reclamation module.
[0106] After each slice group is completed, the task recycling module automatically saves the current model's hidden layer state, KVCache, necessary attention intermediate states, and the prefix cache of the generated sequence to NVMe high-speed storage media in the form of incremental snapshots. When a task is at risk of interruption due to memory overflow or hardware failure, the module triggers a state capture mechanism, prioritizing the saving of input tensors of incomplete slices and recoverable intermediate computation results. If a hardware failure has occurred, execution is resumed from the most recent complete snapshot, reducing the risk of losing all computation progress due to interruption. The module achieves a balance between storage overhead and recovery speed by dynamically calculating the compression ratio of snapshot data, ensuring that snapshot generation time does not exceed 5% of the slice execution time.
[0107] The formula for the optimal triggering time of a state snapshot is:
[0108]
[0109] In the formula:
[0110] The optimal snapshot trigger time;
[0111] This is a storage overhead weighting coefficient;
[0112] S(t) represents the storage overhead of generating a snapshot at time t;
[0113] R(t) represents the computational cost of recovering from the snapshot at time t.
[0114] The task recovery module employs a differentiated intelligent retry strategy: if the interruption is due to memory overflow, it automatically analyzes the peak memory usage at the time of the overflow, adjusts the fragment granularity, and recalculates the optimal number of fragment groups; if the interruption is due to a temporary hardware anomaly, it directly resumes execution from the most recent snapshot, skipping the completed calculation steps; at the same time, the module uses a lightweight prediction model trained on historical interruption data to identify high-risk task nodes in advance, adding pre-snapshot points before fragments that may be interrupted, further reducing retry costs. In experimental scenarios, the average task recovery time can be reduced to less than 30% of the original solution.
[0115] Among them, intelligent retry priority calculation
[0116]
[0117] In the formula:
[0118] Prioritize task retrying;
[0119] This is a weighting coefficient for task completion.
[0120] This represents the percentage of task completion progress.
[0121] Score the degree of task data dependency;
[0122] The formula for adjusting the granularity after overflow is:
[0123]
[0124] In the formula:
[0125] This is the adjusted number of fragment groups;
[0126] This represents the original number of fragment groups;
[0127] This is the overflow adjustment factor;
[0128] This represents the percentage of video memory overflow.
[0129] The hardware optimization module has a multi-level preloading mechanism. It preloads the computational core data of the three most recently used segments and the input tensors of the next two segments through the video memory prefetch buffer, asynchronous data transfer channel, and GPU cache access area. The preloading trigger is set when the current segment execution progress reaches 70%. For CPU and GPU collaborative computing scenarios, the module sets the data transmission bandwidth threshold to 10GB / s. When the actual transmission bandwidth is lower than the threshold, the data compression transmission algorithm is automatically enabled, and the compression ratio is dynamically adjusted between 0.4 and 0.8 to balance transmission speed and data volume.
[0130] Example 2: Online inference using a 70B parameter large language model, specifically including:
[0131] Resource monitoring module: It collects three core indicators of A100 GPU simultaneously, with a sampling period of 50 milliseconds: memory usage, core computing load, and memory bandwidth. It sets a warning threshold of 85% memory usage and an auxiliary trigger threshold of 80% core load. When the memory data exceeds the threshold for three consecutive samplings, or when the core load exceeds the threshold for a single time, it immediately sends a scheduling trigger signal to the task sharding module and synchronizes the real-time hardware status data to the hardware optimization module.
[0132] Task sharding module: The original 32 inference requests in a single batch are split into 8 independent mini-batches, each containing 4 inference requests. The weight parameters are split according to the hierarchical structure of the Transformer model, with each layer of weights being approximately 1.2GB. A pipeline mode of "load-inference-release" is adopted, releasing the corresponding weight memory immediately after completing the inference of a layer, reserving space for loading the weight of the next layer.
[0133] Hardware optimization module: Enables time-sharing memory reuse mechanism, sets 200 milliseconds as time slice period, and processes weight loading and current batch inference calculation in parallel within each time slice, with memory reuse rate maintained at around 75%; optimizes data reading path for HBM2 memory architecture, reduces weight loading latency by 20%, and further compresses the overall time consumption of single batch inference.
[0134] Example 3: Multi-task concurrent computational fluid dynamics simulation specifically includes:
[0135] Resource monitoring module: With a sampling period of 100 milliseconds, it monitors three indicators of RTX4090 GPU in real time: core computing load, memory bandwidth usage, and video memory utilization. It sets 90% core load as the trigger threshold and 95% memory bandwidth as the warning threshold. When the core load exceeds the threshold for two consecutive samplings, it triggers a task sharding instruction. If the memory bandwidth also exceeds the warning threshold, it simultaneously notifies the overflow mitigation module to start the load balancing strategy.
[0136] Task partitioning module: The original single large-scale fluid simulation task is split into 6 independent subtasks. Each subtask corresponds to 1 / 6 of the fluid mesh region for calculation. The mesh partitioning algorithm ensures that the computational error of each subtask is controlled within 5%. Only the dependency relationship of boundary data is retained between subtasks, and parallel or serial scheduling is supported.
[0137] Overflow mitigation module: Sets a dynamic scheduling window for the three subtasks in the non-core flow field region, with an initial execution delay of 1500 milliseconds. At the same time, it monitors the core load changes at a period of 500 milliseconds. When the core load drops below 70%, it automatically shortens the delay time to 500 milliseconds. If the load remains below 60%, it cancels the delay and executes directly. In case of tight memory bandwidth, the data transmission batch of subtasks is adjusted from 8MB to 4MB to reduce the bandwidth consumption of a single data transmission.
[0138] Hardware optimization module: Enables CPU-GPU collaborative computing mode, where the CPU completes the preprocessing of grid data and boundary condition calculation, while the GPU is responsible for the core flow field numerical simulation. High-speed data transmission is achieved through the PCIe 4.0 bus, with a transmission bandwidth threshold of 10GB / s. At the same time, the GPU L2 cache preloading mechanism is enabled to preload the data required by the subtasks into the cache access area, reducing data read latency.
[0139] Example 4: Real-time rendering of 8K game scenes specifically includes:
[0140] Resource monitoring module: It monitors three metrics of RTX4090 GPU: memory usage, texture loading bandwidth, and core rendering load, with a sampling period of 100 milliseconds. It sets 90% memory usage as the overflow warning threshold and 80% texture loading bandwidth as the auxiliary threshold. When the memory data of a single sample exceeds the threshold, it immediately triggers the frame block splitting instruction of the overflow mitigation module. If the texture loading bandwidth is simultaneously lower than the threshold, it notifies the hardware optimization module to enable the texture compression strategy.
[0141] Overflow mitigation module: The original single-frame complete rendering task is split into 16 independent frame blocks of approximately 1920×1080 pixels. The scheduling order is based on scene depth, prioritizing the rendering of frame blocks in the center of the player's field of view. After the rendering calculation of a frame block is completed, the texture data and rendering cache in the corresponding video memory are released immediately, with a release ratio of no less than 15% of the current video memory usage, to ensure that there is enough video memory space for texture loading of subsequent frame blocks.
[0142] Hardware optimization module: Enables the BC7 texture compression algorithm, sets the compression ratio of scene textures to 0.6 by default, and compresses the texture file size to 60% of the original size. At the same time, it optimizes the texture loading pipeline, and performs texture decoding and frame block rendering calculations in parallel with an overlap time of about 100 milliseconds. GPU ray tracing acceleration is enabled for frame block rendering, which can reduce the time of single-frame ray tracing calculation by 30% in experimental scenes.
[0143] From the above, we can conclude that:
[0144] In this invention, the task sharding strategy is dynamically adjusted by sensing the hardware resource status in real time, enabling precise adaptation and efficient utilization of GPU and CPU resources of different architectures. The system's built-in task recycling mechanism can automatically save critical computing states when a task encounters overflow or abnormal interruption, and quickly resume task execution in combination with an intelligent retry strategy. This reduces the risk of losing all computing progress due to a single point of failure, ensuring the continuity and stability of long-cycle complex tasks, and supporting the stable progress of ultra-large model tasks even in environments with limited hardware resources.
[0145] In this invention, multi-module collaboration improves task execution efficiency. An overflow mitigation module provides early warnings and dynamically adjusts sharding granularity, reducing overflow risk at its source. A hardware optimization module customizes caching strategies and data transmission mechanisms for different hardware architectures, further reducing the overhead of computation and data transfer. The entire execution process reduces manual intervention in parameter configuration, with the system automatically handling resource scheduling, state saving, and fault recovery. This reduces the operational complexity of heterogeneous computing tasks, allowing users to focus on the task itself rather than underlying resource management, providing efficient and reliable automated execution support for various heterogeneous computing scenarios.
[0146] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A task fragmentation execution and overflow recycling system for heterogeneous computing, characterized in that: include: Resource monitoring module, task sharding module, overflow mitigation module, task recycling module, and hardware optimization module; The real-time status output of the resource monitoring module is connected to the hardware status input of the task sharding module, and the overflow warning output of the resource monitoring module is connected to the trigger signal input of the overflow mitigation module. The task sharding module's sharding configuration output is connected to the hardware optimization module's task scheduling input, and the task sharding module's execution progress output is connected to the task recycling module's status tracking input. The granularity adjustment instruction output of the overflow mitigation module is connected to the parameter correction input of the task sharding module; the snapshot trigger instruction output of the overflow mitigation module is connected to the snapshot control input of the task recycling module; the resource usage feedback output of the hardware optimization module is connected to the performance data input of the resource monitoring module. The task recovery instruction output of the task recycling module is connected to the restart control input of the task sharding module, and the historical interrupt data output of the task recycling module is connected to the training data input of the resource monitoring module.
2. The task fragmentation execution and overflow recycling system for heterogeneous computing as described in claim 1, characterized in that: The resource monitoring module collects three core indicators in real time: GPU memory usage, core computing load, and memory bandwidth utilization, with a sampling period of 100 milliseconds. It sets memory usage, 85% as a dynamic adjustment threshold, and 90% as an overflow warning threshold. When the sampled data exceeds the overflow warning threshold three times in a row, a pre-scheduling signal is immediately sent to the overflow mitigation module. When the sampled data exceeds the dynamic adjustment threshold in a single instance, the granularity adjustment instruction of the task sharding module is directly triggered.
3. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The task sharding module is composed of four core units working together: resource awareness, strategy decision-making, execution scheduling, and overhead optimization. The resource awareness unit collects dynamic status data such as hardware memory capacity and computing core load in real time. The strategy decision-making unit calculates the optimal sharding granularity and number of groups based on task type and hardware parameters. The execution scheduling unit is responsible for the loading order, data flow, and priority scheduling of sharded tasks. The overhead optimization unit minimizes the additional overhead between shards through technologies such as time-sharing of memory and multi-level cache preloading, enabling efficient phased execution of ultra-large tasks under limited hardware resources.
4. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The task sharding module first obtains parameters such as the available video memory capacity and computing core frequency of a single GPU (e.g., A100) through hardware resource awareness. Combined with the total number of model layers, single-parameter video memory usage and runtime cache overhead, it calculates the optimal number of model layer shard groups. In this way, the complete model, which requires about 75GB of video memory after quantization or compression, is split into 4 shard groups, each of which only requires 22GB of video memory to load and execute, realizing large model inference on a single card. The formula for calculating the number of model layer fragments is as follows: In the formula: G represents the number of model layer slice groups; M represents the total number of layers in the model to be executed; R represents the available video memory capacity of a single compute node; This represents the average video memory usage coefficient for single-model layer parameters and runtime cache.
5. A task fragmentation execution and overflow recycling system for heterogeneous computing as described in claim 1, characterized in that: The task sharding module monitors memory usage and computational load changes in real time. It uses memory time-sharing technology to allow different shard groups to share part of the memory space, and uses the GPU multi-level caching mechanism to reduce data transfer latency between shards. The module dynamically adjusts the timing of data preloading between shards. When the previous shard group has reached the last 10% of its progress, the parameters of the next shard group are preloaded into the cacheable area to reduce the waiting time when switching shards. The optimization objective formula for the total overhead of fragmented execution is: In the formula: O(S) represents the total execution overhead under the sharding strategy S; S is the set of sharding strategies; This is the weighting coefficient for data transmission overhead; This represents the total time for cross-shard data transfer under sharding strategy S; This represents the total computation context switching time under the sharding strategy S.
6. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The task sharding module adaptively adjusts the sharding granularity based on task priority and remaining resources. For high-priority real-time dialogue requests, a smaller sharding granularity is used to reduce single-request latency. For low-priority batch text generation tasks, shard groups are merged to improve overall throughput. At the same time, the module dynamically adjusts the weight coefficient of the sharding strategy based on historical execution data, so that the system maintains better execution efficiency under different load scenarios. The formula for dynamically adjusting the priority of fragment granularity is: In the formula: Rate the priority of the i-th task request; Calculate the weighting coefficients for the complexity of the task; Score the relative computational complexity of the i-th task request; The weighting coefficient represents the degree of data dependency in the task. The relative data dependency score for the i-th task request.
7. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The overflow mitigation module is triggered when the video memory usage exceeds a warning threshold. It dynamically adjusts the fragment granularity adjustment coefficient δ based on the overflow level. When the overflow is within 10%, δ is 0.1; between 10% and 20%, it is 0.2; and above 20%, it is 0.
3. This is achieved through a formula... Calculate the new number of fragment groups and simultaneously activate the temporary memory release mechanism, prioritizing the reclamation of intermediate result caches that have completed fragmentation, with a release ratio of no less than 15% of the current memory usage; when the overflow state is not alleviated after five consecutive sampling data displays, the module automatically triggers the state snapshot generation instruction of the task reclamation module.
8. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: After each shard group is completed, the task recycling module automatically saves the hidden layer state of the current model, KV Cache, necessary attention intermediate states, and prefix cache of the generated sequence to the NVMe high-speed storage medium in the form of incremental snapshots. When a task is at risk of interruption due to memory overflow or hardware failure, the module triggers a state capture mechanism. It prioritizes saving the input tensors of incomplete slices and recoverable intermediate computation results. If a hardware failure has occurred, it resumes execution from the most recent complete snapshot, reducing the risk of losing all computation progress due to interruption. The module dynamically calculates the compression ratio of snapshot data to achieve a balance between storage overhead and recovery speed, ensuring that snapshot generation time does not exceed 5% of slice execution time. The formula for the optimal triggering time of a state snapshot is: In the formula: The optimal snapshot trigger time; This is a storage overhead weighting coefficient; S(t) represents the storage overhead of generating a snapshot at time t; R(t) represents the computational cost of recovering from the snapshot at time t.
9. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The task recycling module adopts a differentiated intelligent retry strategy: if the task is interrupted due to memory overflow, it automatically analyzes the peak memory usage at the time of the overflow, adjusts the fragment granularity, and recalculates the optimal number of fragment groups. If the execution is interrupted due to a temporary hardware failure, it will resume directly from the most recent snapshot, skipping the completed calculation steps; at the same time, the module will use a lightweight prediction model trained on historical interruption data to identify high-risk task nodes in advance. Among them, intelligent retry priority calculation In the formula: Prioritize task retrying; This is a weighting coefficient for task completion. This represents the percentage of task completion progress. Score the degree of task data dependency; The formula for adjusting the granularity after overflow is: In the formula: This is the adjusted number of fragment groups; This represents the original number of fragment groups; This is the overflow adjustment factor; This represents the percentage of video memory overflow.
10. A task fragmentation execution and overflow recovery system for heterogeneous computing as described in claim 1, characterized in that: The hardware optimization module has a multi-level preloading mechanism. It preloads the computational core data of the three most recently used segments and the input tensors of the next two segments through the video memory prefetch buffer, asynchronous data transfer channel, and GPU cache access area. The preloading trigger is set when the current segment execution progress reaches 70%. For CPU and GPU collaborative computing scenarios, the module sets the data transmission bandwidth threshold to 10GB / s. When the actual transmission bandwidth is lower than the threshold, the data compression transmission algorithm is automatically enabled, and the compression ratio is dynamically adjusted between 0.4 and 0.8 to balance transmission speed and data volume.