A parallel test task scheduling method for a large-computing-power GPU chip

Abstract

The present application relates to the field of GPU chip testing, and specifically to a parallel test task scheduling method for high-computing-power GPU chips. The specific implementation process includes: constructing a test task-resource demand matrix, calculating the transient thermal stress generated by the test task and checking the test safety margin; according to the test safety margin, aggregating multiple test tasks into a parallel test execution cluster, planning a GPU test scheduling graph containing space mapping and execution time window for the parallel test execution cluster; setting a fault cause and effect traceability chain to analyze when the test fails, output the fault source context and troubleshoot the fault cause. By constructing the test task-resource demand matrix, using the fine scheduling method of planning first and executing later, and introducing active safety checking, the present application ensures the safety of the GPU chip while achieving efficient parallel testing of high-computing-power GPU chips, effectively improving the execution efficiency of the test task and the accuracy of fault location.

Description

Technical Field

[0001] This invention relates to the field of GPU chip testing, specifically to a method for scheduling parallel test tasks for high-performance GPU chips. Background Technology

[0002] In the field of high-performance GPU chip testing, to overcome the bottleneck of excessively long testing times associated with traditional serial testing methods, the industry has widely adopted parallel testing as the mainstream technical solution. This involves utilizing the thousands of parallel processing cores inherent in the GPU chip, and using parallel computing programming frameworks such as CUDA or OpenCL, transforming test cases that originally needed to be executed one by one into concurrent computing tasks. These concurrent test tasks are submitted to the GPU in batches, relying on a task scheduler for real-time resource allocation and execution management. This large-scale parallel processing significantly shortens the overall testing cycle of the chip.

[0003] However, existing technologies have inherent drawbacks in practice, such as a mismatch between general-purpose GPU schedulers and the specific needs of chip testing and verification. General-purpose schedulers are "blind" to the content and intent of test tasks, unable to optimize hardware resource allocation based on test objectives. This results in parallel efficiency falling far short of theoretical values ​​and easily causing uneven test coverage. The resulting uncertainty in the scheduling process makes fault reproduction and precise localization difficult when tests fail, severely impacting chip yield analysis and debugging efficiency.

[0004] To address this, a parallel test task scheduling method for high-performance GPU chips is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a parallel test task scheduling method for high-performance GPU chips, which enables parallel testing of GPU chips while ensuring the integrity of the GPU chips.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for scheduling parallel test tasks for high-performance GPU chips includes:

[0008] The test task set is parsed and a test task-resource requirement matrix is ​​constructed. The test task-resource requirement matrix contains the resource requirements of each test task for the physical execution units inside the GPU and the matching relationship with the test target.

[0009] Based on the GPU's real-time status, test objectives, and the test task-resource requirement matrix, the transient thermal stress generated by the test tasks on the GPU is calculated and the test safety margin is verified. Based on the test safety margin, multiple test tasks are aggregated into a parallel test execution cluster. A GPU test scheduling graph with spatial mapping from each test task to the GPU physical execution unit and execution window is planned for the parallel test execution cluster.

[0010] A fault cause-and-effect tracing chain is set up for the parallel test task, and the GPU test scheduling graph is associated and bound with the real-time execution log of the corresponding physical execution unit of the GPU. When the test task fails, the fault cause-and-effect tracing chain is used for parsing, the fault source context that caused the failure is output, and the cause of the failure is investigated.

[0011] Preferably, the specific implementation process of parsing the test task set and constructing the test task-resource requirement matrix includes:

[0012] The execution logic and resource requirement descriptions of all test tasks are extracted through syntax analysis. For the execution logic, the types of computing cores and storage resources required by the test task are identified and quantified into functional parameters. For the resource requirement descriptions, the GPU memory capacity and memory bandwidth required for task execution are obtained and quantified into constraints. The functional parameters and constraints are integrated to generate a resource requirement vector for the test task. A data matrix is ​​created with all test tasks as the first dimension and the schedulable physical execution units and resource types within the GPU as the second dimension. The resource requirement vectors corresponding to the test tasks are filled into the data matrix to construct a test task-resource requirement matrix.

[0013] Preferably, the specific implementation process for calculating the transient thermal stress on the GPU generated by the test task and verifying the test safety margin based on the GPU's real-time state, the test objective, and the test task-resource requirement matrix includes:

[0014] The GPU's real-time status, including the current temperature and power consumption baseline of the physical execution units, is obtained. Combined with the resource demand vector corresponding to the test task in the test task-resource demand matrix, the power consumption superposition value generated when executing the test task on each physical execution unit of the GPU is calculated. The power consumption superposition value is used as a heat source input for heat conduction calculation, and transient thermal stress is output. The GPU's operating temperature safety threshold is extracted from the test target, and the peak thermal stress in the transient thermal stress is calculated with the operating temperature safety threshold to obtain the test safety margin.

[0015] Preferably, the specific implementation process of aggregating multiple test tasks into a parallel test execution cluster according to the test safety margin includes:

[0016] With test safety margin as the core constraint, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters according to a preset scheduling strategy. The resource requirement vectors of all test tasks in the candidate set of parallel test execution clusters are updated based on the test task-resource requirement matrix. The transient thermal stress on the GPU is re-evaluated and the remaining test safety margin is verified. When the test safety margin still meets the preset conditions, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters. When the test safety margin does not meet the preset conditions, the expansion of the current candidate set of parallel test execution clusters is terminated and aggregated into a parallel test execution cluster.

[0017] Preferably, the specific implementation process of the GPU test scheduling graph for planning the spatial mapping of each test task to the GPU physical execution unit and the execution time window for the parallel test execution cluster includes:

[0018] Based on the test task-resource requirement matrix, a resource view containing the GPU topology and the real-time availability status of each physical execution unit is obtained. The resource requirements of each test task within the parallel test execution cluster for the physical execution units are extracted. The resource requirements are input into the resource view for fitting calculation to obtain a spatial mapping. The spatial mapping contains the GPU physical execution units to which each test task within the parallel test execution cluster should be loaded. An execution time interval is determined for each test task in the spatial mapping based on the data dependencies between test tasks, forming an execution window. The spatial mapping and the execution window are integrated to generate a GPU test scheduling graph containing complete scheduling instructions.

[0019] Preferably, the specific implementation process of setting up a fault causal chain for the parallel test task and associating the GPU test scheduling graph with the real-time execution logs of the corresponding physical execution units of the GPU includes:

[0020] Based on the GPU test scheduling graph, initialize the data structure of the fault causal tracing chain; when the test task starts to be executed, capture the execution log data stream generated from each physical execution unit of the GPU in real time; parse the execution log data stream and match causal nodes for each execution instance of the test task, the causal nodes record the task identifier, the predetermined physical execution unit mapping relationship and the execution time window information; associate and bind the successfully matched execution instances with the causal nodes to generate tracing information units containing scheduling intent and execution status; link all tracing information units in an orderly manner according to the execution time window order of the test tasks in the GPU test scheduling graph to construct the fault causal tracing chain.

[0021] Preferably, the specific implementation process of parsing the fault cause-and-effect chain when the test task fails, outputting the fault source context that caused the failure, and investigating the cause of the failure includes:

[0022] Upon receiving a test failure signal, the system retrieves and locates the end-point tracing information unit directly associated with the failure event in the fault causal tracing chain based on the timestamp carried in the fault signal. Starting from the end-point tracing information unit, the system performs a reverse tracing traversal along the fault causal tracing chain. During the reverse tracing traversal, the system extracts in batches the GPU real-time execution log fragments and corresponding GPU test scheduling graph information encapsulated by each preceding tracing information unit. The extracted real-time execution log fragments and scheduling graph information are then time-aligned and state-associated to reconstruct a continuous GPU execution state sequence before and after the failure point and output the fault source context. Anomaly detection is performed on the fault source context to investigate the cause of the failure.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. This invention analyzes a set of test tasks and constructs a "test task-resource requirement matrix" to accurately determine the resource requirements of each task on the GPU's internal physical execution units. Simultaneously, it comprehensively considers the GPU's real-time status and test objectives, efficiently aggregating multiple tasks into a "parallel test execution cluster." This fine-grained scheduling approach of "planning first, then executing" effectively improves parallel processing efficiency and shortens the overall chip testing cycle.

[0025] 2. This invention introduces the calculation and verification of "transient thermal stress" and "test safety margin" before task scheduling. By evaluating the thermal effects of parallel task clusters on the GPU, it ensures that the thermal effects do not exceed the GPU's safe operating temperature threshold. This effectively avoids physical damage or instability caused by chip overheating due to excessive test load, ensuring the safety of the testing process and the integrity of the chip.

[0026] 3. This invention associates and binds the GPU test scheduling graph with the real-time execution logs of physical execution units, matching a "causal node" to each execution instance of a test task and setting up a "fault causal tracing chain." When a test fails, the end information unit directly associated with the fault event can be quickly located in the tracing chain based on the timestamp of the fault signal. Simultaneously, by tracing back along the tracing chain, preceding related execution logs and scheduling information are extracted in batches, and the continuous GPU execution state sequence before and after the fault occurrence point is reconstructed. This effectively improves the accuracy and efficiency of fault location, providing strong support for chip yield analysis and debugging. Attached Figure Description

[0027] Figure 1 This is a flowchart of a parallel test task scheduling method for high-performance GPU chips proposed in this invention.

[0028] Figure 2 This is a schematic diagram of the parallel test execution cluster aggregation process proposed in this invention;

[0029] Figure 3 This is a schematic diagram illustrating the principle of troubleshooting faults based on the fault cause tracing chain proposed in this invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It must be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to constitute any limitation on the scope of protection of this invention. Therefore, all equivalent changes or modifications conceived by those skilled in the art based on the content disclosed in this invention without inventive effort should fall within the scope of protection claimed by this invention.

[0031] Reference Figures 1 to 3 This invention provides a method for scheduling parallel test tasks for high-performance GPU chips, the technical solution of which is as follows:

[0032] Example 1:

[0033] Reference Figure 1 This embodiment of a parallel test task scheduling method for high-performance GPU chips includes:

[0034] The test task set is parsed and a test task-resource requirement matrix is ​​constructed. The test task-resource requirement matrix contains the resource requirements of each test task for the physical execution units inside the GPU and the matching relationship with the test target.

[0035] Based on the GPU's real-time status, test objectives, and the test task-resource requirement matrix, the transient thermal stress generated by the test tasks on the GPU is calculated and the test safety margin is verified. Based on the test safety margin, multiple test tasks are aggregated into a parallel test execution cluster. A GPU test scheduling graph with spatial mapping from each test task to the GPU physical execution unit and execution window is planned for the parallel test execution cluster.

[0036] A fault cause-and-effect tracing chain is set up for the parallel test task, and the GPU test scheduling graph is associated and bound with the real-time execution log of the corresponding physical execution unit of the GPU. When the test task fails, the fault cause-and-effect tracing chain is used for parsing, the fault source context that caused the failure is output, and the cause of the failure is investigated.

[0037] Furthermore, the specific implementation process of parsing the test task set and constructing the test task-resource requirement matrix includes:

[0038] The execution logic and resource requirement descriptions of all test tasks are extracted through syntax analysis. For the execution logic, the types of computing cores and storage resources required by the test task are identified and quantified into functional parameters. For the resource requirement descriptions, the GPU memory capacity and memory bandwidth required for task execution are obtained and quantified into constraints. The functional parameters and constraints are integrated to generate a resource requirement vector for the test task. A data matrix is ​​created with all test tasks as the first dimension and the schedulable physical execution units and resource types within the GPU as the second dimension. The resource requirement vectors corresponding to the test tasks are filled into the data matrix to construct a test task-resource requirement matrix.

[0039] The test object is a new type of high-performance computing GPU, which integrates 32 streaming multiprocessors (SMs), each SM containing 64 CUDA cores, for a total of 2048 CUDA cores. The GPU is equipped with 16GB of video memory, with a theoretical memory bandwidth of 512GB / s. The test task set includes three independent test cases: a matrix multiplication test to verify the performance of the matrix multiplication unit, an FFT test to test the Fast Fourier Transform (FFT) function, and a high-load memory bandwidth stress test.

[0040] The system receives test tasks written in the CUDA parallel computing language. For the matrix multiplication test, logical analysis is performed using syntax analysis to identify that its core computational load mainly calls the GPU's CUDA cores for floating-point operations. By extracting thread blocks and grids from the kernel functions in the code, the configuration parameters are set to dimGrid (512, 1) and dimBlock (256, 1), thus quantifying that the task requires at least 131,072 concurrently scheduled threads when running at full load, which will consume a large amount of CUDA core resources. Simultaneously, by parsing its resource requirement description, it is identified that the task requires two 512MB floating-point matrices as input and generates a 512MB output matrix. Therefore, the core type requirement in its functional parameters is quantified as "CUDA core intensive," and the memory capacity requirement in its constraints is explicitly quantified as 1.5GB.

[0041] For the FFT test, the parsing function identified calls to specific signal processing instructions within its execution logic, which can be mapped to TensorCores within the GPU. Therefore, its functional parameters were quantified as having specific requirements for "TensorCores." Simultaneously, the data processing requirement of 512MB of video memory was precisely extracted and used as a constraint.

[0042] For the memory bandwidth stress test, the execution logic was identified as a large-scale, continuous data transfer operation with low demand on computing cores but extremely frequent access to storage resources. The resource requirement description specifies the need to continuously maintain a data transfer rate close to the chip's theoretical limit; therefore, the constraints were quantified as requiring at least 450GB / s of memory bandwidth and 8GB of video memory.

[0043] After quantifying the functional parameters and constraints of all test tasks, this discrete information is integrated to generate a standardized resource requirement vector for each test task. For example, the resource requirement vector for the matrix multiplication test can be represented as (CUDA core requirement: High, TensorCore requirement: None, VRAM capacity: 1.5GB, Memory bandwidth: 80GB / s). A data matrix, the test task-resource requirement matrix, is then created. The first dimension of this matrix consists of all test tasks, including three entries: matrix multiplication test, FFT test, and memory bandwidth pressure. The second dimension consists of all schedulable physical execution units and key resource types within the GPU, such as "CUDA cores," "TensorCores," "VRAM capacity (GB)," and "Memory bandwidth (GB / s)." Finally, the resource requirement vectors previously generated for each test task are precisely filled into the corresponding rows of the data matrix to construct a structured, quantifiable test task-resource requirement matrix. This matrix intuitively reflects the specific hardware resource requirements of each task in the entire test set. This provides an accurate and reliable data foundation for subsequent scheduling decisions, such as transient thermal stress calculation, construction of parallel execution clusters, and generation of fault causal chains.

[0044] This embodiment constructs a test task-resource requirement matrix, enabling the scheduling function to predict the specific impact of each task on hardware resources. This provides a foundation for subsequent efficient task aggregation and parallel scheduling, avoiding problems such as low test efficiency and uneven coverage caused by resource conflicts or overload, and effectively improving the overall efficiency and reliability of testing.

[0045] Furthermore, based on the GPU's real-time state, the test objective, and the aforementioned test task-resource requirement matrix, the specific implementation process for calculating the transient thermal stress generated by the test task on the GPU and verifying the test safety margin includes:

[0046] The GPU's real-time status, including the current temperature and power consumption baseline of the physical execution units, is obtained. Combined with the resource demand vector corresponding to the test task in the test task-resource demand matrix, the power consumption superposition value generated when executing the test task on each physical execution unit of the GPU is calculated. The power consumption superposition value is used as a heat source input for heat conduction calculation, and transient thermal stress is output. The GPU's operating temperature safety threshold is extracted from the test target, and the peak thermal stress in the transient thermal stress is calculated with the operating temperature safety threshold to obtain the test safety margin.

[0047] After constructing a precise test task-resource requirement matrix, the matrix multiplication test and FFT test are planned to be aggregated into a parallel test execution cluster. Before executing this decision, the real-time status of each physical execution unit of the GPU chip is first obtained through the onboard management controller interface. For example, by reading the built-in digital temperature sensor, the initial temperature of all 32 streaming multiprocessors (SMs) is found to be 45°C, the memory controller temperature is 48°C, the chip is in standby mode, the power consumption baseline of each SM is 5 watts, and the power consumption baseline of the memory system is 10 watts.

[0048] By combining the resource requirement vectors corresponding to matrix multiplication and FFT tests in the constructed test task-resource requirement matrix, and using a pre-established power model calibrated with chip power consumption characteristics, the power consumption summation on each physical execution unit of the GPU when these two tasks are executed in parallel is calculated. This power model can accurately map resource requirements, such as CUDA core utilization and memory bandwidth usage, to corresponding power consumption. According to the model calculation, the matrix multiplication test task, whose resource requirement vector shows it as "CUDA core intensive," will increase the power consumption of these 16 SM units by an additional 30 watts on top of the 5-watt baseline if executed on SM0 to SM15; while the FFT test task, due to its call to TensorCore, will increase the power consumption of these 8 SM units by an additional 22 watts if executed on SM16 to SM23. At this point, a non-uniform power consumption distribution map of the chip surface during parallel execution is obtained: the total power consumption of SM0 to SM15 is 35 watts, the total power consumption of SM16 to SM23 is 27 watts, and the remaining SMs and other units maintain the baseline power consumption. This distribution map represents the sum of power consumption values.

[0049] The power consumption summation value was used as the heat source input, and a built-in chip-level thermal conduction calculation model was invoked for transient analysis. This model, based on the finite element method, includes parameters of the chip's physical layout, material thermal conductivity, and heat dissipation system. By simulating the generation, conduction, and dissipation of heat on the silicon wafer, a set of transient thermal stresses representing the chip's temperature change over time during the execution of this task cluster is output within a simulation time of several hundred milliseconds. The simulation calculation of transient thermal stress effectively avoids transient performance instability or even permanent physical damage caused by improper test load summation leading to localized overheating of the chip. The calculation results show that the SM8 temperature reached a peak of 89℃ after approximately 500ms of execution, while the temperature in other areas remained relatively low; this 89℃ is the peak thermal stress evaluated in this study. Simultaneously, the operating temperature safety threshold set to ensure the long-term reliability of the chip was extracted from the test target configuration file for this test; this threshold is 95℃. Finally, the difference between the simulated peak thermal stress of 89℃ and the operating temperature safety threshold of 95℃ was calculated, yielding a result of 6℃. This positive result represents the test safety margin, indicating that the currently planned combination of parallel tasks is thermally safe, and the maximum temperature it generates is still within an acceptable safety range. Therefore, the scheduling scheme can be verified.

[0050] This embodiment elevates the passive overheat protection mechanism to an active safety verification by introducing heat conduction simulation based on an accurate power consumption model before task scheduling. This not only ensures the integrity of high-value test chips but also effectively shortens the overall testing cycle, improving the efficiency and quality of testing and verification.

[0051] Furthermore, the specific implementation process of aggregating multiple test tasks into a parallel test execution cluster based on the aforementioned test safety margin includes:

[0052] With test safety margin as the core constraint, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters according to a preset scheduling strategy. The resource requirement vectors of all test tasks in the candidate set of parallel test execution clusters are updated based on the test task-resource requirement matrix. The transient thermal stress on the GPU is re-evaluated and the remaining test safety margin is verified. When the test safety margin still meets the preset conditions, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters. When the test safety margin does not meet the preset conditions, the expansion of the current candidate set of parallel test execution clusters is terminated and aggregated into a parallel test execution cluster.

[0053] Reference Figure 2 After obtaining specific test safety margin values, a parallel test execution cluster is constructed through iterative verification. The test task set includes multiple tasks awaiting scheduling, such as matrix multiplication tests, FFT tests, and ray tracing tests.

[0054] The scheduling strategy is based on the importance of the test tasks, assigning a clear priority to each task. For example, tasks verifying fundamental functions such as the chip's core computing units (e.g., CUDA cores) are given the highest priority, while high-load stress tests or auxiliary function verifications are placed in lower priority. An iterative verification method is used when constructing the execution cluster, selecting tasks from the set of tasks under test in descending order of priority and adding them to the candidate set one by one. This ensures that all tasks that pass the safety verification are combined and aggregated into a parallel test execution cluster that prioritizes the verification of critical functions while maintaining thermal stability and efficiency.

[0055] Using test safety margin as the core constraint, the highest-priority matrix multiplication test is first selected from the test task set and added to an initially empty candidate set of parallel test execution clusters. Based on the test task-resource requirement matrix, the cumulative resource requirement vector of all tasks in the candidate set (currently containing only matrix multiplication tests) is updated. Through thermal stress assessment, the peak thermal stress generated by executing this task alone is calculated to be 70℃. The set operating temperature safety threshold is 95℃, and the remaining test safety margin is verified to be 95℃-70℃=25℃. Since this safety margin still meets the preset condition of being greater than 0℃, it is determined that tasks can continue to be added to the candidate set.

[0056] Continue by selecting the second-highest priority FFT test from the remaining test tasks and adding it to the candidate set. At this point, the candidate set contains (matrix multiplication test, FFT test). Again, based on the test task-resource requirement matrix, update the combined resource requirement vector of the two tasks in this candidate set on different physical execution units, and re-evaluate the transient thermal stress based on this. The calculation yields a peak thermal stress of 89℃ for this task combination. Based on this, the remaining test safety margin is verified again, resulting in 95℃ - 89℃ = 6℃. This result still satisfies the preset condition of being greater than 0℃, indicating that the thermal load of the current candidate set is still within the safe range, therefore the expansion process continues.

[0057] Next, we attempted to add the next priority task, the extremely power-intensive ray tracing test, to the candidate set containing (matrix multiplication test, FFT test). Simultaneously, we updated the combined resource requirement vector and performed a new round of thermal stress assessment. Due to the extremely heavy load of the ray tracing test, its power consumption, combined with the previous two tasks, resulted in a peak thermal stress of 98°C from the heat conduction calculation. When verifying the test safety margin at this point, the calculation result was 95°C - 98°C = -3°C. At this point, the test safety margin no longer met the preset condition of being greater than 0°C. This result clearly indicates that adding the ray tracing test to the current candidate set would cause the chip temperature to exceed the safe threshold. Therefore, we terminated the expansion of the current parallel test execution cluster candidate set and removed the last ray tracing test task we attempted to add from the candidate set. Finally, the candidate set containing (matrix multiplication test, FFT test) was confirmed as a stable and safe combination and formally aggregated into a parallel test execution cluster, awaiting subsequent resource mapping and scheduling execution.

[0058] This embodiment achieves intelligent construction of parallel test task combinations through an iterative aggregation method with safety margin as closed-loop feedback. It can adaptively explore and determine the maximum parallel task capacity of each scheduling batch while ensuring chip safety, thereby improving the throughput and device utilization of GPU parallel testing.

[0059] Furthermore, the specific implementation process of planning the spatial mapping from each test task to the GPU physical execution unit and the GPU test scheduling graph for the execution time window for the parallel test execution cluster includes:

[0060] Based on the test task-resource requirement matrix, a resource view containing the GPU topology and the real-time availability status of each physical execution unit is obtained. The resource requirements of each test task within the parallel test execution cluster for the physical execution units are extracted. The resource requirements are input into the resource view for fitting calculation to obtain a spatial mapping. The spatial mapping contains the GPU physical execution units to which each test task within the parallel test execution cluster should be loaded. An execution time interval is determined for each test task in the spatial mapping based on the data dependencies between test tasks, forming an execution window. The spatial mapping and the execution window are integrated to generate a GPU test scheduling graph containing complete scheduling instructions.

[0061] The planned parallel test execution clusters are (matrix multiplication test, FFT test). First, a GPU resource view is generated in real-time by querying the GPU driver interface. This resource view includes not only the static topology of the GPU with 32 streaming multiprocessors (SMs), but also the real-time availability status of each physical execution unit. For example, at this moment, all 32 SMs are idle, more than 14GB of the 16GB of video memory is available, and the PCIe bus and memory controller are also in standby mode.

[0062] From the constructed test task-resource requirement matrix, the quantitative resource requirements of the physical execution units for the matrix multiplication test and the FFT test within this execution cluster are extracted. The requirement vector for the matrix multiplication test indicates that it requires 16 SMs for parallel computing power and 1.5GB of video memory; the FFT test requires 8 SMs with TensorCore functionality and 0.5GB of video memory.

[0063] The resource requirement vectors of each task within the parallel test execution cluster are matched and fitted with a real-time resource view that includes the GPU's static physical topology and dynamic availability status. First, a constraint satisfaction search is performed to generate preliminary allocation schemes that meet basic requirements such as compute unit type, quantity, and memory capacity. Among all feasible schemes, a multi-objective optimization strategy is used to select the best one, prioritizing schemes that maximize communication locality (mapping the compute units required by a single task to physically contiguous or adjacent regions) and minimize resource fragmentation. This transforms abstract task requirements into an efficient execution layout tightly coupled with the hardware. The calculation results output a clear spatial mapping scheme: matrix multiplication test tasks are assigned to SM0 to SM15 for execution; while FFT test tasks are assigned to SM16 to SM23 for execution. This spatial mapping defines the physical "landing point" of each test task on the chip.

[0064] Meanwhile, the execution window is determined based on the logical relationships between tasks. The FFT test is designed to verify the spectral characteristics of the matrix multiplication test results; therefore, there is a clear data dependency between the two. Specifically, the FFT test can only begin after the matrix multiplication test has completed and produced its result matrix. The execution time of the matrix multiplication test on the 16 SMs is approximately 120ms. Based on this data dependency, an execution time interval is determined for each test task in the spatial mapping: the execution window for the matrix multiplication test is set from time T0 (the start of scheduling) to T0+120ms; the execution window for the FFT test is set from T0+120ms to T0+200ms (its own execution time is 80ms). This series of non-overlapping time intervals together constitute the execution window.

[0065] By integrating the aforementioned spatial mapping with the determined execution window, a GPU test scheduling graph containing complete scheduling instructions is generated. This scheduling graph is a structured data object whose contents can be directly sent to the GPU's underlying task scheduler. For example, one instruction is explicitly described as: "At time T0, load the matrix multiplication test computation kernels into SM0-SM15, transfer the input data from host memory to the allocated 1.5GB video memory area, and start execution"; another instruction is: "Monitor the test status of SM0-SM15, and once a completion signal is received, immediately load the FFT test computation kernels into SM16-SM23, point its input to the video memory address where the result matrix of the matrix multiplication test is located, and start execution."

[0066] This embodiment generates a GPU test scheduling graph containing precise spatial mapping and execution windows for the aggregated parallel test execution clusters, transforming the originally abstract and loosely structured set of parallel tasks into a concrete set of hardware execution instructions. This ensures that each task obtains its necessary computing units and GPU memory, effectively reducing on-chip communication latency and improving the overall efficiency of parallel execution.

[0067] Furthermore, the specific implementation process of setting up a fault causal chain for the parallel test task and associating the GPU test scheduling graph with the real-time execution logs of the corresponding physical execution units of the GPU includes:

[0068] Based on the GPU test scheduling graph, initialize the data structure of the fault causal tracing chain; when the test task starts to be executed, capture the execution log data stream generated from each physical execution unit of the GPU in real time; parse the execution log data stream and match causal nodes for each execution instance of the test task, the causal nodes record the task identifier, the predetermined physical execution unit mapping relationship and the execution time window information; associate and bind the successfully matched execution instances with the causal nodes to generate tracing information units containing scheduling intent and execution status; link all tracing information units in an orderly manner according to the execution time window order of the test tasks in the GPU test scheduling graph to construct the fault causal tracing chain.

[0069] When preparing to issue the GPU test scheduling graph generated by the execution cluster (matrix multiplication test, FFT test), the fault causal chain data structure is initialized in memory according to the content of the scheduling graph. In its initial state, this data structure contains two ordered, incomplete causal nodes, corresponding to the matrix multiplication test and the FFT test, respectively. The first causal node pre-loads the task identifier for the matrix multiplication test, its predetermined physical execution unit mapping relationship (i.e., SM0 to SM15), and its execution window information (i.e., 120ms starting from time T0).

[0070] When the GPU begins executing the test task at time T0, it captures the execution log data streams generated by each physical execution unit in real time from the GPU's hardware performance monitor and debug registers. These data streams contain low-level events with nanosecond-level timestamps. For example, "Unit SM5 executes a floating-point multiply-accumulate instruction at time T0+15ms" and "The memory controller reports an address misalignment access at time T0+45ms". When the log from SM5 with the timestamp T0+15ms is parsed, the current causal node is immediately queried and determined to be the node corresponding to the matrix multiplication test. Since the source of the log (SM5) and the timestamp (T0+15ms) both fall within the predetermined physical mapping relationship (SM0-SM15) and execution window (T0 to T0+120ms) recorded by this causal node, the match can be determined to be successful.

[0071] Upon successful matching, the specific execution log data stream is associated and bound as an "execution liveness" to the causal node of the matrix multiplication test. This node itself records the "scheduling intent." This binding operation generates a complete source information unit, indivisibly containing all information about what was planned to be done (scheduling intent) and what actually happened (execution liveness). As the test progresses, all logs originating from its mapping unit within the matrix multiplication test execution window are continuously bound to the source information unit. The source information units are permanently and orderly linked together according to the preset execution window order in the GPU test scheduling graph, making the underlying event flow more logically connected and meaningful. When the matrix multiplication test is completed, it automatically moves to the next node in the source chain to begin matching and binding logs from SM16 to SM23 for the FFT test execution instance. Finally, when the entire execution cluster completes the test, a complete, coherent fault causal source chain corresponding one-to-one with the scheduling plan is constructed.

[0072] This embodiment constructs a fault causal chain for each parallel test task, making the test execution process no longer a "black box." This not only helps in quickly diagnosing failures but also allows for the analysis of performance bottlenecks and the detection of potential hardware instability. Consequently, it provides robust, structured data support for chip yield analysis, driver optimization, and firmware debugging, shortening the chip verification and yield improvement cycle.

[0073] Furthermore, the specific implementation process of parsing the fault cause-and-effect chain when the test task fails, outputting the fault source context that caused the failure, and investigating the cause of the failure includes:

[0074] Upon receiving a test failure signal, the system retrieves and locates the end-point tracing information unit directly associated with the failure event in the fault causal tracing chain based on the timestamp carried in the fault signal. Starting from the end-point tracing information unit, the system performs a reverse tracing traversal along the fault causal tracing chain. During the reverse tracing traversal, the system extracts in batches the GPU real-time execution log fragments and corresponding GPU test scheduling graph information encapsulated by each preceding tracing information unit. The extracted real-time execution log fragments and scheduling graph information are then time-aligned and state-associated to reconstruct a continuous GPU execution state sequence before and after the failure point and output the fault source context. Anomaly detection is performed on the fault source context to investigate the cause of the failure.

[0075] Reference Figure 3 When an execution cluster containing (matrix multiplication test, FFT test) issues a test failure signal during the test, the system receives the fault signal, which includes a timestamp T0+125.7ms, an error code 0x1F04 (indicating illegal memory access), and the physical hardware unit ID that reported the exception.

[0076] Based on the timestamp T0+125.7ms carried in the fault signal, a rapid retrieval is performed in the constructed fault causal tracing chain. Since this time point falls within the execution window of the FFT test task (T0+120ms to T0+200ms), the tracing information unit associated with the FFT test execution instance is immediately locked and marked as the end tracing information unit directly associated with the fault event.

[0077] Starting from the locked FFT test tracing information unit, the process traces backwards along the pre-established link pointers in the fault causal tracing chain. The traversal operation will trace back from the FFT test node to the tracing information unit of its preceding matrix multiplication test.

[0078] During the reverse traversal, batch data extraction is performed. From the FFT test tracing information unit, which serves as the starting point for parsing, all GPU real-time execution log segments from T0+120ms to the fault occurrence time T0+125.7ms, along with their corresponding scheduling graph information, are extracted, representing the FFT test execution plan on SM16 to SM23. From the backtracked preorder matrix multiplication test tracing information unit, execution log segments from T0 to T0+120ms, along with the scheduling plan executed on SM0 to SM15, are extracted.

[0079] Two temporally consecutive but logically separate real-time execution log segments are temporally aligned and state-associated with their respective scheduling graph information. This allows the discrete underlying logs to be pieced together and reconstructed into a continuous GPU execution state sequence with high-level scheduling intent annotations, starting from time T0 until the failure point T0+125.7ms. For example, the reconstructed sequence clearly shows that at T0+119.8ms, the matrix multiplication test writes a pointer value to its predetermined output memory address at the end of its execution phase, but this value is 0x00000000 (null pointer); immediately following, at T0+125.6ms, the FFT test, based on the data dependencies in the scheduling graph, attempts to read input data from the aforementioned address, thus triggering an illegal memory access exception in the physical hardware unit. This fully reconstructed execution sequence, containing the cause-and-effect relationships, constitutes the failure source context that led to the failure.

[0080] The built-in anomaly detection function performs a deep analysis of the fault source context of this output. Using preset diagnostic rules, such as "detecting the pattern of immediately performing a read operation after writing to a null pointer," it quickly identifies the erroneous output of the matrix multiplication test as the root cause of the FFT test failure. Finally, a clear diagnostic report is generated, pointing out that the direct cause of the failure is an illegal memory access in the FFT test, while the root cause lies in the logical flaw in the matrix multiplication test task itself, thus completing a precise investigation of the cause of the failure.

[0081] This embodiment analyzes the fault causal chain when a test fails, directly locating the final traceability information unit by utilizing the timestamp of the fault signal. This avoids blind searching in the data log, automatically and selectively focusing on historical execution segments directly related to the fault, reducing interference from irrelevant information and shortening the chip development and verification cycle.

[0082] This embodiment elevates passive overheat protection to proactive risk avoidance by introducing predictive safety verification based on accurate power consumption and thermal models. It effectively prevents tasks that could cause chip temperatures to exceed limits from being incorrectly aggregated, thus avoiding physical damage to the chip caused by improper test load stacking.

[0083] Intelligent task aggregation maximizes the number of tasks that can be executed in parallel while ensuring chip security. Merging multiple tasks into parallel execution clusters shortens the testing cycle and improves the throughput of the chip testing platform.

[0084] By constructing a "fault causal chain" that strongly correlates scheduling intent with execution reality and employing a precise matching mechanism based on hardware ID tags, debugging efficiency is effectively improved. This enables the automatic and rapid extraction of complete event sequences related to faults from underlying logs and accurate tracing of the root cause of the fault in complex concurrent execution environments, thereby shortening the chip development verification and yield improvement cycle.

[0085] Example 2:

[0086] This embodiment fully deploys the above-mentioned parallel test task scheduling method for high-computing-power GPU chips in the X Chip Test Center to realize parallel testing of GPU chips.

[0087] At the start of the testing process, the scheduling server receives a set of test tasks containing multiple independent test cases. These include a matrix multiplication test to verify the peak performance of the CUDA cores, a Fast Fourier Transform test to verify the dedicated TensorCore units, and a high-load ray tracing test. The syntax analysis function uses an LLVM-based static code analyzer to quantify the execution logic and resource requirements of the tasks. For the matrix multiplication test, the ratio of floating-point instructions to global memory access instructions in the CUDA core is analyzed (statistically approximately 8.2). When this ratio exceeds a preset threshold of 5.0, its computational core requirement is precisely quantified as "CUDA Core: High". By scanning and summing the call parameters of all cudaMalloc functions in the code, the video memory requirement is quantified as 1.5GB; combined with thread configuration and memory access mode, the average memory bandwidth requirement is predicted to be approximately 80GB / s using a built-in empirical estimation model. The Fast Fourier Transform test, due to its use of the TensorCore-specific API, is quantified as "TensorCore: Medium" and requires 0.5GB of video memory. The quantified information is integrated into a resource requirement vector and populated into the test task-resource requirement matrix.

[0088] By obtaining the real-time status of the GPU chip through the management interface, it was found that the initial temperature of all streaming multiprocessors (SMs) was 42°C, the baseline power consumption of each SM was 5 watts, and the chip's design safe temperature threshold was 95°C. Based on the test task-resource requirement matrix constructed in the previous step, a multivariate linear regression power model was used to calculate the power consumption distribution generated by this task combination. This power model describes the power consumption of each SM unit as a linear combination of multiple variables such as baseline power consumption and CUDA core utilization and TensorCore unit activity level. The coefficients were obtained by fitting the model after extensive benchmark testing on the actual chip. The calculation showed that if the matrix multiplication test was scheduled for SM0 to SM15, the power consumption of these units would increase to 35 watts, while the power consumption of SM16 to SM23, which performed the Fast Fourier Transform test, would increase to 27 watts. This non-uniform power consumption distribution was then used as a heat source input for heat conduction calculation. A heat conduction model based on two-dimensional finite element analysis was used. The chip's physical layout was abstracted into a mesh, and thermal conductivity parameters were assigned to key materials such as silicon. Simultaneously, a boundary condition with an equivalent thermal resistance of 0.2 K / W was used to simulate heat exchange with the cooling system. After a brief transient thermal simulation, the results showed that the temperature in the SM8 region reached a peak of 88℃ at approximately 480 ms. The final calculated safety margin was 7℃ (95℃-88℃), thus determining that this task combination is thermally safe.

[0089] A parallel test execution cluster is constructed based on the calculated test safety margin. To ensure the determinism and reproducibility of scheduling decisions, a multi-level sorting scheduling strategy is adopted: all tasks under test are first sorted in descending order of predefined priority, and if priorities are the same, they are sorted in ascending order of required video memory capacity. First, the matrix multiplication test with the highest priority is added to the candidate set. Thermal verification shows its peak temperature of 70°C when executed alone, with a remaining safety margin of 25°C. Then, the Fast Fourier Transform test with the next lower priority is added to the candidate set. The peak temperature after joint thermal verification is 88°C, with a remaining safety margin of 7°C, still meeting the condition of being greater than the preset buffer value of 5°C. When attempting to add the extremely power-intensive ray tracing test, the peak temperature predicted by joint thermal verification reaches 99°C, exceeding the preset safety threshold. Therefore, this addition is rejected, and the expansion of the current execution cluster is terminated. Finally, a parallel test execution cluster containing matrix multiplication and Fast Fourier Transform tests is officially created.

[0090] After the execution cluster is constructed, a GPU test scheduling graph, including spatial mapping and execution windows, is planned for it. The computational kernels for the matrix multiplication test are precisely assigned and loaded into SM0 to SM15, while the Fast Fourier Transform (FFT) test is mapped to SM16 to SM23. Considering that the FFT test needs to utilize the computational results of the matrix multiplication test, and based on the performance model's estimated 120ms execution time, an execution window of T0 to T0+120ms is set for the matrix multiplication test, and a subsequent execution window of T0+120ms to T0+200ms is set for the FFT test. These precise spatial and temporal instructions are integrated into a structured scheduling graph object, ready to be issued to the GPU driver for execution.

[0091] During the test execution phase, to ensure accurate correlation between logs and scheduling intents, a hardware-assisted matching mechanism was employed when issuing the scheduling graph: unique runtime IDs (such as 0x1A01 and 0x1A02) were assigned to the matrix multiplication test and the Fast Fourier Transform test, respectively, and these IDs were written to the specific context register of the SM unit to which the task would be executed. The chip's hardware logging system was configured to automatically append the current context ID when generating any event record. In an actual test, the matrix multiplication test, due to a logic defect, wrote a null pointer to its output address at T0+119.8ms; this write operation was automatically marked with ID: 0x1A01 in the log. Immediately afterwards, the Fast Fourier Transform test started at T0+125.6ms and attempted to read data from that address, triggering an illegal memory access interrupt in the physical hardware unit; the relevant log was marked with ID: 0x1A02. Upon receiving an interrupt signal, the system immediately locates the Fast Fourier Transform (FFT) test node where the fault occurred within a pre-built tracing chain based on the timestamp and the ID in the log, and then traces back along the chain to its preceding matrix multiplication test node. By batch extracting all log segments marked 0x1A01 and 0x1A02 and performing timing reconstruction, the complete fault sequence from "erroneous write" to "illegal read" is automatically and clearly restored. The final diagnostic report clearly indicates that the direct cause of the fault is an illegal memory access during the FFT test, while the root cause is an erroneous output during the matrix multiplication test.

[0092] It should be clarified that the embodiments described above are merely exemplary and are intended to aid in understanding the present invention, not to limit it. Those skilled in the art can make various changes and modifications after grasping the core ideas of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for scheduling parallel test tasks for high-performance GPU chips, characterized in that, include: The test task set is parsed and a test task-resource requirement matrix is ​​constructed. The test task-resource requirement matrix contains the resource requirements of each test task for the physical execution units inside the GPU and the matching relationship with the test target. Based on the GPU's real-time status, the test objective, and the test task-resource requirement matrix, the transient thermal stress generated by the test task on the GPU is calculated and the test safety margin is verified. The GPU's real-time status, including the current temperature and power consumption baseline of the physical execution units, is obtained. Combined with the resource requirement vector corresponding to the test task in the test task-resource requirement matrix, the power consumption superposition value generated when executing the test task on each physical execution unit of the GPU is calculated. The power consumption superposition value is used as the heat source input for heat conduction calculation, and the transient thermal stress is output. Extract the GPU's operating temperature safety threshold from the test target, and calculate the difference between the peak thermal stress in the transient thermal stress and the operating temperature safety threshold to obtain the test safety margin. Based on the test safety margin, multiple test tasks are aggregated into a parallel test execution cluster; a GPU test scheduling graph is planned for the parallel test execution cluster, outlining the spatial mapping from each test task to a GPU physical execution unit and the execution window; based on the test task-resource requirement matrix, a resource view containing the GPU topology and the real-time availability status of each physical execution unit is obtained, and the resource requirements of each test task within the parallel test execution cluster for the physical execution unit are extracted; the resource requirements are input into the resource view for fitting operations to obtain a spatial mapping; the spatial mapping includes the GPU physical execution unit to which each test task within the parallel test execution cluster should be loaded; based on the data dependencies between test tasks, an execution time interval is determined for each test task in the spatial mapping, forming an execution window; the spatial mapping and the execution window are integrated to generate a GPU test scheduling graph containing complete scheduling instructions; A fault cause-and-effect tracing chain is set up for the parallel test task, and the GPU test scheduling graph is associated and bound with the real-time execution log of the corresponding physical execution unit of the GPU. When the test task fails, the fault cause-and-effect tracing chain is used for parsing, the fault source context that caused the failure is output, and the cause of the failure is investigated.

2. The parallel test task scheduling method for high-computing-power GPU chips according to claim 1, characterized in that, The specific implementation process of parsing the test task set and constructing the test task-resource requirement matrix includes: The execution logic and resource requirement descriptions of all test tasks are extracted through syntax analysis. For the execution logic, the types of computing cores and storage resources required by the test task are identified and quantified into functional parameters. For the resource requirement descriptions, the GPU memory capacity and memory bandwidth required for task execution are obtained and quantified into constraints. The functional parameters and constraints are integrated to generate a resource requirement vector for the test task. A data matrix is ​​created with all test tasks as the first dimension and the schedulable physical execution units and resource types within the GPU as the second dimension. The resource requirement vectors corresponding to the test tasks are filled into the data matrix to construct a test task-resource requirement matrix.

3. The parallel test task scheduling method for high-computing-power GPU chips according to claim 1, characterized in that, The specific implementation process of aggregating multiple test tasks into a parallel test execution cluster based on the aforementioned test safety margin includes: With test safety margin as the core constraint, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters according to a preset scheduling strategy. The resource requirement vectors of all test tasks in the candidate set of parallel test execution clusters are updated based on the test task-resource requirement matrix. The transient thermal stress on the GPU is re-evaluated and the remaining test safety margin is verified. When the test safety margin still meets the preset conditions, test tasks are selected from the test task set and added to the candidate set of parallel test execution clusters. When the test safety margin does not meet the preset conditions, the expansion of the current candidate set of parallel test execution clusters is terminated and aggregated into a parallel test execution cluster.

4. The parallel test task scheduling method for high-computing-power GPU chips according to claim 1, characterized in that, The specific implementation process of setting up a fault causal chain for the parallel test task and associating the GPU test scheduling graph with the real-time execution logs of the corresponding physical execution units of the GPU includes: Based on the GPU test scheduling graph, initialize the data structure of the fault causal tracing chain; when the test task starts to be executed, capture the execution log data stream generated from each physical execution unit of the GPU in real time; parse the execution log data stream and match causal nodes for each execution instance of the test task, the causal nodes record the task identifier, the predetermined physical execution unit mapping relationship and the execution time window information; associate and bind the successfully matched execution instances with the causal nodes to generate tracing information units containing scheduling intent and execution status; link all tracing information units in an orderly manner according to the execution time window order of the test tasks in the GPU test scheduling graph to construct the fault causal tracing chain.

5. The parallel test task scheduling method for high-computing-power GPU chips according to claim 1, characterized in that, When a test task fails, the process of parsing the fault cause-and-effect chain, outputting the fault source context that led to the failure, and investigating the cause of the failure includes: Upon receiving a test failure signal, the system retrieves and locates the end-point tracing information unit directly associated with the failure event in the fault causal tracing chain based on the timestamp carried in the fault signal. Starting from the end-point tracing information unit, the system performs a reverse tracing traversal along the fault causal tracing chain. During the reverse tracing traversal, the system extracts in batches the GPU real-time execution log fragments and corresponding GPU test scheduling graph information encapsulated by each preceding tracing information unit. The extracted real-time execution log fragments and scheduling graph information are then time-aligned and state-associated to reconstruct a continuous GPU execution state sequence before and after the failure point and output the fault source context. Anomaly detection is performed on the fault source context to investigate the cause of the failure.

Images