Method for obtaining highest utilization of operator, electronic device, and storage medium

Abstract

The application relates to the technical field of chip design, in particular to a method for obtaining the highest utilization rate of an operator, an electronic device and a storage medium, which comprises an experimental kernel function of the operator, the experimental kernel function only comprises reading instructions and synchronization instructions, and K empty instructions are inserted between adjacent reading instructions; the value of K in the experimental kernel function is dynamically adjusted in descending order, the adjusted experimental kernel function is executed and tested on a GPU hardware; the trend of the actual reading time obtained by each test changing with the value of K is analyzed, the inflection point at which the actual reading time changes from being equal to the theoretical reading time to being greater than the theoretical reading time is obtained, and the test is ended; a plurality of candidate tests with continuously reduced K values before the inflection point are obtained; the calculation utilization rates of the calculation instructions corresponding to all the candidate tests are obtained, the maximum value in all the calculation utilization rates is determined as the highest utilization rate of the operator, and the highest utilization rate of the memory-intensive operator is accurately calculated.

Description

Technical Field

[0001] This invention relates to the field of chip design technology, and in particular to a method for obtaining the highest utilization rate of an operator, an electronic device, and a storage medium. Background Technology

[0002] Deep learning operators executing on modern graphics processing units (GPUs) can be broadly categorized into two types: computationally intensive and memory-intensive. This classification isn't absolute but rather determined by the ratio of data read to computation (bytes / cycle); a higher ratio generally classifies an operator as memory-intensive. For computationally intensive operators, since their computational demands far exceed their data access needs, their actual efficiency can ideally approach 100% of the multi-precision matrix multiplication / accumulation (MMA / MTE) utilization, making their actual performance ceiling relatively clear. However, for the more widely used memory-intensive operators, the situation is far more complex. The actual performance ceiling (not the theoretical performance ceiling) of these operators depends not only on their computational power but, more importantly, on their data read and usage efficiency. Therefore, the actual performance of these operators is often difficult to predict in advance, and optimization typically relies on referencing competitor examples and continuous experimentation—a method that is inefficient and often yields unsatisfactory results. Memory-intensive operators involve long data read chains during execution, encompassing multiple modules including VLS, VL1 (ATUL1, Partition), L2C, and HBM. Each module, both internally and externally, has instruction or data buffering mechanisms. Due to the massive data demands of memory-intensive operators, these buffers frequently become full, leading to queuing and unstable data return times.

[0003] Furthermore, during operator execution, waves share data through a level 1 cache, and work groups share data through a level 2 cache. However, due to the lack of a pre-ordering mechanism, execution speeds differ between waves and work groups, making it impossible to determine whether a data read request on a particular execution unit will hit a certain level of cache, further exacerbating the volatility of data return time. Simultaneously, each level of bandwidth has its own upper limit. During operator execution, each module intermittently impacts its own bandwidth limit, causing overall bandwidth fluctuations rather than maintaining a stable, fixed value. Coupled with known or unknown GPU design limitations or defects, unexpected data read latency and bandwidth constraints are added. Considering that different settings for the frequency, power consumption, and temperature limits of each module during GPU execution can also produce different performance results, and given the limitations of GPU hardware resources, the reasonableness of the operator's memory-to-computation ratio, and the degree of code optimization, accurately calculating the maximum utilization of memory-intensive operators becomes an extremely challenging task. Therefore, a method for accurately calculating the maximum utilization of memory-intensive operators is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention employs the following technical solution: a method for obtaining the maximum utilization rate of an operator, the method comprising the following steps:

[0005] S100, Obtain the experimental kernel function of the operator, wherein the instructions in the experimental kernel function of the operator include only read instructions and synchronization instructions, and K empty instructions are inserted between adjacent read instructions.

[0006] S200, dynamically adjust the value of K in the experimental kernel function, execute and test the adjusted experimental kernel function on GPU hardware; analyze the trend of the actual reading time obtained in each test with the change of K value, and when K is adjusted to a certain value, the inflection point where the actual reading time changes from being equal to the theoretical reading time obtained in the previous test to being greater than the theoretical reading time, then the test ends.

[0007] S300, obtain multiple candidate tests for continuously decreasing K values ​​before the inflection point.

[0008] S400: Based on the ratio of computation instructions to read instructions in the benchmark kernel function of the operator, obtain the computation utilization rate of the computation instructions corresponding to all candidate tests, and determine the maximum value among all computation utilization rates as the highest utilization rate of the operator.

[0009] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, wherein the at least one instruction or the at least one program segment is loaded and executed by a processor to implement the above-described method.

[0010] In addition, the present invention provides an electronic device including a processor and the aforementioned non-transitory computer-readable storage medium.

[0011] The present invention has at least the following beneficial effects:

[0012] This invention provides a method, electronic device, and storage medium for obtaining the highest execution efficiency of an operator. It utilizes an experimental kernel function for the operator, which includes only read instructions and synchronization instructions, with K empty instructions inserted between adjacent read instructions. The value of K is dynamically adjusted and tested to obtain the highest utilization rate of the computation instructions. By replacing the computation instructions in the baseline kernel function with empty instructions to obtain the experimental kernel function, and by obtaining the inflection point through the experimental kernel function, it can accurately evaluate the utilization rate of computing resources under real memory access pressure. This avoids the bottleneck misjudgment caused by parallel computation and memory access in traditional performance analysis, significantly improving the targeting and efficiency of performance optimization, and achieving accurate calculation of the highest utilization rate of memory-intensive operators. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating a method for obtaining the highest execution efficiency of an operator, provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this invention have the same meaning as commonly understood by those skilled in the art.

[0017] Please see Figure 1 It illustrates a method for obtaining the highest execution efficiency of an operator, the method comprising the following steps:

[0018] S100, Obtain the experimental kernel function of the operator. The instructions in the experimental kernel function of the operator include only read instructions and synchronization instructions, with K empty instructions inserted between adjacent read instructions. Where K is greater than or equal to 1.

[0019] It should be noted that the experimental kernel function of the operator is obtained by directly reusing the data reading logic of the operator's baseline kernel function, and modifying it accordingly. This eliminates the need to redesign the experimental kernel function, resulting in high efficiency and ensuring that the differences between the experimental and baseline kernel functions lie only in the modified parts.

[0020] In one implementation, S100 further includes a step of constructing the experimental kernel function of the operator:

[0021] S110, obtain the baseline kernel function of the operator.

[0022] The baseline kernel function is the original kernel function used to implement the operator. The execution of the baseline kernel function involves instructions such as reading, computation, and synchronization. Its performance is affected by factors such as memory access mode, thread arrangement, cache line utilization, data path (e.g., entering BSM), and synchronization mechanisms (e.g., barriers). The memory access mode is either row-major or column-major.

[0023] S120, remove all instructions except for read instructions and thread group synchronization instructions. The synchronization instructions are used to force all threads within the thread group to complete the current phase, ensuring that subsequent reads depend on previously loaded data.

[0024] S130, K null instructions are inserted between two adjacent read instructions to obtain the experimental kernel function. The purpose of inserting null instructions is to facilitate the calculation of theoretical time and to ensure that the intervals between all read instructions are consistent. The data return of a single read instruction may require 1000 to 2000 cycles, which is much longer than the interval between read instructions. Multiple read instructions are executing simultaneously; a new read instruction can only be issued when a currently executing read instruction returns data, releasing the occupied hardware resources; otherwise, a waiting time occurs.

[0025] It should be noted that, to accurately test hardware performance, the experimental kernel functions of the constructed operators maintained consistency in memory access patterns, thread arrangement, cache line utilization, and data path synchronization mechanisms. Read instructions were evenly distributed across each thread, with the difference being that the same number of null instructions (no computation instructions) were placed between any two adjacent read instructions. This was achieved through assembly language to avoid compiler interference. For example, the test interval between read instructions was adjusted to 92 cycles, 100 cycles, 108 cycles, etc., by inserting the number of null instructions.

[0026] S200: Dynamically adjust the value of K in the experimental kernel function in descending order, and actually execute and test the adjusted experimental kernel function on GPU hardware; analyze the trend of the maximum actual read time obtained in each test as a function of the K value; when K is adjusted to a certain value, and the actual read time changes from being equal to the theoretical read time in the previous test to being greater than the theoretical read time, the test ends. It should be noted that when testing with K values ​​from smallest to largest, it is also necessary to obtain the inflection point, which is when the actual read time changes from being greater than the theoretical read time in the previous test to being equal to the theoretical read time.

[0027] The theoretical read time is the minimum computation time required for the hardware computing unit to complete memory access and no-instruction tasks without waiting or pauses. When the actual read time equals the theoretical read time, it indicates that there is no idle time between read instructions, and the memory access system utilization is low. The K value can be further reduced and tested again to observe the change in actual read time. If the actual read time still equals the theoretical read time, it means the inflection point has not yet been reached, and the K value can be further reduced for further observation. However, if after further reducing the K value and testing again, the actual read time is greater than the theoretical read time, it indicates that there is idle time required for read instructions to complete, meaning that the actual read time corresponding to the K value before adjustment was the minimum time required to complete these read operations. It should be noted that after each dynamic adjustment of the K value, testing is required again. The actual read time needs to be obtained after testing, and the theoretical read time needs to be recalculated, rather than reusing the previous value.

[0028] In one implementation, in step S200, the step of obtaining the theoretical reading time includes: obtaining the sum of the sending interval between reading instructions and the execution time of the synchronization instruction to obtain the actual interval time between two reading instructions; and multiplying the actual interval time by the total number of reading instructions in the stable execution time window to obtain the theoretical reading time.

[0029] It should be noted that when the experimental kernel function of an operator is scheduled for execution, it is scheduled in parallel across different APs. Each AP includes multiple PEUs, and each PEU executes multiple thread groups. Therefore, the theoretical execution time is the same for different thread groups of the same experimental kernel function of an operator, and there is no need for repeated calculations. Furthermore, when adjusting K, since K is a parameter in the experimental kernel function of the operator, the adjustment will be applied to all scheduled thread groups.

[0030] In one implementation, the stable execution time window is a continuous period during kernel function execution where read instructions are continuously issued and the system is in a steady state. This excludes the initial warm-up phase and the idle phase at the end where only issued instructions are being waited for to complete, ensuring that the acquired instruction behavior and execution time accurately reflect the computational efficiency of the real load. Specifically, to ensure that the number and proportion of acquired read instructions accurately reflect the real load characteristics, the warm-up phase of the experimental kernel function's startup phase needs to be excluded, as the distribution of read instructions fluctuates significantly because the cache, prefetcher, and memory subsystem have not yet reached a stable state. Therefore, this embodiment of the invention only collects instruction behavior within the stable execution time window after the warm-up. To avoid underestimating the utilization of computational resources, the idle waiting period at the end of the experimental kernel function's execution, where only issued instructions are being waited for to complete without any new read or computation instructions being issued, needs to be excluded.

[0031] In one implementation, S200 further includes a step for determining the inflection point during each test, wherein, when the value of K in the experimental kernel function is dynamically adjusted for the j-th time, and the adjusted experimental kernel function is actually executed and tested on the GPU hardware, the step for determining the inflection point includes:

[0032] S210, based on the experimental kernel function obtained from the j-th dynamic adjustment, perform the j-th test and select a stable execution time window from the test results. It should be noted that the principle for selecting a stable execution time window is consistent for each test.

[0033] S220: Obtain the actual read time of a stable execution time window. If the actual read time of the j-th test is greater than the theoretical time and the actual read time of the (j-1)-th test is equal to the theoretical time, then the j-th test is the inflection point. If the actual read time of the j-th test is greater than the theoretical time and the actual read time of the (j-1)-th test is greater than the theoretical time, then the value of K in the experimental kernel function is dynamically increased and the (j+1)-th test is performed; if the actual read time of the j-th test is equal to the theoretical time, then the value of K in the experimental kernel function is dynamically decreased and the (j+1)-th test is performed. It should be noted that, according to the adjustment rules in S220, before reaching the inflection point, if the actual read time is equal to the theoretical time, then the value of K is decreased and the test is performed again; if the actual read time is greater than the theoretical time, then the value of K is increased and the test is performed again. If the actual reading time in the (j-1)th test is equal to the theoretical time, the K value will be reduced before the jth test is performed. Reducing the K value means that the sending interval between reading instructions is reduced, which is equivalent to increasing the number of reading instructions before the test is performed. If the actual reading time in the jth test is greater than the theoretical time, it means that there is a reading instruction that needs to be waited for, which means that there is an inflection point from no waiting instructions to waiting instructions. Therefore, the jth test is the inflection point.

[0034] In one implementation, the smallest computing unit in a processing unit group is taken as the analysis object. The read time of all threads in the analysis object within a stable execution time window is obtained, and the maximum value among the read times of all analysis objects is the actual read time. As an example, a GPU's Accelerator Processor (AP) can run eight thread groups simultaneously on one PEU (Processing Element Unit), with each thread group consisting of 64 threads.

[0035] It should be noted that the purpose of obtaining the actual read time is to find the worst-case read time, because the system's execution efficiency depends on the worst-case read time.

[0036] In one implementation, the experimental kernel function is executed in parallel on a processing unit group of the GPU, the processing unit group comprising N minimum computing units, where N is greater than or equal to 1; wherein the actual read time is the maximum read time in the processing unit group.

[0037] S300, obtain multiple candidate tests for continuously decreasing K values ​​before the inflection point.

[0038] It should be noted that before the inflection point, there may be only one candidate test, or there may be two or more candidate tests.

[0039] S400: Based on the ratio of computation instructions to read instructions in the benchmark kernel function of the operator, obtain the computation utilization rate of the computation instructions corresponding to all candidate tests, and determine the maximum value among all computation utilization rates as the highest utilization rate of the operator.

[0040] It should be noted that the target read time reflects the time required to complete the read instruction on the GPU, and characterizes the limitation of the memory access subsystem on the data supply capacity.

[0041] In one implementation, step S400 further includes a step of obtaining the computational utilization rate of the computational instruction corresponding to the i-th candidate test:

[0042] S410: Obtain the stable execution time window in the i-th candidate test. It should be noted that the steps for obtaining the stable execution time window in the candidate test are the same as those in S210; when the i-th candidate test is the j-th test, the stable execution time window in both refers to the same thing.

[0043] S420, obtain the actual read time and the total number of read instructions for the stable execution time window.

[0044] S430, based on the ratio of computation instructions to read instructions in the operator's baseline kernel function, calculate the total number of computation instructions within the stable execution time window. It should be noted that for a fully designed operator, the ratio of read instructions to computation instructions is fixed.

[0045] S440, the computational utilization rate corresponding to the i-th candidate test is generated based on the total number of computation instructions, the execution time of a single computation instruction, and the actual reading time.

[0046] In one implementation, the computational utilization rate is equal to the product of the total number of computation instructions and the execution time of a single computation instruction, divided by the actual read time. Here, the product of the total number of computation instructions and the execution time of a single computation instruction essentially represents the theoretical computation time of the computation instructions.

[0047] As an example, when the overall matrix M×N×K is 3328×128×4096, the operator size m×n is 64×64. Therefore, it is divided into 52 parts in the m direction and 2 parts in the n direction, resulting in a total of 104 sub-matrices. These 104 operators are then distributed across 104 APs for execution. Each PEU executes 1024 MMA computation instructions and 256 read instructions, with a ratio of 4:1. During testing, the number of read instructions within a stable execution time window was 240, and the number of MMA computation instructions obtained according to the ratio was 960. Since each MMA computation instruction requires 16 cycles to execute, the 960 MMA computation instructions require a total of 15360 cycles to execute. With the preceding conditions remaining the same, only the time interval between read instructions and the actual read time differ in each test. When the inflection point is the 12th test, and the candidate tests include tests 9-11: In test 9, the interval between read instructions and the actual read time interval (cycle) are 92 cycles and 31008 cycles respectively, with a utilization rate of 15360 / 31008 = 49.5%; In test 10, the interval between read instructions and the actual read time interval (cycle) are 100 cycles and 30845 cycles respectively, with a utilization rate of 15360 / 30845 = 49.8%; In test 11, the interval between read instructions and the actual read time interval (cycle) are 108 cycles and 31217 cycles respectively, with a utilization rate of 15360 / 31217 = 49.2%; then the maximum utilization rate of the computational instructions corresponding to all candidate tests is 49.8%.

[0048] In summary, the embodiments of the present invention provide a method for obtaining the highest execution efficiency of an operator. This method utilizes an experimental kernel function of the operator, which includes only read instructions and synchronization instructions, with K empty instructions inserted between adjacent read instructions. The value of K is dynamically adjusted and tested to obtain the highest utilization rate of computation instructions. This method can accurately assess the utilization rate of computing resources under real memory access pressure, avoiding bottleneck misjudgments caused by parallel computation and memory access in traditional performance analysis, and significantly improving the targeting and efficiency of performance optimization.

[0049] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0050] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0051] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0053] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.

Claims

1. A method for obtaining the highest utilization of an operator, characterized in that, The method comprises the following steps: S100, obtaining an experimental kernel function of an operator, instructions in the experimental kernel function of the operator only comprising reading instructions and synchronization instructions, and K empty instructions inserted between adjacent reading instructions; S200, dynamically adjusting a value of K in the experimental kernel function in descending order, and actually executing and testing the adjusted experimental kernel function on GPU hardware; analyzing a trend of an actual reading time obtained by each test varying with the value of K, when the value of K is adjusted to a certain value, the actual reading time obtained by the last test is equal to a theoretical reading time, and the actual reading time is changed to a turning point greater than the theoretical reading time, and then the test is ended; S300, obtaining a plurality of candidate tests of continuously decreasing K values before the turning point; S400, obtaining a calculation utilization rate of a calculation instruction corresponding to all candidate tests according to a proportional relationship between the calculation instruction and the reading instruction in a benchmark kernel function of the operator, and determining a maximum value in all calculation utilization rates as a highest utilization rate of the operator.

2. The method of claim 1, wherein, The turning point judgment step in each test process in S200 comprises: S210, performing the jth test according to the experimental kernel function obtained by the jth dynamic adjustment, and selecting a stable execution time window in a test result; S220, obtaining an actual reading time of the stable execution time window; if the actual reading time of the jth test is greater than the theoretical time and the actual reading time of the j-1th test is equal to the theoretical time, then the jth test is a turning point; if the actual reading time of the jth test is greater than the theoretical time and the actual reading time at the j-1th time is greater than the theoretical time, then the value of K in the experimental kernel function is dynamically increased and the j+1th test is performed; if the actual reading time of the jth test is equal to the theoretical time, then the value of K in the experimental kernel function is dynamically decreased and the j+1th test is performed.

3. The method of claim 2, wherein, In S200, the obtaining step of the theoretical reading time comprises: obtaining a sum of a sending interval between reading instructions and an execution time of a synchronization instruction to obtain an actual interval time between two reading instructions; and multiplying the actual interval time and a total number of reading instructions in the stable execution time window to obtain the theoretical reading time.

4. The method according to claim 2 or 3, characterized in that, The stable execution time window is a continuous period in which the reading instructions are continuously emitted and the system is in a steady state during kernel function execution.

5. The method of claim 1, wherein, The obtaining step of the calculation utilization rate of the calculation instruction corresponding to the ith candidate test in S400 comprises: S410, obtaining a stable execution time window in the ith candidate test; S420, obtaining the actual reading time and a total number of reading instructions of the stable execution time window; S430, calculating a total number of calculation instructions in the stable execution time window according to a proportional relationship between the calculation instruction and the reading instruction in the benchmark kernel function of the operator. S440, generating the computing utilization rate corresponding to the ith candidate test according to the total number of the computing instructions, the execution time of a single computing instruction and the actual reading time.

6. The method of claim 1, wherein, The computing utilization rate of the computing instruction is equal to the product of the total number of the computing instruction and the execution time of a single computing instruction and divided by the target reading time.

7. The method of claim 1, wherein, The step of constructing the experimental kernel function of the operator in S100 further comprises: S110, obtaining a benchmark kernel function of the operator; S120, removing instructions other than reading instructions and thread group synchronization instructions; S130, inserting K empty instructions between two adjacent reading instructions to obtain the experimental kernel function.

8. The method of claim 1, wherein, In S200, the experimental kernel function is executed in parallel on a processing unit group of a GPU, and the processing unit group comprises N minimum computing units, where N is greater than or equal to 1; wherein the actual reading time is the maximum reading time in the processing unit group. 9.A non-transitory computer-readable storage medium having stored therein at least one instruction or at least one piece of program, characterized in that, The at least one instruction or the at least one program is loaded and executed by the processor to implement the method of any one of claims 1-8.

10. An electronic device, comprising: The non-transitory computer readable storage medium of claim 9 is included in the processor. The non-transitory computer readable storage medium of claim 9 is included in the processor.

Images