Methods, computing devices, media, and program products for optimizing operator performance

Abstract

This invention relates to a method, computing device, medium, and program product for optimizing operator performance. The method includes: acquiring grouped data of a target operator via hardware performance counters to aggregate the acquired counter data into performance counter description information; parsing the performance counter description information to extract multiple performance metrics and classify the extracted performance metrics; comparing the values ​​of the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison result; and in response to triggering at least one decision rule, outputting at least one priority-ranked optimization strategy, wherein the optimization strategy at least includes information for indicating operator code changes. This invention can automatically and efficiently output priority-based optimization suggestions for operator performance tuning.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of artificial intelligence, and more specifically to a method, computing device, computer-readable storage medium, and computer program product for optimizing operator performance. Background Technology

[0002] In modern deep learning inference and training systems, the execution efficiency of operators (e.g., but not limited to kernel functions) on computing devices (e.g., but not limited to graphics processing units, GPUs) directly determines the end-to-end latency of the model and the system throughput. Once the initial implementation of a GPU operator is completed, it is necessary to identify the performance bottlenecks of the current operator and what optimization methods should be adopted.

[0003] Traditional methods for optimizing operators rely heavily on senior performance engineers with deep hardware architecture knowledge to interpret hardware performance counter (PFC) metrics one by one and to determine bottleneck types and select optimization strategies based on experience. However, this process is neither repeatable nor automated, and cannot be directly executed by junior engineers or AI agents.

[0004] Traditional methods for optimizing operators also include using analysis tools to diagnose operator performance. For example, some analysis tools generate general suggestions for all kernel functions based on a set of static rules, without distinguishing the specific bottleneck severity and priority of different kernel functions. Engineers need to determine optimization priorities and specific implementation schemes themselves. Another example is that some analysis tools search the search space by enumerating parameter configurations, without utilizing the rich bottleneck information contained in the PFC to guide the search direction, resulting in a significant waste of search resources on irrelevant configuration spaces.

[0005] In summary, the shortcomings of traditional methods for optimizing operators in computing devices are at least as follows: they cannot automatically determine the priority of operator optimization schemes, require a large amount of search resources, and cannot efficiently provide operator optimization schemes. Summary of the Invention

[0006] The present invention provides a method, computing device, computer-readable storage medium, and computer program product for optimizing operators, which can automatically and efficiently output optimization suggestions for operator performance tuning with priority.

[0007] According to a first aspect of the present invention, a method for optimizing operator performance is provided. The method includes: acquiring grouped data of a target operator via a hardware performance counter; parsing performance counter description information to extract multiple performance metrics and classifying the extracted performance metrics; comparing the values ​​of the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison result; and in response to triggering at least one decision rule, outputting at least one optimization strategy ordered by priority, wherein the optimization strategy at least includes information for indicating operator code changes.

[0008] In some embodiments, comparing the index values ​​of the classified performance indicators with the corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results includes: comparing the index values ​​of the classified performance indicators with the corresponding predetermined trigger thresholds in the optimization mapping decision table, wherein the optimization mapping decision table includes multiple structured decision rules, and each decision rule includes: hardware performance counter index name, trigger threshold, bottleneck classification, and recommended specific optimization strategy.

[0009] In some embodiments, the bottleneck classification indicates the performance bottleneck type corresponding to the decision rule, and the performance bottleneck type includes memory-constrained, pipeline bubble, and synchronization overload; and the recommended specific optimization strategy is used to distinguish the severity of the bottleneck based on different thresholds of the same hardware performance counter index, and recommend different levels of optimization strategy accordingly.

[0010] In some embodiments, in response to triggering at least one decision rule, outputting at least one priority-ordered optimization strategy includes: in response to determining that the same kernel function triggers multiple decision rules, determining the priority of the multiple decision rules based on a predetermined priority order; and generating the priority-ordered optimization strategy based on the determined priority via mutual exclusion rules, combination rules, and / or iterative convergence strategies, wherein the optimization strategy at least includes information for indicating operator code changes.

[0011] In some embodiments, parsing performance counter description information to extract multiple performance metrics and classifying the extracted multiple performance metrics includes: parsing performance counter description information to extract multiple performance metrics from the performance counter description information; and classifying the extracted multiple performance metrics into multiple metric types, each metric type corresponding to a specific performance dimension. The multiple metric types include: overview metrics, utilization metrics, stall analysis metrics, memory metrics, cache metrics, and load balancing metrics.

[0012] In some embodiments, the decision rules include multiple of the following: a first decision rule for optimizing burst transmission patterns; a second decision rule for adjusting the algorithm of the current operator; a third decision rule for eliminating execution bubbles in the pipeline; a fourth decision rule for reducing the overhead of synchronization operations; a fifth decision rule for increasing the configuration parameter value of burst transmission; a sixth decision rule for adapting blocks to the L2 cache; a seventh decision rule for improving the parallelism of task processing; and an eighth decision rule for migrating computational tasks to tensor kernel execution.

[0013] In some embodiments, comparing the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison result includes: obtaining an indicator of the execution unit of the current kernel function of the graphics processor being stalled due to loading instructions; comparing the indicator of the execution unit being stalled due to loading instructions with a preset first stall threshold and a preset second stall threshold, wherein the preset first stall threshold is greater than the preset second stall threshold; in response to determining that the indicator of the execution unit being stalled due to loading instructions is greater than or equal to the preset first stall threshold, determining that the current kernel function is a memory-limited type of first severity, and triggering a second decision rule; in response to determining that the indicator of the execution unit being stalled due to loading instructions is less than the preset first stall threshold and greater than or equal to the preset second stall threshold, determining that the current kernel function is a memory-limited type of second severity, and triggering a first decision rule.

[0014] In some embodiments, comparing the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison result includes: obtaining an indicator of the execution unit of the current kernel function of the graphics processor being stalled due to loading instructions; comparing the indicator of the execution unit being stalled due to loading instructions with a preset first stall threshold and a preset second stall threshold, wherein the preset first stall threshold is greater than the preset second stall threshold; in response to determining that the indicator of the execution unit being stalled due to loading instructions is greater than or equal to the preset first stall threshold, determining that the current kernel function is a memory-limited type of first severity, and triggering a second decision rule; in response to determining that the indicator of the execution unit being stalled due to loading instructions is less than the preset first stall threshold and greater than or equal to the preset second stall threshold, determining that the current kernel function is a memory-limited type of second severity, and triggering a first decision rule.

[0015] In some embodiments, comparing the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results further includes: in response to determining that the metric of the execution unit pausing due to loading instructions is less than a preset second pausing threshold, determining whether the metric of the execution unit pausing due to a synchronization barrier is greater than or equal to a predetermined third pausing threshold; and in response to determining whether the pausing metric of the execution unit pausing due to a synchronization barrier is greater than or equal to the predetermined third pausing threshold, determining that there are too many or too wide ranges of synchronization operations in the kernel function, and triggering a fourth decision rule.

[0016] In some embodiments, comparing the extracted performance index values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results further includes: in response to the number of instructions executed per cycle being less than a predetermined number of instructions threshold and the no-operation instruction ratio exceeding a predetermined ratio threshold, determining that there are multiple empty bubbles in the execution unit instruction pipeline, and triggering a third decision rule.

[0017] In some embodiments, comparing the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results further includes: comparing the burst transmission utilization metric with a predetermined utilization threshold to determine whether to trigger a fifth decision rule; comparing the L2 cache hit rate metric with a first predetermined hit threshold to determine whether to trigger a sixth decision rule; comparing the execution unit utilization metric with a second predetermined hit threshold to determine whether to trigger a seventh decision rule; and comparing the tensor kernel utilization metric with a predetermined utilization threshold to confirm whether to trigger an eighth decision rule.

[0018] In some embodiments, comparing the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results further includes: in response to determining that the current kernel function is a memory-limited type of first severity or a memory-limited type of second severity, determining whether there are minor bottlenecks such as bursty underutilization of transmission or cache jitter, so as to generate decision rules when multiple bottlenecks coexist in the current kernel function.

[0019] In some embodiments, the predetermined priority order includes: a second decision rule and a first decision rule triggered by a comparison of indicators regarding the execution unit's stall due to loading instructions, which are of first priority; a fourth decision rule triggered by a comparison of indicators regarding the execution unit's stall due to synchronization barriers, which are of second priority; a third decision rule triggered by a comparison of indicators regarding the number of instructions executed per cycle and the percentage of no-operation instructions, which are of third priority; and a fifth, sixth, seventh, and eighth decision rules triggered by indicators regarding burst transfer utilization, L2 cache hit rate, execution unit utilization, and tensor kernel utilization, respectively, which are of fourth priority.

[0020] In some embodiments, generating a priority-ordered optimization strategy based on the determined priority, via mutual exclusion rules, combination rules, and / or iterative convergence strategies, includes: in response to triggering a first decision rule and a second decision rule respectively for different threshold segments of the index that causes the same execution unit to stall due to loading instructions, outputting an optimization strategy corresponding to the second decision rule based on the mutual exclusion rules; and in response to completing each round of operator optimization, re-collecting grouped data of the target operator via hardware performance counters, so as to re-evaluate the triggering status of all decision rules based on the collected counter data.

[0021] In some embodiments, generating priority-ordered optimization strategies based on determined priorities via mutual exclusion rules, combination rules, and iterative convergence strategies includes: in response to determining a combination that simultaneously triggers the following decision rules, outputting the optimization strategies corresponding to each decision rule in the combination of decision rules: first decision rule, fifth decision rule; first decision rule, sixth decision rule; and first decision rule, third decision rule.

[0022] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy sorted by priority includes: in response to triggering a second decision rule, outputting the following optimization strategies: reducing the number of data traversals; merging multiple adjacent element-wise operators into a single kernel function, thereby eliminating the overhead of transferring intermediate kernel function results via high-bandwidth memory; and converting the computation of first rearranging image data into a matrix and then calling general matrix multiplication into directly performing convolution operations.

[0023] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy ordered by priority includes: in response to triggering a third decision rule, outputting the following optimization strategy: fully unrolling the loop by a compilation instruction to eliminate loop control overhead; and eliminating division and modulo calculations.

[0024] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy ordered by priority includes: in response to triggering a fourth decision rule, outputting the following optimization strategies: reducing the number of synchronization points; narrowing the synchronization range; and using asynchronous channels.

[0025] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy sorted by priority includes: in response to triggering a fifth decision rule, outputting the following optimization strategy: checking the vector width of the current data type; increasing the vector width; adjusting the loop step size and boundary handling logic accordingly; and ensuring that the data arrangement in memory meets the burst transfer alignment requirements.

[0026] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy sorted by priority includes: in response to triggering a sixth decision rule, outputting the following optimization strategy: determining the L2 cache capacity; calculating the working set size of a single block under the current block strategy; in response to the working set exceeding the L2 cache capacity, reducing the block size until the working set fits the L2 cache capacity; and selecting the block dimension with the highest data reuse rate while satisfying the L2 cache capacity constraint.

[0027] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy sorted by priority includes: in response to triggering the seventh decision rule, outputting the following optimization strategy: modifying the grid configuration at kernel startup to allow more stream processor cores to be used for computation; and using startup boundary qualifiers to increase the upper limit of the number of threads per stream processor core.

[0028] In some embodiments, in response to triggering at least one decision rule, outputting at least one optimization strategy sorted by priority includes: in response to triggering an eighth decision rule, outputting the following optimization strategy: identifying matrix multiplication or convolution computation kernels in kernel functions; determining the block size of tensor computation kernels; configuring data cache loading strategies for the input left and right matrices; replacing vector computation kernel operations with tensor computation kernel interface calls; and configuring a double-buffered ping-pong pipeline mechanism.

[0029] According to a second aspect of the present invention, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the computing device to perform the method of the first aspect of the present invention.

[0030] According to a third aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0031] According to a fourth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0032] This invention achieves this by using hardware performance counters to collect grouped data from target operators, aggregating the collected counter data into performance counter description information. The performance counter description information is then parsed to extract and classify multiple performance metrics. This not only reduces data acquisition noise and ensures efficiency through hardware performance counter-based grouped data collection, but also compares the extracted performance metric values ​​with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule. This allows for the automatic triggering of decision rules through the comparison of performance metrics with predetermined trigger thresholds. Furthermore, in response to triggering at least one decision rule, at least one priority-ranked optimization strategy is output. Each optimization strategy includes information indicating operator code changes. This invention enables the mapping of PFC metrics to specific optimization strategies and code change indications. Moreover, the priority ranking quantifies the priority of optimization strategies, thus allowing for the automatic and efficient output of priority-based operator performance tuning optimization suggestions. Furthermore, this invention, by executing a standardized process of "collecting PFC data → extracting performance metrics → comparing metric values ​​with predetermined trigger thresholds → obtaining optimization strategies," can obtain optimization suggestions comparable to those of experts. This significantly reduces the human barrier to performance tuning of computing devices (e.g., GPUs) and makes the optimization strategies directly actionable.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0034] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0035] Figure 1 The schematic diagram illustrates the architecture of a computing system according to some embodiments of the present invention.

[0036] Figure 2 A schematic diagram of a computing device according to some embodiments of the present invention is shown.

[0037] Figure 3A flowchart of a method for optimizing operator performance according to some embodiments of the present invention is shown.

[0038] Figure 4 A flowchart of a method for determining whether at least one decision rule is triggered based on comparison results, according to some embodiments of the present invention, is shown.

[0039] Figure 5 A flowchart illustrating an implementation of a method for triggering the fourth and third decision rules according to some embodiments of the present invention is shown.

[0040] Figure 6 A flowchart of a method for generating priority-ordered optimization strategies according to some embodiments of the present invention is shown.

[0041] Figure 7 A flowchart of a method for outputting an optimization strategy in response to triggering a sixth decision rule, according to some embodiments of the present invention, is shown.

[0042] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0043] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0044] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0045] As mentioned earlier, the shortcomings of traditional methods for optimizing operators in computing devices are at least as follows: they cannot automatically determine the priority of operator optimization schemes, require a large amount of search resources, and cannot efficiently provide operator optimization schemes. To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of the present invention propose a method for optimizing operator performance. The present invention reduces data acquisition noise and ensures efficiency through hardware performance counter-based grouped data acquisition; it automatically triggers decision rules by comparing performance metrics with predetermined trigger thresholds; and it maps PFC metrics to specific optimization strategies and code change instructions. Furthermore, by prioritizing optimization strategies, the present invention quantifies their priority. Therefore, the present invention can automatically and efficiently output priority-based optimization suggestions for operator performance tuning. Further, by executing a standardized process of "acquiring PFC data → extracting performance metrics → comparing metric values ​​with predetermined trigger thresholds → obtaining optimization strategies," the present invention can obtain optimization suggestions comparable to those of experts. This significantly reduces the human barrier to operator performance tuning for computing devices (e.g., GPUs) and makes optimization strategies directly operable. The following combination Figure 1 A schematic diagram illustrating the architecture of a computing system 100 implementing a method for optimizing operator performance according to some embodiments of the present invention is provided. Figure 1 As shown, the computing system 100 includes, for example, a hardware performance counter data acquisition layer 102, an indicator extraction layer 104, a threshold determination layer 106, a strategy matching layer 108, and an optimization output layer 110. Regarding the hardware performance counter data acquisition layer 102, it is configured to: acquire grouped data of the target operator in order to aggregate the acquired counter data into performance counter description information. In some embodiments, the hardware performance counter data acquisition layer 102 includes a raw counter data acquisition unit 112 and a parsing unit 114. The raw counter data acquisition unit 112, for example, uses the suprof tool to acquire a predetermined number of groups (e.g., but not limited to five groups) of counters from the target kernel function, with each group of counters corresponding to a type of hardware performance counter. The parsing unit 114, for example, uses the doctor_perf tool to aggregate the dispersed raw counter data acquired from the predetermined number of groups (e.g., but not limited to five groups) of counters into structured performance counter description information (e.g., a PFC report in report.txt or report.json format). Regarding the metric extraction layer 104, it is configured to parse performance counter description information to extract multiple performance metrics and classify the extracted performance metrics. In some embodiments, the metric extraction layer 104 includes: an overview metric extraction unit 116, a utilization metric extraction unit 118, a stall metric extraction unit 120, a memory metric extraction unit 122, a cache metric extraction unit 124, and a balanced metric extraction unit 126. The overview metric extraction unit 116 is used to extract overview metrics, such as execution cycles, register usage, and other operational overviews. The utilization metric extraction unit 118 is used to extract utilization metrics, such as execution unit (EU) utilization, instructions per cycle (IPC), Tcore utilization, and other efficiency metrics. The stall metric extraction unit 120 is used to extract stall metrics, such as the percentage distribution of EUs stalling for various reasons. The memory metric extraction unit 122 is used to extract memory metrics, such as read / write bandwidth, burst utilization, and other memory access efficiency. The cache metric extraction unit 124 is used to extract cache metrics, such as L2 read / write hit rate. The load balancing metric extraction unit 126 is used to extract load balancing metrics, such as the load balancing degree between stream processing clusters (SPCs). Regarding the threshold determination layer 106, it is configured to compare the value of the extracted performance metric with the corresponding predetermined trigger threshold. Regarding the strategy matching layer 108, it is configured to determine whether to trigger at least one decision rule based on the comparison result. Regarding the optimization output layer 110, it is configured to output at least one optimization strategy sorted by priority in response to triggering at least one decision rule.

[0046] The following combination Figure 2 A computing apparatus 200 is described, according to some embodiments of the present invention, for implementing a method for optimizing operator performance. For example... Figure 2As shown, the computing device 200 may have one or more processing units, including dedicated processing units such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or general-purpose computing on graphics processing units (GPGPUs), as well as general-purpose processing units such as CPUs. In some embodiments, the computing device 200 includes: a performance counter description information aggregation unit 202, a multiple performance metric extraction and classification unit 204, a decision rule triggering unit 206, and an optimization strategy output unit 208. In some embodiments, the performance counter description information aggregation unit 202, the multiple performance metric extraction and classification unit 204, the decision rule triggering unit 206, and the optimization strategy output unit 208 are software modules running on one or more processors.

[0047] Regarding the performance counter description information aggregation unit 202, it is used to collect grouped data of the target operator via hardware performance counters, so as to aggregate the collected counter data into performance counter description information.

[0048] Regarding the multi-performance index extraction and classification unit 204, it is used to parse the performance counter description information in order to extract multiple performance indicators and classify the extracted multiple performance indicators.

[0049] Regarding the decision rule triggering unit 206, it is used to compare the value of the extracted performance index with the corresponding predetermined trigger threshold, so as to determine whether to trigger at least one decision rule based on the comparison result.

[0050] Regarding the optimization strategy output unit 208, it is used to output at least one optimization strategy sorted by priority in response to triggering at least one decision rule, and the optimization strategy contains at least information for indicating operator code changes.

[0051] The following will combine Figure 3 A method 300 for optimizing operators of a computing device, according to embodiments of the present invention, is described. It should be understood that method 300 can, for example, be implemented in... Figure 2 The described computing device 200 performs the operation. Method 300 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0052] At step 302, the computing device 200 performs grouped data acquisition on the target operator via a hardware performance counter, so as to aggregate the acquired counter data into performance counter description information.

[0053] In some embodiments, via the hardware performance counter data acquisition layer, the suprof tool is used to perform grouped data acquisition of a predetermined number of counters (e.g., but not limited to five groups) on the target kernel function, with each group of counters corresponding to a type of hardware performance counter.

[0054] The doctor_perf tool aggregates the scattered raw counter data collected from a predetermined number of counter groups (e.g., but not limited to five groups) into structured performance counter description information. In some embodiments, the performance counter description information includes both report.txt and report.json formats to provide standardized data input.

[0055] At step 304, the computing device 200 parses the performance counter description information in order to extract multiple performance indicators and classify the extracted multiple performance indicators.

[0056] Regarding performance counter description information, in some embodiments, it is, for example, a PFC report.

[0057] Regarding multiple performance indicators, there are, for example, more than 30 performance indicators.

[0058] In some embodiments, a metric extraction layer extracts multiple performance metrics (e.g., but not limited to more than 30 performance metrics) from performance counter description information (e.g., PFC reports), and classifies these performance metrics into six categories based on functional dimensions: overview metrics, utilization metrics, stall metrics, memory metrics, cache metrics, and load balancing metrics. It should be understood that classifying the metrics allows subsequent threshold determination to be performed based on specific performance characteristics.

[0059] Overview metrics, for example, are used to reflect operational status, including execution cycles, register usage, and other operational details.

[0060] Regarding utilization metrics, these include at least efficiency metrics such as EU utilization, IPC, and Tcore utilization. Utilization metrics are used to measure the efficiency of a target operator in utilizing various hardware computing resources during execution, and are a core basis for identifying computing bottlenecks and resource idleness.

[0061] Regarding stagnation indicators, these include, for example, at least the percentage distribution of EU stagnation due to various reasons.

[0062] Regarding memory metrics, these include, at a minimum, memory access efficiency such as read / write bandwidth and burst utilization.

[0063] Regarding caching metrics, these include, for example, at least L2 read / write hit rate.

[0064] Regarding load balancing metrics, these include, for example, at least the load balancing degree between SPCs.

[0065] Table 1 shows the relevant information regarding the overview metrics. As shown in Table 1, the overview metric categories include, but are not limited to, the following metrics: job cycles, latency, average active compute block interface cycles (cbi_cycles_avg), maximum active compute block interface cycles (cbi_cycles_max), minimum active compute block interface cycles (cbi_cycles_min), active execution unit cycles (eu_cycles), register usage per thread (registers / thread), data read per stream processor cluster (data_read per SPC), and data write per stream processor cluster (data_write per SPC). After extracting the overview metrics, the values ​​of each metric are interpreted one by one to evaluate the running status of the target operator. For example, the job cycles metric is converted into a time metric in microseconds using the following formula: time(µs) = job_cycles / freq_mhz. Where freq_mhz is the operating frequency of the graphics processor (in megahertz). The resulting time metric directly reflects the total time consumed by the kernel function from startup to termination, serving as a basic reference value for overall performance evaluation.

[0066] In some embodiments, the minimum active cycle time metric of the compute block interface is compared with the average active cycle time metric of the compute block interface. The difference between cbi_cycles_min and cbi_cycles_avg reflects the idle time spent by the stream processor cluster that completes its work fastest while waiting for other stream processor clusters to synchronize. If the difference is determined to exceed a predetermined idle threshold, it is determined that there is an excessive synchronization wait overhead. Table 1 Overview of Indicators

[0067] Table 2 illustrates the relevant information regarding utilization metrics. As shown in Table 2, utilization metrics include, but are not limited to, the following: percentage of execution unit utilization (eu_util_rate_pct), instructions per cycle (IPC), general matrix multiplication unit utilization (gemm_util), and tensor computation core utilization (tcore_util). For example, the IPC value is compared with a predetermined instruction count threshold (e.g., 0.3). IPC reflects the average number of effective instructions issued and completed by the execution unit per clock cycle and is a core metric for measuring pipeline efficiency. In response to determining that IPC < 0.3, the current kernel function is marked as a pipeline efficiency evaluation state. This determination itself does not directly trigger decision rule R3 but is part of a composite condition determination. Specifically, when IPC < 0.3, a comprehensive determination is also required in conjunction with EU Stall by Nop % Table 2 Utilization Rate Indicators

[0068] Table 3 shows the relevant information about the stagnation indicator. As shown in Table 3, the stalling metrics include, but are not limited to, the following: the percentage of execution unit stalled due to load instructions (EU Stall by Load %), which indicates the percentage of time the execution unit stalled while waiting for memory data to return; the percentage of execution unit stalled due to load instructions with non-fixed latency (EU Stall by Load_nonfix %), which indicates the percentage of time the execution unit stalled due to load operations with non-fixed latency (such as global memory access); the percentage of execution unit stalled due to no-operation instructions (EU Stall by Nop %), which indicates the percentage of time the execution unit stalled due to no-operation instructions occupying the pipeline; the percentage of execution unit stalled due to instruction cache misses (EU Stall by InstrMiss %), which indicates the percentage of time the execution unit stalled due to instruction cache misses; the percentage of execution unit stalled due to data dependencies (EU Stall by DE %), which indicates the percentage of time the execution unit stalled due to data dependencies (subsequent instructions waiting for the results of preceding instructions); and the percentage of execution unit stalled due to synchronization barriers (EU Stall by Barrier %), which indicates the percentage of time the execution unit stalled due to waiting for synchronization barriers. Through the above methods, this invention realizes a systematic automatic judgment and decision rule triggering based on stagnation analysis indicators, and supports refined diagnosis of bottleneck types and iterative optimization after bottleneck migration through auxiliary indicators, which significantly improves the accuracy and comprehensiveness of performance tuning. Table 3 Stagnation Indicators

[0069] Table 4 shows the relevant information regarding memory metrics. As shown in Table 4, memory metric categories include, but are not limited to, the following metrics: CUs Read Bytes / Cycle (also known as LDM read bandwidth), used to indicate the stream processor cluster's memory read bandwidth (number of bytes read per cycle); CUsWrite Bytes / Cycle (also known as STM write bandwidth), used to indicate the stream processor cluster's memory write bandwidth (number of bytes written per cycle); L1P5 Access Bandwidth (LSC Access L1P5 bandwidth), used to indicate the total access bandwidth of the L1P5 cache layer; compute_mem_ratio, used to indicate the ratio of compute operations to memory access operations; burst transfer mode utilization (burst_score); and L1P5 Core Gap Ratio (l1p5_core_gap_ratio), used to indicate the idle ratio between the L1P5 cache and the cores.

[0070] Taking the L1P5 total bandwidth metric as an example, this metric reflects the overall throughput of the kernel function accessing data through the L1P5 cache layer. The total L1P5 bandwidth is compared with the peak L1P5 bandwidth in the reference bandwidth table. If the total L1P5 bandwidth is determined to be significantly lower than the peak bandwidth (e.g., less than 60% of the peak bandwidth), then the memory access throughput efficiency is considered low, possibly caused by at least one of the following reasons: unmerged memory accesses, insufficient burst transfer utilization, or low cache hit rate. Taking the burst transfer mode utilization metric as an example, the burst_score value is compared with a predetermined utilization threshold (e.g., 50%). If the burst_score is determined to be less than the predetermined utilization threshold (e.g., 50%), then the current kernel function's burst transfer mode is considered underutilized, triggering the fifth decision rule R5.

[0071] In some embodiments, the formula for calculating burst_score is: burst_score = actual burst length / maximum burst length × 100%. Table 4 Memory Indicators

[0072] Table 5 illustrates relevant information about cache metrics. As shown in Table 5, cache metrics quantify the efficiency and hit rate of target operators accessing the L2 cache during execution, and are the core basis for identifying cache thrashing and the optimization space of the tiling strategy. Cache metric categories include, but are not limited to, the following metrics: L2 Read Hit Rate, L2 Write Hit Rate, L2 Read Requests, and L2 Write Requests. Table 5 Cache Metrics Table

[0073] Table 6 illustrates the relevant content regarding load balancing metrics. As shown in Table 6, load balancing metrics quantify the degree of workload distribution among various hardware resources across different stream processor clusters during the execution of the target operator. They are the core basis for identifying load imbalance problems and their root causes. Load balancing metric categories include, but are not limited to, the following: Execution Unit Load Balance (EU balance), indicating the load balance of execution units across different stream processor clusters; L2 Read Balance, indicating the load balance of L2 cache read operations across different stream processor clusters; and L2 Write Balance, indicating the load balance of L2 cache write operations across different stream processor clusters.

[0074] Taking the execution unit load balancing index as an example, this index is obtained by statistically analyzing the active cycles of execution units in each stream processor cluster. Its value typically ranges from 0 to 1. The closer the value is to 1.0, the more balanced the computational load among the stream processor clusters. A value significantly lower than 1.0 (e.g., below 0.85) indicates a large difference in the computational workload undertaken by different stream processor clusters. When the load balancing index is significantly lower than 1.0, at least one of the following measures can be taken: adjust the task partitioning granularity to distribute approximately equal workloads to each stream processor cluster; modify the grid parameters in the kernel startup configuration so that the number of data blocks is divisible by the number of stream processor clusters, avoiding some stream processor clusters being assigned extra workloads; for operators with irregular data shapes (e.g., variable-length sequence processing), use dynamic scheduling strategies (such as work stealing or round-robin allocation) instead of static partitioning to balance the runtime load of each stream processor cluster. Table 6 Equilibrium Index Table

[0075] At step 306, the computing device 200 compares the index values ​​of the classified performance indicators with the corresponding predetermined trigger thresholds in order to determine whether to trigger at least one decision rule based on the comparison results. The decision rule corresponds to the recommended optimization strategy for the operator.

[0076] The predetermined trigger threshold is a critical value obtained based on historical kernel function tuning experience statistics, used to distinguish between performance bottlenecks that are in a tolerable state and those that require intervention.

[0077] Regarding the decision rules, they include, for example, several of the following: a first decision rule (or R1) for optimizing burst transmission patterns; a second decision rule (or R2) for adjusting the algorithm of the current operator; a third decision rule (or R3) for eliminating execution bubbles in the pipeline; a fourth decision rule (or R4) for reducing the overhead of synchronization operations; a fifth decision rule (or R5) for increasing the configuration parameter value of burst transmission; a sixth decision rule (or R6) for adapting block partitioning to the L2 cache; a seventh decision rule (or R7) for increasing the parallelism of task processing; and an eighth decision rule (or R8) for migrating computational tasks to tensor kernel execution.

[0078] In some embodiments, the threshold determination layer compares the values ​​of the classified performance metrics with the corresponding predetermined trigger thresholds in the optimization mapping decision table. The optimization mapping decision table includes multiple structured decision rules, each including: a hardware performance counter metric name, a trigger threshold, a bottleneck classification, and a recommended specific optimization strategy. The bottleneck classification indicates the type of performance bottleneck corresponding to the decision rule, including memory-constrained, pipeline bubble (or cavitation), and excessive synchronization. The recommended specific optimization strategy is used to distinguish the severity of the bottleneck based on different thresholds of the same hardware performance counter metric and recommend different levels of optimization strategies accordingly.

[0079] Table 7 shows the optimization strategy table containing multiple decision rules. As shown in Table 7, for some decision rules, a single-index threshold comparison method is used, that is, only the index value of a single performance index is compared with the corresponding predetermined trigger threshold. If the index value exceeds or reaches the threshold, the corresponding decision rule is triggered. For example, decision rule R1 is triggered if the index of execution unit stalling due to loading instructions (EU Stall by Load) > 30%. If the EU Stall by Load index > 50%, the current kernel function is determined to be a severely memory-bound type, and decision rule R2 is triggered. For example, decision rule R4 is triggered if the index of execution unit stalling due to synchronization barrier (EU Stall by Barrier) > 10%, the current kernel function has too many or too wide range of synchronization operations, and decision rule R4 is triggered.

[0080] In some embodiments, a composite condition comparison method is used for some decision rules. That is, the values ​​of multiple performance indicators are compared with their corresponding predetermined trigger thresholds, and it is determined whether multiple conditions are met simultaneously. The decision rule is triggered only when all predetermined trigger thresholds are met. In the step of extracting performance indicators, in addition to extracting stagnation analysis indicators, IPC from the utilization indicators and the percentage of no-operation instructions (EU Stall by Nop) from the stagnation analysis indicators are also extracted. For example, if the decision rule is R3, and IPC < 0.3 and pipeline cavitation (Nop) > 15%, then decision rule R3 is triggered.

[0081] Table 7. Optimization Strategy Table Containing Multiple Decision Rules It should be noted that the specific values ​​in the above embodiments (such as thresholds of 50%, 30%, 10%, 0.3%, 15%, 80%, 40%, etc.) are all exemplary settings. In practical applications, they can be adaptively adjusted according to specific hardware architecture and performance tuning requirements. These adjustments are all within the protection scope of this invention.

[0082] At step 308, in response to the computing device 200 triggering at least one decision rule, at least one optimization strategy sorted by priority is output, and the optimization strategy includes at least information for indicating operator code changes.

[0083] The computing device 200, for example, via a policy matching layer, matches a set of recommended optimization policies for each triggered decision rule. The policy description at least indicates the name of the specific performance optimization technique. The names of the specific performance optimization techniques include, for example, burst mode, double-buffer, loop unrolling, etc. In some embodiments, the policy description also indicates the applicable scenario for the specific optimization technique. This provides engineers or AI agents with a clear direction for optimization.

[0084] In some embodiments, for the triggered decision rule R1, the corresponding performance optimization method is to enable burstmode, for example, replacing the short vector data type bf162 with the long vector data type bf168. Before the change, the kernel function uses the short vector data type bf162, and reads data from one thread-local register (TLR) for each WTI load operation. Specifically, the data type is declared as bf162 a; the load operation is implemented by calling wti::__load_activation(&a, A, Coordinate(n,c, h, w)), where Coordinate(n,c,h,w) specifies the access position in tensor A in the batch dimension n, channel dimension c, height dimension h, and width dimension w; the loop step size in the height direction h is 4.

[0085] After the change, the short vector data type bf162 is replaced with the long vector data type bf168 (corresponding to the burst transfer mode with BURST=4), enabling each WTI load operation to read data from four thread-local registers (i.e., four TLRs) at once. Specifically, the data type declaration is changed to bf168 a; the load operation is still implemented by calling wti::__load_activation(&a, A,Coordinate(n, c, h, w)), but because the data type has been changed to bf168, the compiler automatically generates a burst transfer instruction of length 4; the loop step size of the height direction h is adjusted from 4 to 16 (because the height range covered by each load is increased by 4 times).

[0086] It should be understood that replacing bf162 with bf168 reduces the number of instruction loads to one-quarter of the original number for the same amount of data, thereby reducing instruction issuance pressure and memory request queue queuing latency, and effectively improving memory bandwidth utilization.

[0087] In some embodiments, in response to triggering decision rule R1, computing device 200 also outputs an optimization strategy that replaces scalar tensor internal function (STI) interface with thread bundle tensor internal function (WTI) interface. The WTI interface is a read / write interface specifically optimized for tensor data, which automatically utilizes the hardware's tensor layout to achieve more efficient burst transfers.

[0088] The computing device 200 can also output optimization strategies that enable data prefetching, such as using a Tensor Buffer to preload upcoming data into the L2 cache, reducing cache miss latency on the first access. For example, prefetch call instructions can be inserted at appropriate locations within the kernel function. Prefetch calls are implemented by calling the `wti::__preload_tensor_buffer` function. The insertion point of the prefetch call should be just before the current computation block is completed and the data for the next computation block is about to be used, ensuring that the prefetch operation has sufficient time to load data into the cache. Additionally, tensor buffer attributes can be configured on the host side.

[0089] Specifically, the `configTensorBuffer` member function of the tensor object is called (e.g., `A.configTensorBuffer(0, A.size())`) to configure the starting offset and size of the tensor buffer. The first parameter represents the starting offset of the buffer (in elements), and the second parameter represents the size of the buffer (in elements). This configuration operation is performed on the host side before the kernel function starts, and is used to notify the hardware to allocate buffer resources for the specified tensor.

[0090] In some embodiments, the computing device 200, for example, recommends decision rules and, via an optimization output layer, summarizes the optimization strategies matched by all triggered decision rules to output a list of optimization strategies in order of priority; and provides information on operator code changes corresponding to the optimization strategies. In some embodiments, the optimization output layer also provides an estimate of expected performance gains determined based on historical tuning data.

[0091] The optimization strategies corresponding to the eight decision rules (R1 to R8) defined in this invention are divided into the following three levels according to their hierarchical function and technical characteristics: implementation-level optimization layer, algorithm-level optimization layer, and hardware resource optimization layer.

[0092] Implementation-level optimization refers to optimization strategies that improve execution efficiency by improving code implementation and memory access patterns while keeping the kernel function algorithm logic and computation strategy unchanged. The characteristic of this type of optimization is that it does not change the computational complexity or total data access volume of the algorithm, but rather reduces the number of instructions and latency by more efficiently utilizing hardware features.

[0093] Regarding the burst transfer mode optimization, corresponding to the first decision rule (R1), this optimization enables hardware burst transfer capability by replacing the short vector data type (bf162) with the long vector data type (bf168), allowing four consecutive data units to be transferred per memory transaction. In practice, on a 16 SPC platform, the computation block interface cycle count (cbi_cycles) of the ReLU operator was reduced by 21.7% from the baseline.

[0094] Regarding loop unrolling optimization, this corresponds to the third decision rule (R3). This optimization fully unrolls the loop body using compiler directives (e.g., `#pragma unroll`), eliminating the overhead of loop control instructions (comparisons, branch jumps) and providing the compiler with a larger instruction scheduling window to discover instruction-level parallelism. Real-world testing shows that depthwise separable convolution (DWC) and the Softmax operator achieve a 30% to 100% improvement in instructions per cycle on a 16 SPC platform.

[0095] In the above scheme, the present invention can reduce data acquisition noise and ensure efficiency by grouping data acquisition through hardware performance counters; realize the automatic triggering of decision rules by comparing performance indicators with predetermined trigger thresholds; and realize the mapping of PFC indicators to specific optimization strategies and code change instructions. Moreover, the priority of optimization strategies is quantified by priority sorting. Therefore, the present invention can automatically and efficiently output optimization suggestions for operator performance tuning with priority.

[0096] The following will combine Figure 4 The present invention describes a method 400 for determining whether at least one decision rule is triggered based on a comparison result. It should be understood that method 400 may, for example, be used in... Figure 2 The described computing device 200 performs the operation. Method 400 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0097] At step 402, in response to the computing device 200 determining that the same kernel function triggers multiple decision rules, the priority of the multiple decision rules is determined based on a predetermined priority order.

[0098] Regarding the predetermined priority order, it includes, for example, the following: a second decision rule and a first decision rule triggered by comparing indicators of execution unit stalling due to loading instructions are of first priority; a fourth decision rule triggered by comparing indicators of execution unit stalling due to synchronization barriers is of second priority; a third decision rule triggered by comparing indicators of the number of instructions executed per cycle and the percentage of no-operation instructions is of third priority; and a fifth, sixth, seventh, and eighth decision rules triggered by burst transfer utilization, L2 cache hit rate, execution unit utilization, and tensor kernel utilization, respectively, are of fourth priority.

[0099] Table 8 shows the priority order of multiple decision rules. As shown in Table 8, the computing device 200 assigns the first priority (highest priority) to the second decision rule (R2). It should be understood that in this invention, when the indicator of execution unit stalling due to loading instructions is greater than or equal to 50%, it indicates that more than half of the execution unit's active cycles are waiting for memory data. The root cause of this bottleneck is the excessive data access volume of the algorithm itself. Under this priority, in response to determining that EU Stall byLoad is greater than or equal to 50% (triggering R2), even if other decision rules (e.g., R1, R5, or R6) are triggered simultaneously, the computing device still prioritizes outputting the algorithm-level change strategy corresponding to R2.

[0100] The computing device 200 assigns the second priority to the eighth decision rule (R8). In response to determining that R8 is triggered (tcore_util equals 0% and the algorithm includes matrix multiplication or convolution operations), and simultaneously triggering other decision rules (e.g., R1 or R5), the computing device 200, after outputting the optimization strategy corresponding to R2, prioritizes outputting the Tcore migration strategy corresponding to R8, and then processes the other rules. This approach achieves ordered optimization execution when multiple decision rules are triggered, avoiding redundant work or insufficient performance improvement due to improper optimization order, and ensuring the systematic nature and efficiency of the optimization strategy in multi-bottleneck scenarios. Table 8. Priority order of multiple decision rules

[0101] At step 404, the computing device 200 generates an optimization strategy sorted by priority based on the determined priority, via mutual exclusion rules, combination rules and / or iterative convergence strategies. The optimization strategy includes at least information for indicating operator code changes.

[0102] Methods for generating priority-ordered optimization strategies include: mutual exclusion rules, combinatorial rules, and / or iterative convergence strategies.

[0103] A method for generating priority-ordered optimization strategies via mutual exclusion rules includes, for example, the following: in response to different threshold segments of an indicator that causes the same execution unit to stall due to loading instructions, triggering a first decision rule and a second decision rule respectively, an optimization strategy corresponding to the second decision rule is output based on the mutual exclusion rules.

[0104] For example, the first and second decision rules are mutually exclusive, targeting different threshold ranges for the same metric (EU Stall by Load). When Load Stall > 50%, only R2 is triggered (R1 is not triggered simultaneously) because the algorithm change strategy of R2 implicitly includes the implementation optimization of R1.

[0105] Methods for generating priority-ordered optimization strategies via iterative convergence strategies include, for example, re-collecting grouped data of the target operators via hardware performance counters after each round of operator optimization, so as to re-evaluate the triggering state of all decision rules based on the collected counter data.

[0106] For example, after resolving the memory-bound issue, the kernel may become compute-bound, at which point the R3 / R7 rules that were not previously triggered may become new trigger points.

[0107] Methods for generating priority-ordered optimization strategies via combination rules include, for example, outputting optimization strategies corresponding to each decision rule in the combination of decision rules in response to determining a combination that simultaneously triggers the following decision rules: first decision rule, fifth decision rule; first decision rule, sixth decision rule; and first decision rule, third decision rule.

[0108] The combination of the first and fifth decision rules can doubly improve memory access efficiency.

[0109] The combination of the first and sixth decision rules can simultaneously improve access patterns and cache utilization.

[0110] The combination of the first and third decision rules reduces control commands after loop expansion, creating more command slots for burst mode.

[0111] In the above scheme, by defining the priority order and conflict resolution strategies of multiple rules (such as mutual exclusion, combined execution, iterative convergence, etc.), a reasonable execution order can be given when multiple rules are triggered, thus avoiding confusion in the optimization direction.

[0112] The following will combine Figure 5 This invention describes a method 500 for triggering a fourth decision rule and a third decision rule, according to embodiments of the present invention. It should be understood that method 500 can, for example, be implemented in... Figure 1 The described computing system executes at 100 points; it can also be implemented in... Figure 2 The described computing device 200 performs the operation. Method 500 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0113] Method 500 is an indicator of an execution unit stalling due to loading instructions.

[0114] At step 502, the computing device 200 obtains an indicator that the execution unit of the current kernel function of the graphics processor has stalled due to loading instructions.

[0115] Indicators regarding the stalling of the current kernel function execution unit of the graphics processing unit due to loading instructions, such as the EU Stall by Load metric, are common sources of performance bottlenecks in the GPU kernel. It should be understood that this invention prioritizes indicators of stalling due to loading instructions because the kernels of most unoptimized computing devices (e.g., GPUs) will initially exhibit memory access efficiency issues.

[0116] At step 504, the computing device 200 compares the index of the execution unit stalling due to loading instructions with a preset first stall threshold and a preset second stall threshold, wherein the preset first stall threshold is greater than the preset second stall threshold.

[0117] Regarding the preset first threshold, it is, for example, but not limited to, 50%.

[0118] Regarding the preset second threshold, it is, for example, but not limited to, 30%.

[0119] For example, the computing device 200 first compares the EU Stall by Load index with a preset first threshold (50%) and a preset second threshold (30%).

[0120] At step 506, in response to the computing device 200 determining that the index of the execution unit stalling due to loading instructions is greater than or equal to a preset first stall threshold, the current kernel function is determined to be a memory-limited type of the first severity, and a second decision rule is triggered.

[0121] Regarding the first severity of memory-bound types, such as severe Memory-bound.

[0122] Regarding the second optimization strategy, it is, for example, the second optimization strategy at the algorithm level, namely optimization strategy R2.

[0123] For example, if the computing device 200 determines that the EU Stall by Load index exceeds 50%, it is considered a serious memory-bound problem, requiring an algorithm-level change, namely optimization strategy R2.

[0124] At step 508, in response to the computing device 200 determining that the index of the execution unit stalling due to loading instructions is less than a preset first stall threshold and greater than or equal to a preset second stall threshold, the current kernel function is determined to be a memory-limited type of the second severity, and the first decision rule is triggered.

[0125] The first severity of memory-bound types is, for example, the general Memory-bound.

[0126] Regarding the first optimization strategy, it is, for example, the first optimization strategy at the algorithm level, namely optimization strategy R1.

[0127] For example, if the computing device 200 determines that the EU Stall by Load index is between 30% and 50%, it is judged as a general memory-bound index, which can be alleviated by implementation-level optimization, i.e., optimization strategy R1.

[0128] In step 510, in response to the computing device 200 determining that the index of the execution unit pausing due to the loading instruction is less than a preset second pausing threshold, it is determined whether the index of the execution unit pausing due to the synchronization barrier is greater than or equal to a predetermined third pausing threshold.

[0129] Regarding the preset third threshold, it is, for example, but not limited to, 10%.

[0130] For example, when the EU Stall by Load metric is not severe, such as less than 30%, it is necessary to further compare and examine the metrics for execution unit stalling due to synchronization barriers, i.e., the EU Stall by Barrier metric. For example, confirm whether the metrics for execution unit stalling due to synchronization barriers exceed 10%.

[0131] At step 512, in response to the computing device 200 determining that the stagnation index of the execution unit caused by the synchronization barrier is greater than or equal to a predetermined third stagnation threshold, it determines that there are too many or too wide ranges of synchronization operations in the kernel function and triggers the fourth decision rule.

[0132] For example, if computing device 200 determines that EU Stall by Barrier exceeds 10%, it indicates that there are too many or too wide ranges of synchronization operations in the kernel. In this case, the fourth optimization strategy, namely R4, is matched.

[0133] In the above scheme, the embodiments of the present invention adopt a judgment order with decreasing priority: first, check the EU Stallby Load indicator (a common source of performance bottlenecks in GPU Kernel); when Load Stall is not serious, check the stagnation caused by synchronization barrier; then check pipeline bubble determination, burst utilization, L2 cache hit rate, EU utilization and Tcore utilization in sequence, and additionally check whether there are secondary bottlenecks after each branch, supporting the generation of combined optimization strategies, reflecting the common situation of multiple bottlenecks coexisting in actual Kernel.

[0134] Through the above methods, the present invention ensures that a reasonable execution order and combination scheme can be given when multiple rules are triggered, avoiding confusion in the optimization direction and improving the systematicness and effectiveness of the optimization strategy.

[0135] The following will combine Figure 6 The present invention describes a method 600 for generating a priority-ordered optimization strategy, according to embodiments of the invention. It should be understood that method 600 can, for example, be implemented in... Figure 2 The described computing device 200 performs the operation. Method 600 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0136] At step 602, in response to triggering the second decision rule, the following optimization strategies are output: reduce the number of data traversals; merge multiple adjacent element-wise operators into a single kernel function, thereby eliminating the overhead caused by transferring intermediate kernel function results through high-bandwidth memory; and convert the computation of first rearranging image data into a matrix and then calling general matrix multiplication into directly performing convolution operations.

[0137] For example, taking Softmax as an example, the standard implementation requires three data traversals (Pass 1 to find the maximum value, Pass 2 to find the exponent sum, and Pass 3 to normalize). The Online Softmax algorithm combines the above three traversals into one traversal. During the single traversal, the online update formula is used to maintain all intermediate states at the same time, thereby reducing the amount of data access to 1 / 3 of the original, and the overall performance can be improved by about 44%.

[0138] Operator fusion refers to merging multiple adjacent element-wise operators into a single kernel (e.g., merging BatchNorm + ReLU into a single kernel) to eliminate the overhead of intermediate results being passed through HBM. The fusion criterion is that fusion can be performed when the data dependency between two operators is element-wise or broadcast, and the number of registers used after fusion does not exceed hardware limitations. Operator fusion typically eliminates approximately 30% to 60% of redundant HBM accesses.

[0139] For example, for convolution operators, the two-step strategy of rearranging image data into a matrix and performing general matrix multiplication is replaced by direct convolution or Winograd transform. Rearranging image data into a matrix requires expanding the input data into a matrix form and writing it to a temporary buffer, introducing additional HBM writes and reads. Converting to direct convolution operations allows computation to be performed directly on the original data layout, thus eliminating the intermediate buffer.

[0140] At step 604, in response to triggering the third decision rule, the following optimization strategies are output: fully unroll the loop by a compilation instruction to eliminate loop control overhead; and eliminate division and modulo calculations.

[0141] It is understandable that the bottleneck mechanism of the third decision rule is that: the proportion of loop control instructions (comparison, jump) is too high, which compresses the proportion of effective calculation instructions; and the division and modulo operations generate multi-cycle delays, causing subsequent instructions to queue up and wait.

[0142] Based on this, loop unrolling (e.g., unrolling the loop body using the `#pragma unroll` compiler directive) can eliminate the overhead of loop control instructions (comparisons, branch jumps) while providing the compiler with a larger instruction scheduling window to discover more instruction-level parallelism. Loop unrolling can improve IPC by 30% to 100%. Regarding the elimination of division and modulo operations, when the divisor is a power of 2, bitwise operations are used instead (right shift replaces division, bitwise AND replaces modulo); when division appears in loop index calculations, a nested loop structure is used to directly generate the index, completely eliminating division / modulo operations.

[0143] At step 606, in response to triggering the fourth decision rule, the following optimization strategies are output: reduce the number of synchronization points; narrow the synchronization range; and use asynchronous channels.

[0144] An EU Stall by Barrier exceeding 10% indicates that EUs spend more than 10% of their time waiting at synchronization barriers. The overhead of synchronization barriers comes from two aspects: one is the execution delay of the barrier itself; the other is the time difference between different warp / SPCs arriving at the barrier, meaning that the warp / SPC that arrives first must wait for the warp / SPC that arrives last, and the waiting time is completely wasted.

[0145] Based on this, the synchronization scope of the entire SPC (SYNC_SPC) is reduced to the synchronization scope of the (Compute Unit, CU) (SYNC_CU), reducing the number of threads that need to be coordinated. Tcore's asynchronous load / compute pipeline is used to implement fine-grained producer-consumer synchronization through semaphores, replacing coarse-grained barrier synchronization. Reducing the synchronization scope from SYNC_SPC to SYNC_CU can typically reduce barrier stand time by more than 50%.

[0146] Regarding the use of asynchronous channels, it can utilize Tcore's three independent asynchronous instruction streams (e.g., loadA, loadB, compute) and achieve precise synchronization of the producer-consumer pattern through TCI-P semaphores (e.g., wait_a_load, post_a_calc), avoiding coarse-grained waiting due to global barriers.

[0147] At step 608, in response to triggering the fifth decision rule, the following optimization strategy is output: check the vector width of the current data type; increase the vector width; adjust the loop step size and boundary handling logic accordingly; and ensure that the data arrangement in memory meets the burst transfer alignment requirements.

[0148] The vector width for the current data type can be bf162, bf164, bf168, etc. Vector widths of bf162, bf164, and bf168 correspond to reading 2, 4, and 8 bfloat16 elements per load operation, respectively.

[0149] Increasing the vector width, for example, to bf168 (corresponding to burst=4), allows the compiler to use longer burst transfer configurations when generating memory access instructions, ensuring that the access granularity of the WTI interface is aligned with burst mode. For example, increasing from burst=2 to burst=4 can theoretically reduce the number of memory access instructions by about 50%, resulting in a performance improvement of about 15%-25%.

[0150] In the above scheme, the present invention not only provides a complete mapping from PFC index to specific optimization strategies, but also provides specific code modification methods for each rule, making the optimization strategies directly operable.

[0151] The following will combine Figure 7 This invention describes a method 700 for outputting an optimization strategy in response to triggering a sixth decision rule, according to embodiments of the present invention. It should be understood that method 700 can, for example, be implemented in... Figure 2The described computing device 200 performs the operation. Method 700 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0152] At step 702, in response to triggering the sixth decision rule, the following optimization strategy is output: determine the L2 cache capacity; calculate the working set size of a single block under the current block partitioning strategy; in response to the working set exceeding the L2 cache capacity, reduce the block size until the working set fits the L2 cache capacity; and select the block dimension with the highest data reuse rate while satisfying the L2 cache capacity constraint.

[0153] For example, an L2 cache miss could be caused by an inappropriate kernel chunking strategy, where the amount of data processed in each chunk exceeds the effective capacity of the L2 cache, causing the data in the current chunk to evict the data in the previous chunk (cache thash). To address this, an optimization strategy can be adopted to reduce the chunk size, i.e., reduce the size of each chunk in each dimension, so that the working set of a single chunk perfectly fits the L2 cache capacity. In other embodiments, an optimization strategy can be adopted to adjust the chunk dimensions, i.e., select the dimension with the highest data reuse rate for partitioning, maximizing the number of times data in the L2 cache is reused. In other embodiments, an optimization strategy of Tensor Buffer preloading can also be adopted, i.e., when computing the current chunk, the data of the next chunk is preloaded into the L2 cache.

[0154] For example, taking 2D Softmax (shape [M, N]) as an example, if the tile size along the N dimension is too large, causing the input data to exceed the L2 capacity, the optimization strategy can reduce Tile_N from 1024 to 512 or 256, so that the input data of each tile (Tile_N × sizeof(float)) can be fully contained in L2. Subsequent passes can be read from L2 with high bandwidth (approximately 120.81 B / cycle per SPC). In this way, when the L2 hit rate is increased from 60% to 95%, for multi-pass algorithms (such as Softmax), the L2 hit read bandwidth (approximately 120.81 B / cycle per SPC) is 3.26 times that of the HBM miss bandwidth (approximately 37.06 B / cycle per SPC), and the read speed of subsequent passes can be improved by more than 3 times.

[0155] At step 704, in response to triggering the seventh decision rule, the following optimization strategies are output: modify the grid configuration at kernel startup to allow more stream processor cores to be used for computation; and increase the thread limit for each stream processor core by utilizing the startup boundary qualifier.

[0156] Regarding modifying the mesh configuration at kernel startup, the number of SPCs can be increased. It should be understood that when modifying the mesh configuration, consideration should be given to ensuring that the data volume is divisible (or nearly divisible) by the new number of SPCs, and that under a NUMA memory layout, the SPCs should access the corresponding local HBM partition.

[0157] Regarding launch boundary qualifiers, such as the `launch_bounds_` qualifier, they can be added to the kernel function declaration. In this invention, the constraint relationship between register usage per thread and the number of threads per SPC is as follows: when the number of threads per SPC is 512, a maximum of 256 registers can be allocated per thread bundle (32 threads); when the number of threads per SPC is 1024, a maximum of 128 registers can be allocated per thread bundle.

[0158] The above constraints indicate that increasing the number of threads per SPC increases the number of concurrently running thread bundles, and with the total number of register resources remaining constant, the average number of registers available per thread bundle decreases accordingly. Therefore, the feasibility of increasing the thread count limit depends on whether the current kernel function's per-thread register usage allows it to remain within the per-thread bundle register allocation limit even with a higher thread count configuration.

[0159] It should be understood that before performing optimizations to increase the thread count limit, the per-thread register usage of the current kernel function should be queried first. For example, the register usage of the current kernel can be queried using the `brcc --print-reg-count` command. Additionally, for scenarios requiring multiple launches of small kernels (such as layer-by-layer convolution), a persistent kernel can be used to handle all the work in a single launch, eliminating the overhead of multiple kernel launches.

[0160] Taking the Add operator as an example, the above optimization strategy can improve performance by approximately 15 times when scaling from 1 SPC to 16 SPC. Taking the DWC operator as an example, the above optimization strategy can reduce cycles by 41.6% when scaling from 512 threads to 1024 threads.

[0161] At step 706, in response to triggering the eighth decision rule, the following optimization strategies are output: identify the matrix multiplication or convolution calculation core in the kernel function; determine the block size of the tensor calculation core; configure the data cache loading strategy for the input left and right matrices; replace the operation of the vector calculation kernel with the interface call of the tensor calculation kernel; and configure a double-buffered ping-pong pipeline mechanism.

[0162] For example, Tcore is a dedicated computational unit designed specifically for matrix multiplication-accumulation (MMA) and convolution (Conv), and its matrix operation computing power is far superior to that of the general-purpose Vcore. By replacing the manual matrix multiplication loop in Vcore with the tci_p::__mma or tci_p::__conv interface calls, Tcore's matrix computation throughput is typically more than 10 times that of Vcore.

[0163] In some embodiments, the manual matrix multiplication loop in Vcore can be replaced with the `tci_p::__mma` or `tci_p::__conv` interface calls; based on the tile size limitations of Tcore (FP32: 32×32×8, BF16: 64×64×16), the loading strategy for the A / B buffer and the iteration scheme in the K direction are designed; utilizing the three independent asynchronous instruction streams of Tcore's load A / load B / compute, a double-buffered ping-pong pipeline mechanism is implemented. Thus, after migrating the Multihead MMA operator from Vcore to Tcore, the number of cycles is significantly reduced. The matrix computation throughput of Tcore is typically more than 10 times that of Vcore.

[0164] In the above solution, this invention systematically encodes the diagnostic logic used by senior performance engineers when analyzing PFC reports through multiple decision rules, covering six major bottleneck categories: memory-bound, pipeline bubbles, excessive synchronization, cache jitter, insufficient utilization, and lack of Tcore. Junior engineers or AI agents do not need in-depth knowledge of GPU architecture; they only need to execute a standardized process of "collecting PFC → extracting metrics → querying the decision table → obtaining optimization strategies" to obtain optimization suggestions equivalent to those of experts.

[0165] The various processes and procedures described above, such as methods 300 to 700, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 300 to 700 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 300 to 700 described above can be performed.

[0166] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0167] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0170] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. A method for optimizing operator performance, characterized in that, include: The target operator is grouped and collected using hardware performance counters, so that the collected counter data can be aggregated into performance counter description information. The performance counter description information is parsed in order to extract multiple performance indicators and classify the extracted performance indicators. The extracted performance index values ​​are compared with the corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results. as well as In response to triggering at least one decision rule, at least one optimization strategy, ordered by priority, is output, wherein the optimization strategy contains at least information for indicating operator code changes.

2. The method according to claim 1, characterized in that, The values ​​of the categorized performance indicators are compared with the corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results. The values ​​of the classified performance indicators are compared with the corresponding predetermined trigger thresholds in the optimization mapping decision table. The optimization mapping decision table includes multiple structured decision rules, each of which includes: the name of the hardware performance counter indicator, the trigger threshold, the bottleneck classification, and the recommended specific optimization strategy.

3. The method according to claim 2, characterized in that, The bottleneck classification indicates the performance bottleneck type corresponding to the decision rule, and the performance bottleneck types include memory-constrained, pipeline bubble, and excessive synchronization; and The recommended optimization strategies are used to distinguish the severity of bottlenecks based on different thresholds of the same hardware performance counter metric, and to recommend different levels of optimization strategies accordingly.

4. The method according to claim 1, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to determining that the same kernel function triggers multiple decision rules, the priority of the multiple decision rules is determined based on a predetermined priority order; and Based on the determined priorities, optimization strategies ordered by priorities are generated via mutual exclusion rules, combination rules, and / or iterative convergence strategies. These optimization strategies include at least information for indicating operator code changes.

5. The method according to claim 1, characterized in that, Parsing the performance counter description information to extract multiple performance metrics and classifying the extracted performance metrics includes: Parse the performance counter description information to extract multiple performance metrics from it; and The extracted performance metrics are divided into multiple metric types, each corresponding to a specific performance dimension. These multiple metric types include: overview metrics, utilization metrics, stall analysis metrics, memory metrics, cache metrics, and load balancing metrics.

6. The method according to claim 1, characterized in that, The decision rules include several of the following: The first decision rule used to optimize burst transmission patterns; The second decision rule is used to adjust the algorithm of the current operator; A third decision rule for eliminating execution bubbles in the pipeline; The fourth decision rule is used to reduce the overhead of synchronization operations; The fifth decision rule for increasing the configuration parameter values ​​for burst transmission; The sixth decision rule used to adapt chunks to the second-level cache; The seventh decision rule for improving the parallelism of task processing; as well as The eighth decision rule used to migrate computational tasks to tensor kernel execution.

7. The method according to claim 6, characterized in that, The extracted performance metrics are compared with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results. Obtain an indicator of whether the execution unit of the current kernel function of the graphics processor is stalled due to loading instructions; The index of the execution unit stalling due to loading instructions is compared with a preset first stall threshold and a preset second stall threshold, wherein the preset first stall threshold is greater than the preset second stall threshold. In response to determining that the index of the execution unit stalling due to loading instructions is greater than or equal to a preset first stall threshold, the current kernel function is determined to be a memory-limited type of the first severity, and the second decision rule is triggered; In response to determining that the index of the execution unit stalling due to loading instructions is less than a preset first stall threshold and greater than or equal to a preset second stall threshold, the current kernel function is determined to be of the second severity of memory-limited type, and the first decision rule is triggered.

8. The method according to claim 7, characterized in that, The process of comparing the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results also includes: In response to determining that the index indicating the execution unit's stagnation due to loading instructions is less than a preset second stagnation threshold, it is determined whether the index indicating the execution unit's stagnation due to a synchronization barrier is greater than or equal to a predetermined third stagnation threshold; and In response to determining whether the stall index of the execution unit caused by the synchronization barrier is greater than or equal to a predetermined third stall threshold, it is determined that there are too many or too wide ranges of synchronization operations in the kernel function, and the fourth decision rule is triggered.

9. The method according to claim 6, characterized in that, The process of comparing the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results also includes: If the number of instructions executed per cycle is less than a predetermined instruction count threshold and the percentage of no-operation instructions exceeds a predetermined percentage threshold, it is determined that there are multiple empty bubbles in the instruction pipeline of the execution unit, and the third decision rule is triggered.

10. The method according to claim 7, characterized in that, The process of comparing the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results also includes: The burst transmission utilization rate is compared with a predetermined utilization rate threshold to determine whether the fifth decision rule is triggered. The L2 cache hit rate metric is compared with a first predetermined hit threshold to determine whether the sixth decision rule is triggered. The execution unit utilization metric is compared with a second predetermined hit threshold to determine whether the seventh decision rule is triggered; and The tensor kernel utilization index is compared with a predetermined utilization threshold to determine whether the eighth decision rule has been triggered.

11. The method according to claim 7, characterized in that, The process of comparing the extracted performance metrics with corresponding predetermined trigger thresholds to determine whether to trigger at least one decision rule based on the comparison results also includes: In response to determining whether the current kernel function is a memory-constrained type of the first severity or the second severity, determine whether there are minor bottlenecks such as burst transfer underutilization or cache jitter, so as to generate decision rules when multiple bottlenecks coexist in the current kernel function.

12. The method according to claim 6, characterized in that, The predetermined priority order includes: The second decision rule and the first decision rule, triggered by the comparison of indicators related to the execution unit stalling due to loading instructions, are of the first priority. The fourth decision rule, triggered by the comparison of indicators related to the stagnation of the execution unit due to synchronization barriers, is of the second priority. The third decision rule, triggered by comparing the number of instructions executed per cycle with the percentage of no-operation instructions, is the third priority; and The fifth, sixth, seventh, and eighth decision rules, triggered by burst transmission utilization, L2 cache hit rate, execution unit utilization, and tensor kernel utilization respectively, are of the fourth priority.

13. The method according to claim 6, characterized in that, Based on the determined priorities, priority-ranked optimization strategies are generated through mutual exclusion rules, combination rules, and / or iterative convergence strategies, including: In response to different threshold ranges of the metric causing the same execution unit to stall due to loading instructions, triggering the first and second decision rules respectively, an optimization strategy corresponding to the second decision rule is output based on a mutual exclusion rule; and In response to the completion of each round of operator optimization, the target operator is regrouped and data is collected via hardware performance counters so that the triggering status of all decision rules can be re-evaluated based on the collected counter data.

14. The method according to claim 6, characterized in that, Based on the determined priorities, optimization strategies ordered by priority are generated through mutual exclusion rules, combination rules, and iterative convergence strategies, including: In response to determining that a combination of the following decision rules is triggered simultaneously, the optimization strategy corresponding to each decision rule in the combination of decision rules is output: First Decision Rule, Fifth Decision Rule; First Decision Rule, Sixth Decision Rule; and First decision rule, third decision rule.

15. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the second decision rule, the following optimization strategy is output: Reduce the number of data traversals; By fusing multiple adjacent element-wise operators into a single kernel function, the overhead of transferring intermediate kernel function results via high-bandwidth memory is eliminated; and The calculation method of first rearranging the image data into a matrix and then calling the general matrix multiplication is transformed into directly performing convolution operations.

16. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the third decision rule, the following optimization strategy is output: The loop is fully unrolled via compiler directives to eliminate loop control overhead; and Eliminate division and modulo calculations.

17. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the fourth decision rule, the following optimization strategy is output: Reduce the number of synchronization points; Narrowing the synchronization range; and Use asynchronous channels.

18. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the fifth decision rule, the following optimization strategy is output: Check the vector width of the current data type; Increase the vector width; Adjust the loop step size and boundary handling logic accordingly; and Ensure that the data is arranged in memory to meet the burst transfer alignment requirements.

19. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the sixth decision rule, the following optimization strategy is output: Determine the L2 cache size; Calculate the working set size of a single block under the current partitioning strategy; In response to the working set exceeding the L2 cache capacity, the block size is reduced until the working set fits the L2 cache capacity; and Under the premise of meeting the second-level cache capacity constraint, select the block dimension with the highest data reuse rate.

20. The method according to claim 6, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the seventh decision rule, the following optimization strategy is output: Modify the grid configuration at kernel startup to allocate more stream processor cores to computation; and Increase the maximum number of threads per stream processor core by using startup boundary qualifiers.

21. The method according to claim 1, characterized in that, In response to triggering at least one decision rule, the output includes at least one optimization strategy ordered by priority, including: In response to triggering the eighth decision rule, the following optimization strategy is output: Identify the matrix multiplication or convolution calculation core in the kernel function; Determine the block size of the tensor computation kernel; Configure the data caching and loading strategy for the input left and right matrices; Replace the operations of the vector computation kernel with calls to the tensor computation kernel interface; and Configure a double-buffered ping-pong production line mechanism.

22. A computing device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-21.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-21.

24. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-21.

Images