Performance evaluation method and device of deep learning model based on GPGPU architecture, electronic equipment and medium

Abstract

The present disclosure relates to a method and device for performance evaluation of a deep learning model based on a GPGPU architecture, an apparatus and a medium. The method comprises: obtaining a computation graph of a deep learning model; converting the computation graph into an intermediate representation, wherein the intermediate representation comprises operator parameter information of the operators and a dependency relationship between the computation nodes; obtaining hardware configuration parameters of a GPGPU used for deploying the deep learning model; decomposing each operator corresponding to a computation node into an instruction-level operator sequence according to the hardware configuration parameters of the GPGPU; for each computation node, obtaining a performance evaluation formula for each operator item in the instruction-level operator sequence corresponding to the computation node; determining a node performance indicator of the computation node according to the performance evaluation formula corresponding to each operator item; and determining an execution time and a performance bottleneck of the deep learning model based on the dependency relationship between the computation nodes and the node performance indicator of each computation node.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the field of artificial intelligence and other related technologies, and especially to a performance evaluation method, apparatus, electronic device and computer-readable storage medium for a deep learning model based on a GPGPU architecture. Background Technology

[0002] In recent years, the rapid development of deep learning models has led to their widespread application in various complex computing scenarios. Due to the massive number of parameters in deep learning models and the intensive tensor operations involved in the inference process, the industry typically uses high-performance computing hardware such as general-purpose GPUs (GPGPUs) to accelerate model deployment and inference.

[0003] In the microarchitecture of GPGPUs, in addition to the conventional Streaming Multiprocessor (SM) and core processing units, there is also a multi-level cache system and global memory. During inference, deep learning models, due to their complex network structures and varying memory access behaviors and computational logic among different operators, present significant challenges to their actual hardware performance. Therefore, accurately evaluating the performance of deep learning models on a specific GPGPU architecture is a crucial aspect of model deployment and optimization. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and computer-readable storage medium for evaluating the performance of deep learning models based on GPGPU architecture.

[0005] According to one aspect of this disclosure, a performance evaluation method for a deep learning model based on a GPGPU architecture is provided, comprising: acquiring a computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; converting the computation graph into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the multiple computation nodes and dependencies between computation nodes, wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; acquiring hardware configuration parameters of the GPGPU used to deploy the deep learning model; and decomposing the operators corresponding to each computation node in the intermediate representation into hardware instruction-level operators based on the hardware configuration parameters of the GPGPU. The subsequence, wherein the instruction-level operator sequence includes at least one operator item from primitive operators and composite operators, and wherein the composite operator is generated by combining the primitive operators according to a preset pattern; for each computing node, for each operator item in the instruction-level operator sequence corresponding to that computing node, a pre-set performance evaluation formula corresponding to that operator item is obtained, wherein the performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation caused by the operator item during actual execution; based on the performance evaluation formula corresponding to each operator item, the node performance index of that computing node is determined; and based on the dependencies between the computing nodes and the node performance index of each computing node, the execution time and performance bottleneck of the deep learning model are determined.

[0006] According to another aspect of this disclosure, a performance evaluation device for a deep learning model based on a GPGPU architecture is provided, comprising: a first acquisition module configured to acquire a computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; a conversion module configured to convert the computation graph into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the multiple computation nodes and dependencies between computation nodes, wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; a second acquisition module configured to acquire hardware configuration parameters of the GPGPU used to deploy the deep learning model; and a decomposition module configured to decompose the operators corresponding to each computation node in the intermediate representation into hardware instruction level parameters according to the hardware configuration parameters of the GPGPU. The instruction-level operator sequence includes at least one operator item from primitive operators and composite operators, wherein the composite operator is generated by combining the primitive operators according to a preset pattern; a third acquisition module is configured to, for each computing node, acquire a pre-set performance evaluation formula corresponding to each operator item in the instruction-level operator sequence corresponding to that computing node, wherein the performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation caused by the operator item during actual execution; a first determination module is configured to determine the node performance index of the computing node according to the performance evaluation formula corresponding to each operator item; and a second determination module is configured to determine the execution time and performance bottleneck of the deep learning model based on the dependencies between the computing nodes and the node performance index of each computing node.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor, wherein each of the at least one processor includes: a processor cache; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0009] According to another aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0010] According to embodiments of this disclosure, a performance evaluation method for a deep learning model based on a GPGPU architecture is provided. This method involves acquiring the deep learning computation graph and converting it into an intermediate representation. Then, combining GPGPU hardware configuration parameters, high-level operators are refined and decomposed into a sequence of low-level instruction-level operators. Subsequently, a preset performance evaluation formula is called for each operator in the sequence to derive and calculate the performance metrics of the entire node, as well as the execution time and performance bottlenecks of the entire graph. Thus, through fine-grained decomposition at the instruction level, the evaluation process can more accurately reflect the mapping of the software algorithm to the underlying hardware resources. The application of a scale influence factor compensates for the discrepancy between theoretical calculations and actual operation, enhancing the realism and accuracy of execution time prediction.

[0011] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A flowchart illustrating a performance evaluation method for a deep learning model based on a GPGPU architecture according to an embodiment of the present disclosure is shown. Figure 2 A flowchart illustrating a portion of the process of a performance evaluation method for a deep learning model based on a GPGPU architecture according to an embodiment of the present disclosure is shown. Figure 3 An exemplary block diagram of a performance evaluation apparatus for a deep learning model based on a GPGPU architecture according to embodiments of the present disclosure is shown; and Figure 4 A block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0014] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0015] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0016] In related technologies, evaluating the performance of deep learning models on GPGPUs mainly relies on two methods: one is a pure hardware simulator based on precise clocks, which is time-consuming and cannot be used for rapid tuning; the other is static analysis based on the Roofline model, which cannot accurately capture the performance degradation caused by microarchitecture resource contention.

[0017] To address one or more of the problems existing in related technologies, this disclosure provides a performance evaluation method for deep learning models based on a GPGPU architecture. By acquiring the deep learning computation graph and converting it into an intermediate representation, and combining GPGPU hardware configuration parameters, high-level operators are refined and decomposed into a sequence of low-level instruction-level operators. Subsequently, for each operator item in the sequence, a preset performance evaluation formula is called to derive and calculate the performance indicators of the entire node, as well as the execution time and performance bottlenecks of the entire graph. Thus, through fine-grained decomposition at the instruction level, the evaluation process can more accurately reflect the mapping of the software algorithm to the underlying hardware resources. The application of a scale influence factor compensates for the deviation between theoretical calculations and actual operation, enhancing the realism and accuracy of execution time prediction.

[0018] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0019] Figure 1 A flowchart illustrating an exemplary process of a performance evaluation method 100 for a deep learning model based on a GPGPU architecture according to an embodiment of the present disclosure is shown.

[0020] like Figure 1 As shown, the performance evaluation method 100 for deep learning models based on GPGPU architecture includes: Step S101: Obtain the computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; Step S102: Convert the computation graph into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the plurality of computation nodes and the dependencies between computation nodes, wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; Step S103: Obtain the hardware configuration parameters of the GPGPU used to deploy the deep learning model; Step S104: According to the hardware configuration parameters of the GPGPU, decompose the operator corresponding to each computing node in the intermediate representation into a hardware instruction-level operator sequence, wherein the instruction-level operator sequence includes at least one operator item among primitive operators and composite operators, and wherein the composite operator is generated by combining the primitive operators according to a preset pattern. Step S105: For each computing node, for each operator item in the instruction-level operator sequence corresponding to the computing node, obtain a pre-set performance evaluation formula corresponding to the operator item, wherein the performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation generated by the operator item during actual operation. Step S106: Determine the node performance index of the computing node according to the performance evaluation formula corresponding to each operator item; and Step S107: Based on the dependencies between the computing nodes and the node performance metrics of each computing node, determine the execution time and performance bottleneck of the deep learning model.

[0021] In step S101, for example, a virtual tensor with a specific dimension can be input into the deep learning model to be evaluated, and forward propagation computation can be performed without triggering gradient updates. This records the call order and data flow relationship between various computation nodes within the model. Thus, through the execution of the actual computational logic, a computational graph reflecting the model's topological structure during the inference phase can be obtained. Since only forward propagation is involved, eliminating the differentiation and state preservation in backpropagation, the dynamic logical structure of the deep learning model when processing inputs of specific sizes can be captured with lower resource overhead, providing a topological basis for subsequent performance decomposition.

[0022] In step S102, the captured computation graph undergoes depth-first traversal and parsing, transforming the abstract computation nodes in the graph into intermediate representations containing specific attributes. For example, this intermediate representation records the operator type of each computation node and extracts attribute information of the input and output tensors, such as the shape dimension and data precision of the tensors. Thus, unstructured graphical data is transformed into a quantifiable task sequence, achieving initial decoupling between algorithm-side requirements and underlying hardware specifications. This provides an objective data foundation for accurately mapping the computational load of the software layer to the physical constraints of the hardware side.

[0023] In step S103, the physical specifications of the GPGPU used to deploy the deep learning model are read to obtain its hardware configuration parameters, thereby establishing the physical performance boundaries of the target hardware. By obtaining the hardware configuration parameters, the performance evaluation process can be anchored to real physical limitations, avoiding purely theoretical calculations detached from hardware architecture characteristics, and improving the reference value of the evaluation results for specific deployment environments.

[0024] In step S104, based on the acquired hardware configuration parameters, the operators originally at the algorithm level in the intermediate representation are decomposed into a sequence of hardware instructions. Within this sequence, according to a preset decomposition logic, operators are broken down into indivisible primitive operators (such as basic multiply-accumulate instructions or memory access instructions), or into composite operators composed of multiple primitive operators combined according to a specific execution mode. This reconstructs the true execution trajectory of the operators at the microarchitecture level. This instruction-level decomposition allows for a more detailed observation of the scheduling behavior of operators on underlying resources, providing a prerequisite for identifying performance-limiting factors at the micro level.

[0025] In step S105, a corresponding performance evaluation formula is matched for each decomposed operator item. These formulas not only consider the theoretical cycle of instruction execution but also specifically introduce a scale impact factor to quantify the performance degradation caused by data size, resource contention, or microarchitectural conflicts. Thus, a latency prediction model for the operator item in a real-world operating environment is established. The technical advantage of introducing the scale impact factor is that it effectively captures the microarchitectural losses ignored by the theoretical peak, ensuring that the evaluation results are not merely an idealized representation of computing power but accurately reflect the true execution efficiency of the operator at a specific scale.

[0026] In step S106, the execution metrics of each operator item under the current hardware configuration are calculated according to the performance evaluation formula of each operator item. By logically aggregating the metrics of each operator item in the sequence, the node performance metrics of each computing node (i.e., operator) are determined. This completes the upward aggregation from instruction-level metrics to operator-level metrics. Through this hierarchical calculation scheme, the actual time each computing node occupies hardware resources can be accurately quantified, and the execution logic relationships between different instruction types are taken into account, thus yielding a high-precision operator-level performance evaluation.

[0027] In step S107, based on the dependencies between nodes and the calculated node performance metrics, a time-series simulation of the complete execution flow of the deep learning model on the target GPGPU is performed. This determines the total execution time of the model and identifies key bottlenecks affecting performance, enabling the evaluation method 100 to clearly pinpoint which layers or operators in the model limit overall efficiency. This achieves accurate and global performance evaluation of the deep learning model, providing quantitative data support for subsequent possible operator replacements, parameter adjustments, or hardware selection, significantly shortening the iteration cycle required for model deployment and optimization.

[0028] As can be seen, through the execution of steps S101 to S107, the performance evaluation method 100 for deep learning models based on GPGPU architecture disclosed herein combines the high-level computational logic of the deep learning model with the low-level physical constraints of the target hardware without relying on a real physical testing environment. By decomposing instructions at the instruction level and introducing a scale influence factor to characterize microarchitecture decay, it effectively compensates for the deviation between conventional pure theoretical estimation and real hardware execution. It avoids the huge time and computing power overhead brought by low-level hardware cycle-level simulators and can objectively reflect the scheduling characteristics and resource competition status of operators in real runtime at the micro level. Finally, it outputs the overall execution time of the model with reference value and clarifies the key performance bottlenecks that limit execution efficiency, thus providing quantitative and direct data support for hardware selection, operator logic optimization, and overall computing resource allocation in the early stages of deep learning model deployment.

[0029] In some examples, to capture the dynamic information of a deep learning model (such as a large language model) during runtime, a virtual input tensor containing the target batch size and the target sequence length can be constructed and input into the model for forward propagation. In this process, by utilizing a dynamic tracking mechanism to capture the dynamic features of the tensor shape at each layer changing with the input sequence length, the dimensionality bias of traditional static graph analysis when dealing with variable-length sequences can be avoided.

[0030] In some examples, in step S102, the architectural parameters extracted from the computation graph cover the global design constraints of the model, specifically including the number of network layers (e.g., 32 layers), hidden layer dimensions (e.g., 4096), number of attention heads (e.g., 32), vocabulary size (e.g., 32000), maximum sequence length, and positional encoding method (e.g., Rotational Positional Encoding RoPE). Simultaneously, specific operator parameters can be extracted for different types of computation nodes. For example, for general matrix multiplication operators, the transpose attribute and data type of the matrix (e.g., FP16, FP32, or INT8) are extracted; for attention operators, mask configuration information such as causal masks is extracted; and for reduction operators such as Softmax, the dimension axis information of the computation is extracted. Furthermore, the extracted network topology can be organized into a hierarchical sequence according to the computation graph, with each layer containing a complete attribute description, such as input / output shapes and operator dependencies, thereby constructing a data foundation for layer-by-layer performance evaluation.

[0031] After the intermediate representation is constructed, a coarse-grained pre-screening process can be introduced. For each operator across multiple computing nodes, a coarse-grained Roofline preliminary analysis is performed, assigning corresponding operator classification labels based on the initial arithmetic strength of the operator. For example, labels indicating computationally intensive bottlenecks or memory-intensive bottlenecks can be used. This preliminary classification provides benchmark guidance for subsequent precise verification after the introduction of a block-based strategy, assisting in secondary verification of operator performance.

[0032] According to some embodiments, the hardware configuration parameters of the GPGPU obtained in step S103 include at least one of storage resource parameters, computing resource parameters, and instruction execution characteristic parameters. The storage resource parameters include at least one of shared memory capacity, register capacity, peak memory bandwidth, and cache size. The computing resource parameters include at least one of the number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency. The instruction execution characteristic parameters include at least one of the ideal number of cycles per instruction, instruction execution throughput, and instruction set mapping relationship.

[0033] In the above embodiments, by acquiring multi-dimensional underlying hardware configuration parameters, a realistic physical boundary is established for the evaluation of deep learning models. Specifically, the shared memory capacity and register capacity in the storage resource parameters determine the maximum data block size that the operator can use during execution, while the peak memory bandwidth and cache size are used to deduce the transmission time during data handling. The number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency in the computing resource parameters establish the upper limit of the computing power of the hardware in concurrent instruction processing under ideal conditions, and provide a benchmark for converting clock cycles into absolute time. The instruction execution characteristic parameters delve into the microarchitecture level, providing basic theoretical values ​​for calculating the specific hardware instruction execution cycle. Acquiring hardware configuration parameters allows the evaluation process to move beyond relying on coarse macroscopic indicators and instead establish high-precision underlying digital mappings for GPGPUs with different architectures, thereby improving the accuracy and applicability of cross-hardware platform evaluation.

[0034] After obtaining the detailed underlying hardware configuration parameters and establishing the physical boundaries, method 100 further maps and decomposes the high-level abstract operators to the underlying hardware instructions in order to restore the execution logic at the micro level.

[0035] Figure 2 A flowchart illustrating a portion of the process of a performance evaluation method for a deep learning model based on a GPGPU architecture according to an embodiment of the present disclosure is shown.

[0036] like Figure 2 As shown, step S104 includes: Step S201: For each computing node, based on the operator parameter information corresponding to the computing node, obtain the target algorithm for the operator of the computing node from a set of multiple candidate algorithms. Step S202: Based on the target algorithm and the hardware configuration parameters of the GPGPU, determine the data block parameters for parallel processing corresponding to the computing node; and Step S203: Based on the data block parameters, determine the number of times the operator items are called and the execution order required for the implementation of the target algorithm, so as to generate the instruction-level operator sequence corresponding to the computing node.

[0037] Because operators in high-level deep learning frameworks are often abstract mathematical expressions, there are multiple different algorithmic implementation paths in the underlying hardware. Therefore, by extracting specific parameter information of the operators, such as the dimensionality features of tensors, the optimal underlying implementation strategy is matched from a pre-defined candidate algorithm library. For example, when a computation node is identified as a convolution operator, it can be mapped to an implicit general matrix multiplication algorithm as the target algorithm based on the shape features of its input and output tensors. This process bridges the gap between high-level semantics and low-level execution, ensuring that the evaluation logic is consistent with the algorithm selection behavior of a real compiler.

[0038] In step S202, the data partitioning strategy for parallel computing is derived by combining the determined target algorithm with the resource constraints of the target hardware. Since the high-speed on-chip storage resources of the GPGPU are limited and cannot accommodate the entire tensor at once, in some examples, large-scale matrices need to be divided into smaller blocks for iterative processing.

[0039] As a concrete example, when the target algorithm is an implicit general matrix multiplication algorithm that handles matrix multiplication and addition operations, based on the obtained shared memory capacity limit of a single streaming multiprocessor and the byte size of the data type, the data block size allocated to a single thread block in each operation dimension is derived and determined. For example, the first block parameter in the output spatial location dimension can be set to 128, the second block parameter in the output channel dimension can be set to 128, and the third block parameter in the input channel dimension can be set to 32. Through this block computation under physical constraints, not only is the memory partitioning behavior of the operator reflected in the real hardware scheduling, but the reduction in memory access overhead due to the data reuse mechanism can also be effectively quantified.

[0040] In step S203, based on the previously determined data block parameters, the algorithm is further expanded into a sequence of instructions at the microarchitecture level. Once the size of the data block processed in a single operation is determined, the specific number of times the underlying memory access primitive operators and computation primitive operators need to be called repeatedly to traverse the complete problem scale, as well as the execution order of these operators on the timeline, can be calculated. This step transforms the block strategy into a quantifiable set of micro-instructions, reducing the abstract operator nodes to the instruction level, providing a refined derivation framework for subsequently calling performance evaluation formulas line by line and applying scale influence factors.

[0041] As can be seen, by executing steps S201 to S203, the method provided in this disclosure transforms abstract mathematical computation nodes into a sequence of execution actions that conforms to the characteristics of the underlying hardware microarchitecture. This decomposition mechanism from high-level operators to low-level instructions allows the performance evaluation process to move beyond coarse-grained theoretical computing power estimation and deeply reflect the actual scheduling of physical resources. Specifically, based on the data partitioning strategy and execution order derived from the target algorithm and hardware configuration, the actual movement trajectory of tensor data between multi-level storage structures and the cyclic processing logic of core computing units are reconstructed. This decomposition process establishes a fine-grained derivation framework for subsequent evaluation formulas that include scale influence factors, enabling the final performance indicators to effectively reflect the specific limiting effect of underlying microarchitecture differences on the execution efficiency of deep learning models.

[0042] To illustrate the operator decomposition process more clearly, a detailed specific embodiment is provided below.

[0043] Suppose the computation node being processed is a linear mapping operator in the feedforward neural network layer of a deep learning model. In step S201, based on the two-dimensional matrix multiplication characteristics corresponding to the linear mapping operator and the feature dimension parameters in the tensor properties, a block matrix multiplication algorithm is extracted and matched from the candidate algorithm library as the target algorithm for this computation node.

[0044] In step S202, based on the computational data flow characteristics of the target algorithm and the obtained GPGPU hardware configuration parameters, data block parameters adapted to the current physical environment are derived. During this process, the evaluation method calculates the total shared memory capacity and register file quota of the multiprocessor. Assuming that resource boundary constraint calculations determine that the operator allocates a block size of 128 along the output row dimension, a block size of 128 along the output column dimension, and a block size of 32 along the inner accumulation dimension during parallel processing, this quantized block parameter ensures that the split data blocks reside precisely in the high-speed on-chip memory without overflow.

[0045] In step S203, based on the determined dimensional block parameters, the specific processing logic of the target algorithm in the hardware execution pipeline is deduced, thereby generating the instruction-level operator sequence corresponding to the computing node. A loop is executed for each segmented data block, and the sequence is broken down and arranged into multiple specific instruction-level actions. First, a first memory access primitive operator is generated to move the input feature data block from global memory to shared memory; second, a second memory access primitive operator is generated to synchronously read the weight data block; subsequently, a computation primitive operator is generated to call the tensor computation core to perform intensive multiply-accumulate operations; finally, a third memory access primitive operator is generated to write the computation result back to the target memory after the accumulation loop ends.

[0046] Furthermore, if the linear mapping node in the computation graph and its subsequent nonlinear activation node satisfy a preset fusion condition, the decomposition process can merge the write-back operator and activation operator at the end of the above operator sequence according to a preset fixed pattern, thereby generating a composite operator term that integrates multiply-accumulate and activation operations in the final instruction-level operator sequence. After the above steps, the linear mapping node is transformed into an instruction sequence consisting of alternating underlying actions such as data loading, core operations, and result write-back, providing a substantial derivation basis for subsequent accurate evaluation of computational pipeline stalls and data transfer delays.

[0047] After completing the decomposition from macroscopic abstract computing nodes to the underlying microscopic instruction-level operator sequences, the next step is to quantify the time consumption and performance of these discrete low-level execution actions. To achieve this goal, the method disclosed in this paper introduces a pre-built classification performance evaluation library, which provides support for obtaining the formulas and parameters required for quantization calculations.

[0048] According to some embodiments, the performance evaluation formula corresponding to the operator item is pre-stored in a preset operator performance library, wherein the operator performance library includes: a primitive operator library for storing the primitive operators and their corresponding primitive performance evaluation formulas; and a composite operator library for storing the composite operators and their corresponding fusion performance evaluation formulas.

[0049] In the specific architecture implementation, the primitive operator library serves as the cornerstone of the underlying evaluation. It stores hardware instruction-level operators that cannot be further decomposed, such as floating-point operators, memory access operators, and thread synchronization operators, as well as their corresponding basic mathematical evaluation models. These models serve as basic performance evaluation formulas to quantify the execution time of a single underlying instruction.

[0050] The composite operator library is geared towards execution optimization scenarios, storing high-frequency operator templates composed of the primitive operators combined in a preset fixed pattern, along with their corresponding fusion performance evaluation formulas. In the actual inference of deep learning models, there are numerous frequently invoked operator combination sequences. To accurately quantify the actual performance of these combinations after underlying compilation optimization, the composite operator library covers a variety of typical fusion operator templates. Specifically, the composite operator templates include, but are not limited to: fusion templates for convolution computation, such as the fusion of convolution, bias, and activation functions (ConvBiasReLU), or the fusion of convolution and addition (Conv+add); fusion templates for matrix operations, such as the fusion of general matrix multiplication and different activation functions (GEMM+RELU, GEMM+Silu); composite fusion templates for attention mechanisms, such as the fusion execution template consisting of a query matrix, a key matrix, a value matrix, and a Softmax operation (Softmax(Q·K^T)·V); and fusion templates for basic arithmetic and logical operations, such as the fusion of shift and addition (shift+add), the fusion of addition and activation functions (add+relu), and the fusion of multiplication and addition (mul+add).

[0051] Each composite operator template in the composite operator library is associated with a fused analytical performance model. Since the fused composite operators typically avoid repeated reads and writes of intermediate results to global memory during low-level physical execution, data can be reused locally at the on-chip cache or even register level. Therefore, the derivation of this fused performance evaluation formula fully considers the benefits of register reuse and instruction pipeline overlap, ensuring that the evaluation results objectively reflect the actual hardware latency after the compiler implements operator fusion optimization.

[0052] After establishing the two-tier performance library architecture, the evaluation method needs to establish a dynamic filtering mechanism to determine which level of evaluation formula should be called for a specific operator item in the sequence.

[0053] According to some embodiments, step S105 includes: for each computing node, performing pattern matching between the instruction-level operator sequence corresponding to the computing node and the composite operators in the composite operator library; in response to at least a portion of the continuous primitive operators in the instruction-level operator sequence successfully matching with a target composite operator in the composite operator library, taking the target composite operator as an operator item of the instruction-level operator sequence, and obtaining a fusion performance evaluation formula corresponding to the target composite operator from the composite operator library; and in response to the instruction-level operator sequence including unmatched primitive operators, obtaining a primitive performance evaluation formula corresponding to the unmatched primitive operators from the primitive operator library.

[0054] During the specific operational phase of formula acquisition, the evaluation process traverses the generated instruction-level operator sequence. Mechanisms such as sliding window or subgraph topology recognition can be used to check for instruction sequence fragments that meet optimization conditions. As a concrete example, when the generated instruction sequence contains convolution multiplication-addition, bias addition, and nonlinear activation primitive operators in sequence, the matching mechanism identifies that this continuous segment matches the pre-defined convolution-bias activation fusion template features in the composite operator library. In this case, the evaluation process no longer isolates and accumulates the memory access and computational overhead of these three primitive operators; instead, it extracts them as a whole composite operator term and calls the corresponding fusion performance evaluation formula for overall time consumption calculation, thus eliminating the time consumption of multiple intermediate result write-backs and read-backs that would otherwise occur due to independent execution. For independent primitive operators in the sequence that fail to form a pre-defined template due to specific data dependencies or stride mismatches, their primitive operator form is retained, and the basic evaluation formula is requested from the primitive operator library for calculation.

[0055] This pattern-matching-based dynamic formula acquisition scheme recreates the underlying graph optimization and operator fusion process of an AI compiler. By dynamically identifying consecutive instruction fragments and applying the overall evaluation formula, it avoids the repeated calculation of redundant memory read / write overhead within the fusion operator. This allows the evaluation model to accurately quantify the actual performance gains brought by the compiler optimization strategy, ensuring that the prediction results still possess high accuracy and objectivity when facing complex networks that have undergone underlying optimization.

[0056] After determining the composite operator terms and their fusion evaluation methods through the aforementioned pattern matching mechanism, for independent primitive operators that cannot be merged in the instruction-level operator sequence, the corresponding primitive performance evaluation formula needs to be called separately for their underlying execution characteristics. In the GPGPU microarchitecture, the underlying native operations can be mainly divided into computational behaviors responsible for algebraic operations and memory access behaviors responsible for data flow. Therefore, the primitive operator library disclosed herein constructs fine-grained performance evaluation logic for these two core behaviors.

[0057] According to some embodiments, the primitive operator includes computational primitive operators, and wherein the performance metrics of the computational primitive operators include computation time T. comp The primitive performance evaluation formula corresponding to the computational primitive operator is: T comp = N inst x Cycle inst x (1 + λ comp ), where N inst Cycle represents the number of times the instruction corresponding to the primitive operator of this computation class is executed. inst λ is the ideal cycle number for a single instruction. compThis is the scale influence factor for computational primitive operators.

[0058] For computational primitive operators such as matrix multiplication and addition, and tensor logic operations, the evaluation method delves into the microscopic level of the hardware instruction cycle to deduce the time consumption. Specifically, the derivation involves the number of instruction executions, N. inst Based on the data block parameters and original tensor dimensions determined in the preceding steps, this represents the total number of underlying hardware instructions that the streaming multiprocessor actually needs to issue to complete the computation task of this block size. The ideal number of cycles per instruction (Cycle) inst This refers to the inherent physical properties of the target GPGPU hardware architecture, such as the fixed hardware clock cycles required for specific tensor core multiply-accumulate instructions in a non-blocking state. Furthermore, in actual hardware operation, thread scheduling is often affected by microarchitectural bottlenecks such as register limitations, data dependencies between instructions, and pipeline pauses, resulting in actual execution times exceeding the sum of ideal cycles. Therefore, this formula introduces the scale influence factor λ of computational primitive operators. comp This is used to quantify the aforementioned performance degradation. As a concrete example, when processing large matrix multiplications with large block sizes, a large number of concurrent computation instructions may lead to insufficient registers allocated to each thread, causing register overflows and subsequent data access to slower local memory. In this case, the evaluation logic assigns a corresponding scale impact factor λ based on the register pressure state. comp This is to extend and correct the ideal computation time for pure theory.

[0059] This computational scheme, based on instruction-level physical cycles and incorporating microarchitecture decay factors, allows for a shift in computation time estimation beyond the traditional, coarse framework that relies solely on theoretical peak computing power. This enables the capture of the actual computational efficiency degradation of operators under specific hardware resource constraints, thus ensuring high fidelity and quantization accuracy in performance evaluation of computationally intensive tasks.

[0060] According to some embodiments, the primitive operator further includes a memory access primitive operator, wherein the performance metric of the memory access primitive operator includes memory access time T. mem The primitive performance evaluation formula corresponding to the memory access primitive operator is determined based on the actual amount of data transferred, the peak memory bandwidth of the GPGPU, and the scale influence factor of the memory access primitive operator.

[0061] In contrast to computational operators, for memory access primitive operators responsible for data transfer, such as global memory loading and shared memory writing, the evaluation method focuses on examining the physical limitations at the data throughput level. During the evaluation derivation, the actual amount of data moved is determined by the size of the operator's input and output tensors and the data reuse rate under a specific block partitioning strategy. Ideally, memory access time is a simple ratio of the amount of data moved to the hardware's peak memory bandwidth. However, real physical data transfer heavily depends on the alignment of memory requests and concurrent access patterns. Therefore, the formula introduces a scale factor for memory access primitive operators to reflect the degradation in memory access efficiency. As a concrete example, when executing certain operators with specific memory access step sizes (e.g., non-contiguous memory transpose operations), multiple execution threads accessing the same shared memory storage simultaneously can cause memory access conflicts, forcing memory transactions that could have been parallelized to be serialized by the hardware; or, when reading across different memory blocks, global memory merging is not achieved. In these scenarios, the evaluation logic will determine the appropriate scale impact factor based on the physical severity of the conflict to reflect the microarchitecture attenuation phenomenon where the actual effective transmission bandwidth is far lower than the peak bandwidth.

[0062] By combining the actual amount of physical data movement with the scale-based impact factor characterizing microarchitecture memory access overhead, the evaluation system for memory-constrained operators has been improved. This allows the evaluation process to sensitively identify non-ideal latency masked in theoretical bandwidth calculations, providing compilers with clear and objective quantitative references for optimizing data storage layout and adjusting memory access steps at the underlying level.

[0063] After establishing the basic time consumption evaluation models for various independent operator items, the evaluation process needs to further integrate these discrete operator metrics to obtain the complete execution latency of the computing node. In modern GPGPU microarchitectures, to maximize resource utilization and mask latency, hardware scheduling logic typically supports asynchronous concurrent execution of processing instructions from the computing core and memory access instructions from the memory controller. To reflect this dynamic physical characteristic without running real physical hardware, the evaluation method disclosed herein introduces computational logic with concurrent overlapping execution pipelines.

[0064] According to some embodiments, step S106 includes: calculating the performance index of each operator item according to the performance evaluation formula corresponding to each operator item; in response to determining that the instruction-level operator sequence corresponding to the computing node includes the composite operator, determining the total time consumption of the composite operator based on the performance index of the composite operator; in response to determining that the instruction-level operator sequence corresponding to the computing node includes both the computational primitive operator and the memory access primitive operator, aggregating the performance indexes of all computational primitive operators into a total computation time, and aggregating the performance indexes of all memory access primitive operators into a total memory access time; determining the larger of the total computation time and the total memory access time as the primitive operator overlap time; and determining the node performance index of the computing node based on the total time consumption of the composite operator and the primitive operator overlap time.

[0065] In the above embodiments, the evaluation process performs splitting and parallel processing on the overall time consumption of the computing node. Specifically, for the composite operators determined by pattern matching in the sequence, since they have been compiled into highly integrated execution modules at the underlying level, their internal register-level overlap benefits have been deducted in the corresponding fusion performance evaluation formula. Therefore, the evaluation process first independently extracts and calculates the performance indicators of these composite operators, summing them into the total time consumption of the composite operators. At the same time, for the remaining discrete primitive operators in the sequence, the evaluation process performs similar term aggregation on the time consumption of computational primitive operators responsible for logical operations and the time consumption of memory access primitive operators responsible for data movement. Under the hardware's double-buffered execution mechanism, when the computing core is processing the current data block, the memory controller is often asynchronously loading the next data block. Therefore, the evaluation process does not use simple linear time accumulation, but takes the larger value between the total computing time and the total memory access time as the overlap time consumption of the primitive operator. Finally, the total time consumption of the composite operator is added to the overlap time consumption of the primitive operator to establish the node performance indicator of the computing node.

[0066] As a concrete example, suppose a complex computation node is decomposed into a sequence of operators containing convolution computation and specific boundary processing. The boundary processing part is successfully matched as a composite operator, and its total computation time, calculated using an evaluation formula, is ten microseconds. The core convolution loop is decomposed into multiple independent computational primitive operators and memory access primitive operators. Aggregate calculations show that the total computation time for all computational primitive operators is one hundred microseconds, and the total memory access time for all memory access primitive operators is one hundred and twenty microseconds. Based on the aforementioned overlap logic, the larger of these two values, one hundred and twenty microseconds, is taken as the primitive operator overlap time. Finally, combining the ten-microsecond total computation time of the composite operator with the one hundred and twenty-microsecond overlap time of the primitive operator, the node performance index for this computation node is determined to be one hundred and thirty microseconds.

[0067] This computational scheme, which comprehensively considers the overlap between composite templates and primitives, avoids the severe overestimation caused by simply linearly adding the time of all instructions. It not only takes into account the benefits of operator fusion at the compiler level, but also accurately reproduces the actual execution mechanism in the microarchitecture that uses pipelines to mask latency, so that the node latency of the final output has a very high microscopic physical basis.

[0068] In other embodiments, when the instruction-level operator sequence corresponding to the computing node does not contain any composite operators, that is, when the computing node is completely decomposed into the underlying computational primitive operators and memory access primitive operators, the method for determining the node performance index of the computing node includes: aggregating the performance indexes of all computational primitive operators into total computation time, and aggregating the performance indexes of all memory access primitive operators into total memory access time; and directly determining the larger of the total computation time and the total memory access time as the node performance index of the computing node.

[0069] In this embodiment, the evaluation process handles the concurrent overlap of pure primitive operator sequences. Taking a conventional convolution operator in a certain layer of a deep learning model as an example, after decomposition in step S104, the convolution operator is transformed into a set of pure primitive operators including data loading, tensor multiplication-accumulation synchronous operations, and result storage. During the computation phase, the evaluation process sums up the time consumption of all tensor multiplication-accumulation operation instructions to obtain the total computation time of the convolution operator, assumed to be 100 microseconds; simultaneously, it sums up the time consumption of all instructions responsible for data transfer from global memory to shared memory to obtain the total memory access time of the convolution operator, assumed to be 130 microseconds. In actual underlying physical execution, due to the intervention of the double buffering mechanism, the shorter 100 microseconds of pure computation time is completely hidden within the longer 130 microseconds of data transfer time. Therefore, the evaluation process takes 130 microseconds as the final node performance index of the convolution operator.

[0070] Through the above example of overlapping computation for pure primitive operator sequences, the evaluation process can directly and quantitatively reveal the root cause limiting the execution efficiency of the operator. In this example, since the total memory access time is greater than the total computation time, it not only accurately calculates the actual latency of the node, but also clearly indicates from a numerical relationship that the convolutional layer is currently operating under conditions limited by memory access bandwidth. This ensures that the evaluation mechanism can maintain a high degree of accuracy in capturing underlying hardware bottlenecks even in basic execution scenarios devoid of compiler optimizations.

[0071] To enable the evaluation process to have self-calibration capabilities to adapt to changes in the underlying environment, the method disclosed in this paper further introduces a closed-loop feedback mechanism based on real hardware data.

[0072] According to some embodiments, the performance evaluation method 100 for deep learning models based on GPGPU architecture further includes: running a target operator term on the GPGPU and obtaining the measured latency T of the target operator term running on the GPGPU. real ; Obtain the theoretical reference time T of the target operator under ideal conditions. base Calculate the actual scale impact factor λ using the following formula. real :λ real = T real / T base - 1; Calculate the actual scale impact factor λ. real The deviation value between the target operator item and the pre-stored scale influence factor; and in response to the deviation value being greater than or equal to a threshold, based on the actual scale influence factor λ. real Update the stored scale impact factors.

[0073] In this closed-loop calibration embodiment, the evaluation process executes specific target operator terms individually on the target GPGPU in the form of a lightweight test to obtain the actual execution time on the physical device as the measured latency T. real Furthermore, based on the number of basic instruction executions and the ideal clock cycle of this operator, its theoretical baseline time T, unconstrained by microarchitectural resource contention, is derived. base By applying the above formula, the evaluation process normalizes the additional latency overhead exceeding the ideal state in the measured data, thereby calculating the actual scale impact factor λ, which reflects the actual microarchitecture loss of the current hardware. real Subsequently, the evaluation process compares the calculated actual factors with the old scale influence factors pre-stored in the current operator performance library. If the calculated deviation exceeds the set tolerance threshold, it indicates that the original preset parameters can no longer accurately represent the current underlying hardware state. In this case, the evaluation process will trigger a library update, overwriting or correcting the extracted actual scale influence factors into the corresponding entries in the performance library.

[0074] As a concrete example, suppose the evaluation process performs closed-loop calibration on a convolution operator for a specific block dimension. The theoretical baseline time T for this convolution operator under ideal conditions is derived through formula derivation. base The latency is 100.0 milliseconds, while the scale impact factor for this size is 0.1 pre-stored in the current operator performance library. After running this operator on real GPGPU hardware, the measured latency T was captured. real The value is 130.0 milliseconds. Substituting the above data into the formula, the actual scale impact factor λ is calculated. realThe value is 0.3, which is 130.0 divided by 100.0 and then subtracted by 1. After comparative analysis, the deviation between the actual extracted value of 0.3 and the pre-stored value of 0.1 reached a significant level, exceeding the tolerance threshold for fluctuations in conventional measurements. This indicates that at this specific scale, the latency caused by actual hardware interference such as access conflicts to the underlying shared memory is much higher than the original static estimate. Based on this judgment, the evaluation process updates the scale influence factor parameter of this convolution operator in the operator library to 0.3 for the current scale. When encountering computational tasks of the same scale in the future, the evaluation method will call the updated factor for more accurate calculations.

[0075] This implementation scheme, based on closed-loop calibration using measured data, effectively combines static theoretical analysis and derivation with dynamic physical measurement feedback. On the one hand, it can autonomously correct empirical biases that may exist in the initial theoretical modeling stage; on the other hand, it endows the performance evaluation method with continuous learning characteristics. By periodically or as needed incorporating small amounts of real hardware feedback data, the evaluation process can adaptively compensate for implicit performance drift caused by underlying hardware aging or firmware environment changes, thereby ensuring the robustness and reliability of the performance evaluation results in long-term engineering practice.

[0076] In summary, the method disclosed herein establishes an end-to-end performance evaluation closed loop from macroscopic algorithm architecture to underlying microarchitecture execution. By combining intermediate representations with hardware parameters, it achieves precise dimensionality reduction from operators to underlying instruction-level actions; through a two-layer operator performance library and pattern matching mechanism, it objectively quantifies the actual benefits of compiler fusion optimization; through concurrent overlap derivation of computation and memory access, it reconstructs the physical phenomenon of latency masking by the double buffering mechanism in the microarchitecture; and supplemented by a dynamic calibration mechanism based on measured data, it enables the evaluation model to adaptively evolve with the hardware environment. This scheme effectively bridges the technical gap between the insufficient accuracy of traditional static analysis and the excessive time consumption of underlying hardware simulation, providing a high-fidelity, interpretable, and self-correcting quantitative evaluation system for the efficient deployment of deep learning models on GPGPUs, underlying operator tuning, and computational resource allocation.

[0077] To further clarify the complete execution process of the performance evaluation method disclosed herein, a complete performance evaluation and calibration embodiment comprising five execution stages is provided below, in conjunction with a specific implementation scenario.

[0078] In the first stage, model parsing and computational graph capture are performed. The trained deep learning model is acquired, and its forward propagation logic is traversed using hook mechanisms or computational graph capture techniques to generate the underlying intermediate representation. Subsequently, the model is exported as a computational graph containing basic tensor operations. A specific example of the intermediate representation (IR) is as follows: Node 1: OpType=CONV2D, InputShape=[1, 64, 56, 56], WeightShape=[64,64, 1, 1], Stride=1, Padding=Same Node 2: OpType=BATCH_NORM, InputShape=[1, 64, 56, 56] Node 3: OpType=RELU, InputShape=[1, 64, 56, 56] As can be seen, IR includes a first node, a second node, and a third node. The first node has a 2D convolution CONV2D operator, with an input tensor shape of [1, 64, 56, 56] and a weight tensor shape of [64, 64, 1, 1], a stride of 1, and the same padding. The second node has a BATCH_NORM operator, with an input tensor shape of [1, 64, 56, 56]. The third node has a nonlinear activation RELU operator, with an input tensor shape of [1, 64, 56, 56].

[0079] In the second stage, operator decomposition and hardware instruction mapping are performed. After receiving the aforementioned two-dimensional convolution node, the evaluation process determines, based on its parameter characteristics, that the 1x1 convolution is most suitable for implementation using the ImplicitGemm algorithm, without needing to perform an explicit image-to-matrix transformation. Specifically, the parsed attributes of the convolution node include: input tensor [1, 256, 28, 28] with data type half-precision floating-point; weight tensor [512, 256, 1,1] with data type half-precision floating-point; and output tensor [1, 512, 28, 28].

[0080] The evaluation process further decomposes the algorithm into a sequence of primitive operators executed on the target GPGPU hardware. First, the convolution operation is reconstructed as matrix multiplication, where the output matrix equals the product of the input matrix and the weight matrix. Then, based on the resource constraints of the target architecture, data block parameters are selected, setting the thread block size to 256 threads, with corresponding block dimension parameters of 128, 128, and 256. This means each thread block is responsible for processing 128 elements in the output space, 128 output channels, and 256 input channels. Under this block strategy, a sequence of underlying primitive operators is generated, specifically including: First, global memory access primitive operators. For example, calling tensor memory access instructions, based on the tensor layout descriptor, shared memory base address, and specified feature map coordinates, moves input matrix blocks and weight matrix blocks of a specific size from global memory to shared memory. Second, local loading primitive operators (such as the LDS.128 instruction). These are responsible for further moving data blocks from shared memory to registers. For example, loading a 128×8 block of the input matrix from shared memory involves loading 128 bytes per instruction (corresponding to, for example, a 128×8 local data block). To complete the reading of the entire 128×256 data block, this instruction will be called multiple times in the execution sequence (e.g., 512 times). Similarly, the reading of the weight matrix block is completed through multiple corresponding calls. Third, tensor core computation instructions (such as the HMMA.161616 instruction). These are responsible for performing half-precision matrix multiplication and addition operations in a specific dimension (e.g., 16×16×16) and accumulating the results in a partial sum buffer. To complete the full block operation in the 128×128×256 dimension, this instruction will be called repeatedly in the sequence. This computation process is executed cyclically along the channel dimension until completion, and finally, a write-back instruction is called to store the final result back to global memory.

[0081] In the third stage, the primitive operator library is invoked for analytical computation. The evaluation process involves retrieving the corresponding performance model for each primitive operator in the above sequence. Taking the tensor core computation instruction (such as HMMA.161616) as an example, the corresponding computation time formula is obtained: T comp = N inst x Cycle inst x (1 + λ comp In the derivation process, considering the number of floating-point operations and instruction throughput (e.g., 2 instructions per cycle) of a single instruction processed by the tensor core under a specific hardware configuration, as well as the total number of computational operations required for a block size of 128×128×256, the number of executions N of the computation instruction is calculated. inst Simultaneously, based on the currently set block parameter dimensions, the corresponding scale influence factor λ is obtained by matching from a pre-initialized lookup table. compSpecifically, the dimensional parameters used for lookup matching consist of two sets of core parameters reflecting computational load and scheduling granularity: macroscopic parameters M, N, and K, and microscopic parameters BM, BN, and BK. The three parameters M, N, and K characterize the overall tensor size of the current computing node at the algorithm level. M typically corresponds to the number of rows in the output feature (e.g., the product of batch size and spatial dimension), N corresponds to the number of columns in the output feature (e.g., the number of output channels), and K corresponds to the depth of internal accumulation (e.g., the number of input channels). The reason for introducing macroscopic dimensions in the lookup table is that the total problem size determines the total number of concurrently scheduled thread blocks (Grid Size) on the streaming multiprocessor. When the concurrency is extremely high, it can cause queuing congestion in the global memory controller and contention for the L2 cache; when the concurrency is extremely low (e.g., tail computation), it can lead to the streaming multiprocessor's hardware wavefront not being fully filled, resulting in idle computing power. These macroscopic attenuation characteristics are strongly correlated with the specific values ​​of M, N, and K. The three parameters BM, BN, and BK characterize the exact size of a single hardware block scheduling operation at the underlying level. BM and BN represent the output matrix tile size cached in shared memory for a single thread block, and BK represents the depth of a single inner product iteration. The reason for introducing micro-dimensions into the lookup table is that the block size directly determines the resource utilization within a single streaming multiprocessor. For example, extremely large BM and BN can cause the number of registers allocated to a single thread to approach the physical limit, thus increasing the risk of register overflow; while extremely large BK will consume excessive shared memory capacity, limiting the number of thread blocks that can reside simultaneously on the same streaming multiprocessor. The lookup table stores the corresponding scalar values, i.e., the scale impact factor, at the coordinate points determined by the above six dimension combinations. This factor is a normalized relative deviation coefficient, representing the percentage increase in latency between the actual hardware execution cycle and the purely theoretical instruction throughput cycle when a specific macro-problem scale (M, N, K) and a specific micro-blocking strategy (BM, BN, BK) are adopted. Before deploying the evaluation system, real hardware performance counter data is captured by running test cases covering various typical (M, N, K) and (BM, BN, BK) combinations on the target GPGPU. The measured clock cycle is compared with the theoretical clock cycle, and the deviation coefficients for each dimension are calculated and entered into the table. For intermediate dimension combinations that are not directly matched, the lookup table is usually configured with linear interpolation or nearest neighbor matching query logic.

[0082] In one example, when the size impact factor is matched to 0.15, it indicates that the actual number of execution cycles will increase by 15% compared to the theoretical value due to increased register pressure or the pipeline not being fully filled. Substituting the above parameters into the formula and combining them with the hardware operating frequency (e.g., 1.4GHz), the computation time of the instruction can be calculated (e.g., derived to be 3366.4ns).

[0083] For global memory access primitive operators (such as the MA.TILE.WIN.LDG instructions) responsible for data transfer, the evaluation process calculates the total amount of data to be transferred based on the shape of the loaded matrix block. For example, for an input block with dimensions of 128×256 and a weight block of 256×128, the estimated total amount of data to be transferred using a specific data format is 0.125MB. Dividing this by the theoretical peak memory bandwidth (e.g., 32GB / s) yields the corresponding minimum transfer time T. mem (For example, 3906.25ns). Based on this, the evaluation process utilizes a double-buffering (ping-pong buffer) mechanism to quantify the overlap between computation and memory access. The computation time T is taken as... comp With transmission time T mem The maximum of the two, i.e., max(T) comp , T mem The effective time consumption for the overlapping interval is calculated as follows: The evaluation process involves the independent computation time of all primitive operators, such as local loading, core computation, and result write-back, and then aggregates them through overlapping logic to obtain the final node performance index.

[0084] In the fourth stage, after completing the derivation of a single node, the evaluation process traverses every computation node in the complete computation graph of the deep learning model and repeats the above analytical computation steps to output global performance metrics and operator-level bottleneck analysis.

[0085] In terms of global metrics, the evaluation process outputs the overall theoretical inference latency of the model (e.g., 150ms) and the theoretical average computational core utilization (e.g., 72%). Regarding bottleneck analysis, the evaluation process can directly pinpoint the critical computational nodes with the highest theoretical latency and determine the bottleneck type based on their execution characteristics. For example, when the report indicates that the total memory access time of a specific convolutional layer node is greater than the total computation time (i.e., T...), the bottleneck is identified. mem >T comp When this occurs, it is clearly identified as a memory access bottleneck. Furthermore, by analyzing the scale impact factor, the specific microarchitectural reasons can be revealed. For example, it can be pointed out that due to drastic changes in the number of input / output channels, the current block strategy has caused severe shared memory access conflicts, leading to a decrease in the memory access efficiency factor (i.e., the memory access class scale impact factor λ). mem The value is as low as 0.4. This analysis process provides direct quantitative basis for optimizing data layout or adjusting block size at the underlying level.

[0086] After determining the independent computation and memory access times, the evaluation process simulates the overlap of computation and memory access using a double buffering mechanism at the underlying level. The larger of the total computation time and the total memory access time is taken as the overlap time of the primitive operator sequence. In this example, 3906.25ns is taken as the final execution time of this block, and so on to complete the overlap computation of all primitive operators, thus obtaining the node performance index of the computing node.

[0087] As can be seen in the fourth stage, the evaluation process traverses the complete computation graph of the deep learning model, repeating the above decomposition and computation steps for each node. The final output includes global metrics and an operator-level bottleneck analysis report. Global metrics include the overall theoretical inference latency and theoretical average core utilization. In the operator-level analysis, if a specific computation node (such as the third convolutional layer in the network) is found to have the highest theoretical computation time, and its total memory access time exceeds its total computation time, it is clearly identified as a memory access bottleneck. Furthermore, the evaluation results quantify that the drastic changes in the number of input and output channels in this layer trigger severe shared memory access conflicts under the current block partitioning strategy, resulting in extremely low actual memory access efficiency reflected by its memory access scale impact factor.

[0088] In the fifth stage, performance feedback and closed-loop optimization are performed. The evaluation process generates a report file containing the detailed derivation data mentioned above for feedback. Based on this report, the data partitioning strategy is adjusted according to the bottleneck type of the restricted node, for example, reducing the thread block dimension from 128x128 to 64x128 to alleviate memory access conflicts. After adjustment, the optimization operator is actually run on hardware to obtain the actual measured latency (e.g., 130ms). Subsequently, data points containing the new scale dimension and measured latency are submitted through the calibration interface of the evaluation process. The evaluation process uses a lightweight optimization algorithm to calculate the actual scale impact factor based on the measured data and updates and corrects the parameters in the operator library for the corresponding scale. Through this closed-loop self-learning mechanism, the analytical model can directly output more accurate high-fidelity prediction results when facing operators of similar scale in the future.

[0089] According to another aspect of this disclosure, a performance evaluation device for a deep learning model based on a GPGPU architecture is also provided.

[0090] like Figure 3As shown, a performance evaluation device 300 for a deep learning model based on a GPGPU architecture includes: a first acquisition module 301 configured to acquire a computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; a conversion module 302 configured to convert the computation graph into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the multiple computation nodes and the dependencies between computation nodes, and wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; a second acquisition module 303 configured to acquire hardware configuration parameters of the GPGPU used to deploy the deep learning model; and a decomposition module 304 configured to decompose the operators corresponding to each computation node in the intermediate representation into hardware instruction-level instructions according to the hardware configuration parameters of the GPGPU. The instruction-level operator sequence includes at least one operator item from primitive operators and composite operators, wherein the composite operator is generated by combining the primitive operators according to a preset pattern; a third acquisition module 305 is configured to acquire, for each computing node and for each operator item in the instruction-level operator sequence corresponding to that computing node, a pre-set performance evaluation formula corresponding to that operator item, wherein the performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation caused by the operator item during actual execution; a first determination module 306 is configured to determine the node performance index of the computing node according to the performance evaluation formula corresponding to each operator item; and a second determination module 307 is configured to determine the execution time and performance bottleneck of the deep learning model based on the dependencies between the computing nodes and the node performance index of each computing node.

[0091] The first acquisition module 301 can, exemplarily, input a virtual tensor with a specific dimension into the deep learning model to be evaluated and perform forward propagation computation without triggering gradient updates, thereby recording the call order and data flow relationships between various computation nodes within the model. Thus, through the execution of real computational logic, the first acquisition module 301 can acquire a computational graph reflecting the model's topological structure during the inference phase. Since only forward propagation is involved, eliminating the differentiation and state preservation in backpropagation, the dynamic logical structure of the deep learning model when processing inputs of specific sizes can be captured with lower resource overhead, providing a topological basis for subsequent performance decomposition.

[0092] The conversion module 302 performs a depth-first traversal and parsing of the captured computation graph, transforming the abstract computation nodes in the graph into intermediate representations containing specific attributes. For example, this intermediate representation records the operator type of each computation node and extracts attribute information of the input and output tensors, such as the shape dimension and data precision of the tensors. Thus, unstructured graphical data is transformed into a quantifiable task sequence, achieving initial decoupling between algorithm-side requirements and underlying hardware specifications, and providing an objective data foundation for accurately mapping the computational load of the software layer to the physical constraints of the hardware side.

[0093] The second acquisition module 303 reads the physical specifications of the GPGPU used to deploy the deep learning model and obtains its hardware configuration parameters to establish the physical performance boundaries of the target hardware. By obtaining the hardware configuration parameters, the performance evaluation process can be anchored to real physical limitations, avoiding purely theoretical calculations detached from hardware architecture characteristics, and improving the reference value of the evaluation results for specific deployment environments.

[0094] The decomposition module 304, in conjunction with the acquired hardware configuration parameters, decomposes the operators originally at the algorithm level in the intermediate representation into a sequence of hardware instructions. Within this sequence, according to preset decomposition logic, operators are broken down into indivisible primitive operators (such as basic multiply-accumulate instructions or memory access instructions), or into composite operators composed of multiple primitive operators combined according to specific execution modes. This reconstructs the true execution trajectory of the operators at the micro-architecture level. This instruction-level decomposition allows for a more detailed observation of the scheduling behavior of operators on underlying resources, providing a prerequisite for identifying micro-level performance limiting factors.

[0095] The third acquisition module 305 matches a corresponding performance evaluation formula to each operator item obtained from the decomposition. These formulas not only consider the theoretical cycle of instruction execution but also specifically introduce a scale impact factor to quantify the performance degradation caused by data size, resource contention, or microarchitectural conflicts. Thus, a latency prediction model for the operator item in the actual operating environment is established. The technical effect of introducing the scale impact factor is that it can effectively capture the microarchitectural losses ignored by the theoretical peak, ensuring that the evaluation results are not merely an idealized representation of computing power but accurately reflect the true execution efficiency of the operator at a specific scale.

[0096] The first determining module 306 calculates the execution metrics of each operator item under the current hardware configuration according to the performance evaluation formula of each operator item. By logically aggregating the metrics of each operator item in the sequence, the node performance metrics of each computing node (i.e., operator) are determined. Thus, the upward aggregation from instruction-level metrics to operator-level metrics is completed. Through this hierarchical calculation scheme, the actual time occupied by each computing node on hardware resources can be accurately quantified, and the execution logic relationship between different instruction types can be taken into account, thereby obtaining a high-precision operator-level performance evaluation.

[0097] The second determination module 307 performs a time-series simulation of the complete execution flow of the deep learning model on the target GPGPU based on the dependencies between nodes and the calculated node performance metrics. This determines the total execution time of the model and identifies key bottlenecks affecting performance, enabling the evaluation device 300 to clearly pinpoint which layers or operators in the model are limiting overall efficiency. This achieves accurate and global performance evaluation of the deep learning model, providing quantitative data support for subsequent possible operator replacements, parameter adjustments, or hardware selection, significantly shortening the iteration cycle required for model deployment and optimization.

[0098] As can be seen, the performance evaluation device 300 for deep learning models based on the GPGPU architecture disclosed herein combines the high-level computational logic of deep learning models with the low-level physical constraints of the target hardware without relying on a real physical testing environment. By using fine-grained instruction-level operator decomposition and introducing a scale influence factor representing microarchitectural decay, it effectively compensates for the discrepancy between conventional purely theoretical estimations and real hardware execution. It avoids the huge time and computational overhead of low-level hardware cycle-level simulators and objectively reflects the scheduling characteristics and resource contention status of operators during real-world runtime at a microscopic level. Finally, it outputs a valuable overall model execution time and identifies key performance bottlenecks limiting execution efficiency, thus providing quantitative and direct data support for hardware selection, operator logic optimization, and overall computational resource allocation in the initial deployment of deep learning models.

[0099] According to some embodiments, the hardware configuration parameters of the GPGPU include at least one of storage resource parameters, computing resource parameters, and instruction execution characteristic parameters, wherein the storage resource parameters include at least one of shared memory capacity, register capacity, peak memory bandwidth, and cache size; the computing resource parameters include at least one of the number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency; and the instruction execution characteristic parameters include at least one of the ideal number of cycles per instruction, instruction execution throughput, and instruction set mapping relationship.

[0100] By acquiring multi-dimensional underlying hardware configuration parameters, a realistic physical boundary is established for the evaluation of deep learning models. Specifically, the shared memory capacity and register capacity in the storage resource parameters determine the maximum data block size that operators can use during execution, while peak memory bandwidth and cache size are used to deduce the transmission time during data handling. The number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency in the computing resource parameters establish the upper limit of the computing power of the hardware in concurrent instruction processing under ideal conditions and provide a benchmark for converting clock cycles into absolute time. The instruction execution characteristic parameters delve into the microarchitecture level, providing basic theoretical values ​​for calculating the specific hardware instruction execution cycle. Acquiring hardware configuration parameters allows the evaluation process to move beyond relying on coarse macroscopic indicators and instead establish high-precision underlying digital mappings for GPGPUs with different architectures, thereby improving the accuracy and applicability of cross-hardware platform evaluation.

[0101] According to some embodiments, the decomposition module 304 includes: a first acquisition unit configured to, for each computing node, acquire a target algorithm for the operator of the computing node from a preset plurality of candidate algorithms based on the operator parameter information corresponding to the computing node; a first determination unit configured to, based on the target algorithm and the hardware configuration parameters of the GPGPU, determine the data block parameters for parallel processing corresponding to the computing node; and a second determination unit configured to, based on the data block parameters, determine the number of calls and execution order of the operator items required for the implementation of the target algorithm, so as to generate an instruction-level operator sequence corresponding to the computing node.

[0102] Since operators in high-level deep learning frameworks are often abstract mathematical expressions, there are multiple different algorithmic implementation paths in the underlying hardware. Therefore, by extracting the specific parameter information of the operator, such as the dimensionality features of the tensor, the first acquisition unit matches the optimal underlying implementation strategy from a pre-defined candidate algorithm library. For example, when the computation node is identified as a convolution operator, it can be mapped to an implicit general matrix multiplication algorithm as the target algorithm based on the shape features of its input and output tensors. This process bridges the gap between high-level semantics and low-level execution, ensuring that the evaluation logic is consistent with the algorithm selection behavior of a real compiler.

[0103] The first determining unit, combining the determined target algorithm and the resource constraints of the target hardware, derives the data partitioning strategy for parallel computing. Since the high-speed on-chip storage resources of the GPGPU are limited and cannot accommodate the entire tensor at once, in some examples, large-scale matrices need to be divided into smaller blocks for iterative processing.

[0104] The second determining unit, based on the aforementioned determined data block parameters, further expands the algorithm into a sequence of instructions at the micro-architecture level. Once the size of the data block processed in a single operation is determined, the specific number of times the underlying memory access primitive operators and computation primitive operators need to be called in order to traverse the complete problem scale, as well as the execution order of these operators along the timeline, can be calculated. The second determining unit transforms the block strategy into a quantifiable set of micro-instructions, reducing the abstract operator nodes to the instruction level, providing a refined derivation framework for subsequently calling performance evaluation formulas line by line and applying scale influence factors.

[0105] As can be seen, the decomposition module 304 transforms abstract mathematical computation nodes into a sequence of execution actions that aligns with the characteristics of the underlying hardware microarchitecture. This decomposition mechanism, from high-level operators to low-level instructions, allows the performance evaluation process to move beyond coarse-grained theoretical computing power estimations and deeply reflect the actual scheduling of physical resources. Specifically, based on the data partitioning strategy and execution order derived from the target algorithm and hardware configuration, the actual movement trajectory of tensor data across multi-level storage structures and the cyclic processing logic of core computing units are reconstructed. This decomposition process establishes a fine-grained derivation framework for subsequent evaluation formulas that include scale-related factors, ensuring that the final performance metrics effectively reflect the specific limitations imposed by underlying microarchitecture differences on the execution efficiency of deep learning models.

[0106] According to some embodiments, the performance evaluation formula corresponding to the operator item is pre-stored in a preset operator performance library, wherein the operator performance library includes: a primitive operator library for storing the primitive operators and their corresponding primitive performance evaluation formulas; and a composite operator library for storing the composite operators and their corresponding fusion performance evaluation formulas.

[0107] In the specific architecture implementation, the primitive operator library serves as the cornerstone of the underlying evaluation. It stores hardware instruction-level operators that cannot be further decomposed, such as floating-point operators, memory access operators, and thread synchronization operators, as well as their corresponding basic mathematical evaluation models. These models serve as basic performance evaluation formulas to quantify the execution time of a single underlying instruction.

[0108] The composite operator library is geared towards execution optimization scenarios, storing high-frequency operator templates composed of the primitive operators combined in a preset fixed pattern, along with their corresponding fusion performance evaluation formulas. In the actual inference of deep learning models, there are numerous frequently invoked operator combination sequences. To accurately quantify the actual performance of these combinations after underlying compilation optimization, the composite operator library covers a variety of typical fusion operator templates.

[0109] According to some embodiments, the third acquisition module 305 includes: a matching unit configured to perform pattern matching between the instruction-level operator sequence corresponding to each computing node and the composite operators in the composite operator library; a second acquisition unit configured to, in response to a successful match between at least a portion of the continuous primitive operators in the instruction-level operator sequence and a target composite operator in the composite operator library, use the target composite operator as an operator item in the instruction-level operator sequence and acquire a fusion performance evaluation formula corresponding to the target composite operator from the composite operator library; and a third acquisition unit configured to, in response to a failed match of primitive operators in the instruction-level operator sequence, acquire a primitive performance evaluation formula corresponding to the failed match of primitive operators from the primitive operator library.

[0110] During the specific operation phase of formula acquisition, the generated instruction-level operator sequence is traversed. The matching unit can check whether there are instruction sequence segments in the sequence that meet the optimization conditions through mechanisms such as sliding windows or subgraph topology recognition. As a specific example, when the generated instruction sequence contains convolution multiplication-addition primitive operators, bias addition primitive operators, and nonlinear activation primitive operators in sequence, the matching unit will identify that the continuous segment is consistent with the features of the preset convolution bias activation fusion template in the composite operator library. At this time, the matching unit no longer accumulates the memory access and computation overhead of these three primitive operators in isolation, but extracts them as a whole composite operator item, and the second acquisition unit calls the corresponding fusion performance evaluation formula to calculate the overall time consumption, thereby eliminating the time consumption of multiple intermediate result write-back and read-back caused by independent execution. For independent primitive operators in the sequence that fail to form a preset template due to specific data dependencies or stride mismatch, their primitive operator form is retained, and the basic evaluation formula is requested from the primitive operator library for calculation.

[0111] According to some embodiments, the primitive operator includes computational primitive operators, and wherein the performance metrics of the computational primitive operators include computation time T. comp The primitive performance evaluation formula corresponding to the computational primitive operator is: T comp = N inst x Cycle inst x (1 + λ comp ), where N inst Cycle represents the number of times the instruction corresponding to the primitive operator of this computation class is executed. inst λ is the ideal cycle number for a single instruction. comp This is the scale influence factor for computational primitive operators.

[0112] This computational scheme, based on instruction-level physical cycles and incorporating microarchitecture decay factors, allows for a shift in computation time estimation beyond the traditional, coarse framework that relies solely on theoretical peak computing power. This enables the capture of the actual computational efficiency degradation of operators under specific hardware resource constraints, thus ensuring high fidelity and quantization accuracy in performance evaluation of computationally intensive tasks.

[0113] According to some embodiments, the primitive operator further includes a memory access primitive operator, wherein the performance metric of the memory access primitive operator includes memory access time T. mem The primitive performance evaluation formula corresponding to the memory access primitive operator is determined based on the actual amount of data transferred, the peak memory bandwidth of the GPGPU, and the scale influence factor of the memory access primitive operator.

[0114] By combining the actual amount of physical data movement with the scale-based impact factor characterizing microarchitecture memory access overhead, the evaluation system for memory-constrained operators has been improved. This allows the evaluation process to sensitively identify non-ideal latency masked in theoretical bandwidth calculations, providing compilers with clear and objective quantitative references for optimizing data storage layout and adjusting memory access steps at the underlying level.

[0115] According to some embodiments, the first determining module 306 includes: a calculation unit configured to calculate the performance index of each operator item according to the performance evaluation formula corresponding to each operator item; a third determining unit configured to determine the total time consumption of the composite operator based on the performance index of the composite operator in response to determining that the instruction-level operator sequence corresponding to the computing node includes the composite operator; an aggregation unit configured to aggregate the performance indexes of all computation-type primitive operators into a total computation time and aggregate the performance indexes of all memory-type primitive operators into a total memory access time in response to determining that the instruction-level operator sequence corresponding to the computing node includes both the computation-type primitive operator and the memory access-type primitive operator; an overlap unit configured to determine the larger of the total computation time and the total memory access time as the primitive operator overlap time; and a fourth determining unit configured to determine the node performance index of the computing node based on the total time consumption of the composite operator and the primitive operator overlap time.

[0116] In the above embodiment, the first determining module 306 performs splitting and parallel processing on the overall time consumption of the computing nodes. Specifically, for the composite operators determined by pattern matching in the sequence, since they have been compiled into highly integrated execution modules at the underlying level, the register-level overlap benefits within them have been deducted in the corresponding fusion performance evaluation formula. Therefore, the first determining module 306 first independently extracts and calculates the performance indicators of these composite operators, summarizing them into the total time consumption of the composite operators. At the same time, for the remaining discrete primitive operators in the sequence, the first determining module 306 performs similar term aggregation on the time consumption of computational primitive operators responsible for logical operations and the time consumption of memory access primitive operators responsible for data movement. Under the hardware's double-buffered execution mechanism, when the computing core is processing the current data block, the memory controller is often asynchronously loading the next data block. Therefore, the first determining module 306 does not use simple linear time accumulation, but instead takes the larger value between the total computing time and the total memory access time as the overlap time consumption of the primitive operators. Finally, the total time consumed by the composite operator is added to the time consumed by the overlap of the primitive operator to establish the node performance index of the computing node.

[0117] According to some embodiments, the apparatus further includes: a fourth acquisition module configured to run a target operator term on the GPGPU and acquire the measured latency T of the target operator term running on the GPGPU. real The fifth acquisition module is configured to acquire the theoretical reference time T of the target operator under ideal conditions. base The first calculation module is configured to calculate the actual scale impact factor λ using the following formula. real :λ real = T real / T base - 1; The second calculation module is configured to calculate the actual scale influence factor λ. real The deviation value between the target operator item and the pre-stored scale influence factor; and an update module, configured to, in response to the deviation value being greater than or equal to a threshold, update the actual scale influence factor λ. real Update the stored scale impact factors.

[0118] In this closed-loop calibration embodiment, the fourth acquisition module executes a specific target operator term individually on the target GPGPU in the form of a lightweight test to obtain its actual execution time on the physical device as the measured latency T. real Meanwhile, based on the number of basic instruction executions and the ideal clock cycle of this operator, the fifth acquisition module derives its theoretical baseline time T, which is not limited by microarchitectural resource contention. baseBy applying the above formula, the first calculation module normalizes the additional latency overhead exceeding the ideal state in the measured data, thereby calculating the actual scale impact factor λ, which reflects the actual microarchitecture loss of the current hardware. real Subsequently, the second calculation module compares the calculated actual factor with the old scale influence factor pre-stored in the current operator performance library. If the calculated deviation exceeds the set tolerance threshold, it indicates that the original preset parameters can no longer accurately represent the current underlying hardware state. In this case, the update module will trigger a library update action, overwriting or correcting the extracted actual scale influence factor into the corresponding entry in the performance library.

[0119] It is understood that the components of the performance evaluation device 300 for deep learning models based on GPGPU architecture are used to execute the method steps defined in method 100 to achieve performance evaluation of deep learning models based on GPGPU architecture, which will not be elaborated further by the inventors here.

[0120] According to another aspect of this disclosure, an electronic device is also provided, including at least one processor, wherein each of the at least one processor includes: a processor cache; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0121] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the methods described above.

[0122] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method.

[0123] See Figure 4 Electronic device 400 will now be described as an example of a hardware device (electronic device) that can be applied to various aspects of this disclosure. Electronic device 400 can be any machine configured to perform processing and / or computation, and can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, in-vehicle computer, or any combination thereof. The methods described above can be implemented wholly or at least partially by electronic device 400 or similar devices or systems.

[0124] Electronic device 400 may include elements that are connected to or communicate with bus 402 (possibly via one or more interfaces). For example, electronic device 400 may include bus 402, one or more processors 404, one or more input devices 406, and one or more output devices 408. The one or more processors 404 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more dedicated processors (e.g., special-purpose chips). Input devices 406 may be any type of device capable of inputting information to electronic device 400 and may include, but are not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote control. Output devices 408 may be any type of device capable of presenting information and may include, but are not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Electronic device 400 may also include a non-transitory storage device 410. The non-transitory storage device can be any storage device that is non-transitory and capable of storing data, including but not limited to disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, optical discs or any other optical media, ROM (read-only memory), RAM (random access memory), cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 410 can be detached from an interface. The non-transitory storage device 410 may have data / programs (including instructions) / code for implementing the methods and steps described above. Electronic device 400 may also include a communication device 412. The communication device 412 can be any type of device or system that enables communication with external devices and / or with a network, and may include, but is not limited to, modems, network interface cards, infrared communication devices, wireless communication devices and / or chipsets, such as Bluetooth™ devices, 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0125] Electronic device 400 may also include working memory 414, which may be any type of working memory that can store programs (including instructions) and / or data useful for the operation of processor 404, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0126] Software elements (programs) may reside in working memory 414, including but not limited to operating system 416, one or more application programs 418, drivers, and / or other data and code. Instructions for performing the methods and steps described above may be included in one or more application programs 418, and method 100 described above may be implemented by processor 404 reading and executing the instructions of one or more application programs 418. More specifically, each step in method 100 described above may be implemented, for example, by processor 404 executing an application program 418 having instructions for performing the steps described above. Furthermore, other steps in the method described above may be implemented, for example, by processor 404 executing an application program 418 having instructions for performing the corresponding steps. Executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (e.g., storage device 410 described above) and may be stored in working memory 414 during execution (possibly compiled and / or installed). Executable code or source code of the instructions of the software elements (programs) may also be downloaded from a remote location.

[0127] It should also be understood that various modifications can be made depending on specific requirements. For example, custom hardware can also be used, and / or specific elements can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and apparatus can be implemented by programming hardware (e.g., programmable logic circuits including field-programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) using logic and algorithms according to this disclosure in assembly language or hardware programming languages ​​(such as Verilog, VHDL, C++).

[0128] It should also be understood that the aforementioned methods can be implemented using a server-client model. For example, the client can receive user input data and send it to the server. Alternatively, the client can receive user input data, perform a portion of the processing described in the aforementioned methods, and send the resulting data to the server. The server can receive data from the client, execute the aforementioned methods or a portion thereof, and return the execution result to the client. The client can receive the execution result from the server and, for example, present it to the user via an output device.

[0129] It should also be understood that the components of electronic device 400 can be distributed across a network. For example, some processing can be performed using one processor, while other processing can be performed simultaneously by another processor located far away from that processor. Other components of computing system 400 can also be distributed similarly. Thus, electronic device 400 can be interpreted as a distributed computing system that performs processing in multiple locations.

[0130] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A performance evaluation method for a deep learning model based on a GPGPU architecture, the method comprising: Obtain the computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; The computation graph is converted into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the plurality of computation nodes and the dependencies between computation nodes, and wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; Obtain the hardware configuration parameters of the GPGPU used to deploy the deep learning model; According to the hardware configuration parameters of the GPGPU, the operator corresponding to each computing node in the intermediate representation is decomposed into a hardware instruction-level operator sequence, wherein the instruction-level operator sequence includes at least one operator item among primitive operators and composite operators, and wherein the composite operator is generated by combining the primitive operators according to a preset pattern. For each computing node, for each operator item in the instruction-level operator sequence corresponding to that computing node, a pre-set performance evaluation formula corresponding to that operator item is obtained, wherein the performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation caused by the operator item during actual operation. Based on the performance evaluation formula corresponding to each operator item, determine the node performance index of the computing node; and Based on the dependencies between the computing nodes and the node performance metrics of each computing node, the execution time and performance bottleneck of the deep learning model are determined.

2. The method as described in claim 1, wherein, The step of decomposing the operator corresponding to each computing node in the intermediate representation into a sequence of hardware instruction-level operators based on the hardware configuration parameters of the GPGPU includes: For each computing node, based on the operator parameter information corresponding to that computing node, the target algorithm for the operator for that computing node is obtained from a set of multiple candidate algorithms. Based on the target algorithm and the hardware configuration parameters of the GPGPU, determine the data block parameters for parallel processing corresponding to this computing node; and Based on the data block parameters, the number of times the operator items are called and the execution order required for the implementation of the target algorithm are determined, so as to generate the instruction-level operator sequence corresponding to the computing node.

3. The method as described in claim 1 or 2, wherein, The performance evaluation formulas corresponding to the operator terms are pre-stored in a preset operator performance library, wherein the operator performance library includes: The primitive operator library stores the primitive operators and their corresponding primitive performance evaluation formulas; and A composite operator library is used to store the composite operators and their corresponding fusion performance evaluation formulas.

4. The method of claim 3, wherein, The step of obtaining a pre-set performance evaluation formula corresponding to each operator item in the instruction-level operator sequence for each computing node includes: For each computing node, the instruction-level operator sequence corresponding to that computing node is matched with the composite operators in the composite operator library; In response to a successful match between at least a portion of the consecutive primitive operators in the instruction-level operator sequence and a target composite operator in the composite operator library, the target composite operator is used as an operator item in the instruction-level operator sequence, and a fusion performance evaluation formula corresponding to the target composite operator is obtained from the composite operator library; and In response to the instruction-level operator sequence including unmatched primitive operators, the primitive performance evaluation formula corresponding to the unmatched primitive operator is obtained from the primitive operator library.

5. The method of claim 3, wherein, The primitive operators include computational primitive operators, and the performance metrics of the computational primitive operators include computation time T. comp The primitive performance evaluation formula corresponding to the computational primitive operator is as follows: T comp = N inst x Cycle inst x (1 + λ comp ), Where, N inst Cycle represents the number of times the instruction corresponding to the primitive operator of this computation class is executed. inst λ is the ideal cycle number for a single instruction. comp This is the scale influence factor for computational primitive operators.

6. The method of claim 5, wherein, The primitive operators also include memory access primitive operators, and the performance metrics of the memory access primitive operators include memory access time T. mem The primitive performance evaluation formula corresponding to the memory access primitive operator is determined based on the actual amount of data transferred, the peak memory bandwidth of the GPGPU, and the scale influence factor of the memory access primitive operator.

7. The method of claim 6, wherein, The step of determining the node performance index of the computing node according to the performance evaluation formula corresponding to each operator item includes: Calculate the performance index of each operator based on the performance evaluation formula corresponding to each operator; In response to determining that the instruction-level operator sequence corresponding to the computing node includes the composite operator, the total time consumption of the composite operator is determined based on the performance index of the composite operator; In response to determining that the instruction-level operator sequence corresponding to the computing node includes both the computational primitive operator and the memory access primitive operator, the performance indicators of all computational primitive operators are aggregated into the total computing time, and the performance indicators of all memory access primitive operators are aggregated into the total memory access time. The larger of the total computation time and the total memory access time is determined as the primitive operator overlap time; and Based on the total time consumed by the composite operator and the time consumed by the overlap of the primitive operators, the node performance index of the computing node is determined.

8. The method of any one of claims 1-7, further comprising: Run the target operator term on the GPGPU and obtain the measured latency T of the target operator term running on the GPGPU. real ; Obtain the theoretical reference time T of the target operator under ideal conditions. base ; Calculate the actual scale impact factor λ using the following formula. real :λ real = T real / T base - 1; Calculate the actual scale impact factor λ real The deviation value between the target operator item and the pre-stored scale influence factor; as well as In response to the deviation value being greater than or equal to the threshold, based on the actual scale influence factor λ real Update the stored scale impact factors.

9. The method according to any one of claims 1-8, wherein, The hardware configuration parameters of the GPGPU include at least one of storage resource parameters, computing resource parameters, and instruction execution characteristic parameters, wherein, The storage resource parameters include at least one of shared memory capacity, register capacity, peak memory bandwidth, and cache size; The computing resource parameters include at least one of the following: number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency. The instruction execution characteristic parameters include at least one of the following: ideal cycle count per instruction, instruction execution throughput, and instruction set mapping relationship.

10. A performance evaluation device for a deep learning model based on a GPGPU architecture, the device comprising: The first acquisition module is configured to acquire the computation graph of the deep learning model, wherein the computation graph includes multiple computation nodes; The conversion module is configured to convert the computation graph into an intermediate representation, wherein the intermediate representation includes operator parameter information of the operators of each of the plurality of computation nodes and the dependencies between the computation nodes, and wherein the operator parameter information includes operator type and tensor attributes of the input tensor and output tensor corresponding to the computation node; The second acquisition module is configured to acquire the hardware configuration parameters of the GPGPU used to deploy the deep learning model; The decomposition module is configured to decompose the operator corresponding to each computing node in the intermediate representation into a hardware instruction-level operator sequence according to the hardware configuration parameters of the GPGPU, wherein the instruction-level operator sequence includes at least one operator item among primitive operators and composite operators, and wherein the composite operator is generated by combining the primitive operators according to a preset pattern. The third acquisition module is configured to acquire a pre-set performance evaluation formula corresponding to each operator item in the instruction-level operator sequence corresponding to each computing node. The performance evaluation formula includes a scale influence factor, which is used to characterize the performance degradation of the operator item during actual operation. The first determining module is configured to determine the node performance index of the computing node based on the performance evaluation formula corresponding to each operator item; and The second determining module is configured to determine the execution time and performance bottleneck of the deep learning model based on the dependencies between the computing nodes and the node performance indicators of each computing node.

11. The apparatus of claim 10, wherein, The decomposition module includes: The first acquisition unit is configured to, for each computing node, acquire the target algorithm for the operator of that computing node from a set of multiple candidate algorithms based on the operator parameter information corresponding to that computing node. The first determining unit is configured to determine the data block parameters for parallel processing corresponding to the computing node based on the target algorithm and the hardware configuration parameters of the GPGPU; and The second determining unit is configured to determine the number of times the operator items are called and the execution order required to implement the target algorithm based on the data block parameters, so as to generate the instruction-level operator sequence corresponding to the computing node.

12. The apparatus of claim 10 or 11, wherein, The performance evaluation formulas corresponding to the operator terms are pre-stored in a preset operator performance library, wherein the operator performance library includes: The primitive operator library stores the primitive operators and their corresponding primitive performance evaluation formulas; and A composite operator library is used to store the composite operators and their corresponding fusion performance evaluation formulas.

13. The apparatus of claim 12, wherein, The third acquisition module includes: The matching unit is configured to perform pattern matching between the instruction-level operator sequence corresponding to each computing node and the composite operators in the composite operator library for each computing node. The second acquisition unit is configured to, in response to a successful match between at least a portion of the consecutive primitive operators in the instruction-level operator sequence and a target composite operator in the composite operator library, include the target composite operator as an operator item in the instruction-level operator sequence and acquire, from the composite operator library, a fusion performance evaluation formula corresponding to the target composite operator; and The third acquisition unit is configured to, in response to the instruction-level operator sequence including unmatched primitive operators, acquire from the primitive operator library the primitive performance evaluation formula corresponding to the unmatched primitive operator.

14. The apparatus of claim 12, wherein, The primitive operators include computational primitive operators, and the performance metrics of the computational primitive operators include computation time T. comp The primitive performance evaluation formula corresponding to the computational primitive operator is as follows: T comp = N inst x Cycle inst x (1 + λ comp ), Where, N inst Cycle represents the number of times the instruction corresponding to the primitive operator of this computation class is executed. inst λ is the ideal cycle number for a single instruction. comp This is the scale influence factor for computational primitive operators.

15. The apparatus of claim 14, wherein, The primitive operators also include memory access primitive operators, and the performance metrics of the memory access primitive operators include memory access time T. mem The primitive performance evaluation formula corresponding to the memory access primitive operator is determined based on the actual amount of data transferred, the peak memory bandwidth of the GPGPU, and the scale influence factor of the memory access primitive operator.

16. The apparatus of claim 15, wherein, The first determining module includes: The calculation unit is configured to calculate the performance index of each operator item according to the performance evaluation formula corresponding to each operator item; The third determining unit is configured to determine the total time consumption of the composite operator based on the performance index of the composite operator in response to determining that the instruction-level operator sequence corresponding to the computing node includes the composite operator; The aggregation unit is configured to, in response to determining that the instruction-level operator sequence corresponding to the computing node includes both the computational primitive operators and the memory access primitive operators, aggregate the performance metrics of all computational primitive operators into the total computation time, and aggregate the performance metrics of all memory access primitive operators into the total memory access time; Overlapping units are configured to determine the greater of the total computation time and the total memory access time as the primitive operator overlap time; and The fourth determining unit is configured to determine the node performance index of the computing node based on the total time consumed by the composite operator and the overlapping time consumed by the primitive operator.

17. The apparatus of any one of claims 10-16, further comprising: The fourth acquisition module is configured to run the target operator term on the GPGPU and acquire the measured latency T of the target operator term running on the GPGPU. real ; The fifth acquisition module is configured to acquire the theoretical reference time T of the target operator under ideal conditions. base ; The first calculation module is configured to calculate the actual scale impact factor λ using the following formula. real :λ real = T real / T base - 1; The second calculation module is configured to calculate the actual scale influence factor λ. real The deviation value between the target operator item and the pre-stored scale influence factor; as well as The update module is configured to respond to the deviation value being greater than or equal to a threshold, based on the actual scale influence factor λ. real Update the stored scale impact factors.

18. The apparatus according to any one of claims 10-17, wherein, The hardware configuration parameters of the GPGPU include at least one of storage resource parameters, computing resource parameters, and instruction execution characteristic parameters, wherein, The storage resource parameters include at least one of shared memory capacity, register capacity, peak memory bandwidth, and cache size; The computing resource parameters include at least one of the following: number of streaming multiprocessors, theoretical peak computing power, and hardware operating frequency. The instruction execution characteristic parameters include at least one of the following: ideal cycle count per instruction, instruction execution throughput, and instruction set mapping relationship.

19. An electronic device comprising: At least one processor, wherein each of the at least one processor comprises: Processor cache; and A memory communicatively connected to the at least one processor, wherein The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-9.

Images