An operator development method and device based on an AI inference engine

Abstract

The application provides an operator development method and device based on an AI inference machine; the operator development method based on the AI inference machine comprises the following steps: obtaining an operator golden reference module and a micro-architecture description table of a target chip, and generating an operator intermediate representation model according to the operator golden reference module; calculating a theoretical lower bound time consumption of the operator on a target instruction set according to the operator intermediate representation model and the micro-architecture description table; generating a candidate operator code according to the operator intermediate representation model and the target instruction set; deploying the candidate operator code to the AI inference machine for execution, obtaining a measured result, and performing evaluation according to the measured result, the operator golden reference module and the theoretical lower bound time consumption; and performing evaluation according to the evaluation result. The application improves the stability, reproducibility and development efficiency of the quality of the operator library.

Description

Technical Field

[0001] This application relates to the field of AI operator technology, and in particular to a method and apparatus for developing operators based on AI inference engines. Background Technology

[0002] As artificial intelligence models continue to evolve towards multimodal and large-parameter approaches, AI inference clusters composed of heterogeneous processing chips such as GPUs, NPUs, DSPs, and FPGAs have been widely deployed in the cloud and at the edge. Operators, as key execution units connecting algorithm models and underlying hardware, directly determine the overall performance of the inference cluster through their development efficiency and optimization quality.

[0003] In current engineering practice, operator development mainly faces two technical challenges. Firstly, different manufacturers' chips employ different instruction set architectures. The same operator often requires manual coding and optimization on multiple instruction sets such as CUDA, NEON, HVX, and AVX-512, resulting in significant fragmentation, repetitive investment, and difficulty in ensuring version consistency. Secondly, the operator optimization process lacks objective and quantifiable performance benchmarks. Currently, the industry's judgment on whether an operator is optimized to the appropriate level relies heavily on the individual experience of engineers. Different developers have significantly different performance assessments of the same operator on the same chip, making it difficult to achieve stable convergence in the overall quality of the operator library. Cross-platform porting often requires a significant investment of manpower for adaptation and optimization.

[0004] Therefore, how to achieve the engineering and standardization of operator development in a heterogeneous instruction set environment, and establish an objective and unified performance evaluation benchmark, has become a key bottleneck restricting the full release of computing power in AI inference clusters. Summary of the Invention

[0005] Therefore, it is necessary for the present invention to provide a method and system for developing operators based on AI inference engines, in order to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above objectives, a method for developing operators based on AI inference engines includes the following steps: Obtain the microarchitecture description table of the operator golden reference module and the target chip, and generate the operator intermediate representation model based on the operator golden reference module; Based on the operator intermediate representation model and the microarchitecture description table, calculate the theoretical lower bound time consumption of the operator on the target instruction set; Based on the operator intermediate representation model and the target instruction set, candidate operator codes are generated; The candidate operator code is deployed to the AI ​​inference engine for execution, and the test results are obtained. The evaluation is then performed based on the test results, the time consumption of the operator's golden reference module, and the theoretical lower bound. Based on the evaluation results, candidate operator codes that meet the preset conditions are output to the operator library, or candidate operator codes that do not meet the preset conditions are returned for optimization.

[0007] Preferably, the present invention also provides an apparatus for developing operators based on an AI inference engine, which stores a computer program. The apparatus includes at least an AI inference engine, which includes a main control chip and a computing module composed of several ClusterUnits. Each ClusterUnit includes at least one AI processing chip and is hard-connected via a high-speed inter-chip interconnect bus. The apparatus also includes a processor and a memory, which is coupled to the processor. The memory is used to store computer program code, which includes computer instructions. When the processor reads the computer instructions from the memory, the apparatus executes the method described above.

[0008] The beneficial effects of this invention are as follows: On the one hand, by introducing the operator golden reference module and the theoretical lower bound time calculation driven by the microarchitecture description table, the performance judgment that originally relied on the subjective experience of engineers is transformed into an objective quantitative benchmark based on the chip's physical computing power and storage bandwidth. This enables operator implementations produced by different developers at different times to converge toward the same theoretical optimal boundary, fundamentally eliminating the optimization quality fluctuations caused by differences in human experience and improving the stability and reproducibility of the overall quality of the operator library.

[0009] On the other hand, by decoupling the operator intermediate representation model from the target instruction set, the operator computation logic is abstracted and separated from the specific backend implementation, so that the same operator description can be automatically mapped to multiple heterogeneous instruction sets such as CUDA, NEON, HVX, and AVX-512. This avoids the fragmented development method of repeatedly writing and optimizing for each instruction set in the traditional mode, significantly compresses the cross-platform operator adaptation cycle, and reduces the cost of version consistency maintenance.

[0010] On the other hand, by jointly evaluating the test results of candidate operator code on the AI ​​inference engine with the theoretical lower bound time and the golden reference output, a closed loop of dual constraints on accuracy and performance is constructed. This allows potential accuracy regression or performance degradation issues during the optimization process to be automatically intercepted before the code is put into the library, rather than being exposed in the subsequent application stage, thereby improving the engineering level and delivery reliability of the operator development process. Attached Figure Description

[0011] Figure 1 This is a flowchart of the operator development method based on AI inference engine described in this application.

[0012] Figure 2 for Figure 1The diagram shows the hardware architecture with both offline host machine and AI inference machine deployed on both sides. Detailed Implementation

[0013] The technical method of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0014] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0015] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0016] To achieve the above objectives, please refer to Figures 1 to 2 The present invention provides a method for developing operators based on AI inference engines, which mainly includes the following steps S100 to S500.

[0017] S100: Obtain the microarchitecture description table of the operator gold reference module and the target chip, and generate the operator intermediate representation model based on the operator gold reference module.

[0018] S200: Calculate the theoretical lower bound time consumption of the operator on the target instruction set based on the operator intermediate representation model and the microarchitecture description table.

[0019] S300: Generate candidate operator code based on the operator intermediate representation model and the target instruction set.

[0020] S400: Deploy the candidate operator code to the AI ​​inference engine for execution, obtain the test results, and evaluate the time consumption based on the test results, the operator's golden reference module, and the theoretical lower bound.

[0021] S500: Based on the evaluation results, output the candidate operator code that meets the preset conditions to the operator library, or return the candidate operator code that does not meet the preset conditions for optimization.

[0022] Preferably, steps S010 to S011 may be included before calculating the time taken to reach the theoretical lower bound.

[0023] S010: Based on the hardware topology descriptor of the AI ​​inference engine, determine the inter-chip data transfer amount and inter-chip synchronization overhead of the operator executed across computing cluster units on the target instruction set.

[0024] The hardware topology descriptor is structured data used to characterize the physical interconnection relationships and communication capabilities between the computing cluster units in the AI ​​inference engine. Optionally, the hardware topology descriptor includes the number of computing cluster units, the identifier of each computing cluster unit, the interconnection topology type between computing cluster units, the interconnection link bandwidth, the interconnection link latency, and the local storage capacity of each computing cluster unit.

[0025] In some embodiments, a computing cluster unit refers to the smallest independently schedulable computing unit in an AI inference engine, consisting of multiple homogeneous processing cores and a shared cache. For example, in a GPU-based AI inference engine, the computing cluster unit can be a graphics processing cluster; in an NPU-based AI inference engine, the computing cluster unit can be a neural network computing cluster. The computing cluster units communicate through an inter-chip interconnect network, whose topology includes ring topology, fully connected topology, fat tree topology, or two-dimensional grid topology. Inter-chip data transfer volume refers to the total amount of data communication generated during distributed execution of operators due to the need to migrate input data or intermediate results between the local storage of different computing cluster units. Inter-chip synchronization overhead refers to the time overhead required for synchronization operations performed by each computing cluster unit after completing local computation to maintain data consistency or execution order dependencies.

[0026] S011: Based on the inter-chip data transfer volume and inter-chip synchronization overhead, the operator computation volume and data transfer volume corresponding to the operator intermediate representation model are corrected to obtain the corrected operator computation volume and data transfer volume.

[0027] Here, the operator computational load corresponding to the intermediate representation model refers to the total number of operations derived from the mathematical definition of the operator and the dimensions of the input and output tensors; the data transport volume corresponding to the intermediate representation model refers to the total amount of raw data read and written required for the operator to execute under an ideal single-node continuous storage model. Optionally, the modified operator computational load is used to characterize the equivalent total computational load when executing across computing cluster units, and the modified data transport volume is used to characterize the equivalent total data movement load when executing across computing cluster units.

[0028] In some embodiments, inter-chip synchronization overhead includes synchronization control signaling transmission volume and synchronization reduction computation volume. Synchronization control signaling transmission volume refers to the total amount of control information transmitted by each computing cluster unit for state negotiation and pace alignment when executing synchronization primitives; synchronization reduction computation volume refers to the total amount of local accumulation or comparison operations generated when performing data reduction synchronization between computing cluster units. When correcting operator computation volume, the original operator computation volume is merged with the synchronization reduction computation volume in the inter-chip synchronization overhead to obtain the corrected operator computation volume; when correcting data transfer volume, the original data transfer volume is merged with the inter-chip data transfer volume to obtain the corrected data transfer volume.

[0029] Preferably, step S200 includes steps S201-S204.

[0030] S201: Based on the corrected operator computational load and data transfer load, and combined with the peak computing power and storage bandwidth in the microarchitecture description table, calculate the initial theoretical latency based on the Roofline model; wherein, the initial theoretical latency is the larger of the computation-limited latency and the memory-limited latency, the computation-limited latency is obtained by dividing the corrected operator computational load by the peak computing power, and the memory-limited latency is obtained by dividing the corrected data transfer load by the storage bandwidth.

[0031] In one implementation, peak computing power refers to the upper limit of the nominal number of computational operations that the target chip can complete per unit time when all computing units under the target instruction set are running continuously at maximum theoretical throughput; storage bandwidth refers to the upper limit of the maximum sustainable data transmission rate of the data path between the main memory and the on-chip computing array under the target instruction set.

[0032] In some embodiments, the Roofline model is a theoretical analysis model that determines the performance bottleneck by comparing the computational intensity of an operator with the computing power-to-bandwidth ratio of the hardware platform. Computational intensity is defined as the ratio of the corrected operator computational load to the corrected data transfer load, characterizing the computational operation density corresponding to a unit of data transfer volume. When the operator's computational intensity is greater than the ratio of the hardware platform's peak computing power to its storage bandwidth, the operator performance is limited by computing power, and the initial theoretical latency is determined by the computationally limited latency. When the operator's computational intensity is less than this ratio, the operator performance is limited by data supply capacity, and the initial theoretical latency is determined by the memory access limited latency.

[0033] In some embodiments, computation-constrained time refers to the lower limit of the purely theoretical computation time required to complete the corrected operator computation when all computing units of the target chip are busy and instruction issuance is unblocked; memory access-constrained time refers to the lower limit of the purely theoretical data transfer time required to complete the corrected data transfer when there is no contention at the memory interface and no bus conflicts. The initial theoretical time is the larger of the computation-constrained time and the memory access-constrained time.

[0034] S202: Calculate the inter-chip synchronization time penalty term based on the inter-chip synchronization overhead and the typical synchronization primitive overhead in the microarchitecture description table.

[0035] Typical synchronization primitive overhead refers to the baseline time overhead required for the target chip to execute various inter-chip synchronization operations under the target instruction set, including global barrier synchronization overhead, point-to-point signaling overhead, reduction synchronization overhead, and atomic operation overhead. Optionally, typical synchronization primitive overhead is provided by the chip manufacturer in the microarchitecture description table in the form of a lookup table, or obtained by actual measurement and calibration of idle synchronization instructions through benchmark testing programs. The inter-chip synchronization time consumption penalty term refers to the time compensation amount added to the initial theoretical consumption after mapping various synchronization operation quantities in the inter-chip synchronization overhead to equivalent time dimensions.

[0036] In some embodiments, global barrier synchronization overhead refers to the baseline time required for all participating computing cluster units to wait for all units to be ready at the same synchronization point, which is determined by the longest propagation path of the interconnect topology and the single-hop link delay; point-to-point signaling overhead refers to the baseline time required for two computing cluster units to send a handshake confirmation; reduction synchronization overhead refers to the baseline time required for multiple computing cluster units to perform reduction operations on their respective local data and summarize them into a designated unit; atomic operation overhead refers to the baseline time required to perform an atomic read-modify-write operation across computing cluster units.

[0037] S203: Combine the initial theoretical time consumption with the inter-chip synchronization time consumption penalty term to obtain the theoretical lower bound time consumption.

[0038] The theoretical lower bound execution time refers to the shortest execution time that cannot be exceeded when an operator executes across computing cluster units on the target instruction set, after simultaneously considering the triple physical constraints of the target chip's peak computing power, storage bandwidth, and inter-chip synchronization mechanisms. Optionally, the theoretical lower bound execution time is used as an absolute benchmark for subsequent performance evaluation, and the actual execution time of any candidate operator code cannot be lower than this theoretical lower bound execution time.

[0039] In some embodiments, merging refers to arithmetically adding the initial theoretical execution time and the inter-chip synchronization time penalty term along the critical path of operator execution. Since the initial theoretical execution time only reflects the lower limit of the physical time required for each computing cluster unit to complete computation and memory access locally, while the inter-chip synchronization time penalty term reflects the additional waiting time overhead required between computing cluster units to maintain consistent execution pace, the two are sequentially dependent in the distributed execution process: each computing cluster unit must first complete data reception and synchronization readiness before starting local computation; after local computation is completed, if the results need to be synchronized and summarized again, additional synchronization overhead is required. Therefore, merging the two yields the lower limit of the end-to-end physical time from data distribution readiness to the completion of all computations.

[0040] In some embodiments, if the inter-chip synchronization operation and the computation memory access operation partially overlap in timing, the theoretical lower bound time is still the sum of the initial theoretical time and the inter-chip synchronization time penalty term to ensure that it serves as an absolute benchmark for conservative estimation; if there are multiple inter-chip synchronizations, the time penalty term for each inter-chip synchronization is included in the summation.

[0041] S204: Based on the relative magnitudes of the computationally limited time, the memory access limited time, and the inter-slice synchronization time penalty term, add a bottleneck classification label to the theoretical lower bound time.

[0042] Bottleneck classification labels are discrete identifiers used to identify the main performance limitations of operators under the current hardware microarchitecture. Optionally, bottleneck classification labels include computationally limited, memory-limited, synchronization-limited, computationally and memory-limited balanced, and computationally and synchronously competitive. The computationally limited label indicates that the operator execution time is mainly determined by the peak computing power saturation level; the memory-limited label indicates that the operator execution time is mainly determined by the storage bandwidth supply capacity; the synchronization-limited label indicates that the operator execution time is mainly determined by the inter-chip synchronization waiting overhead; the computationally and memory-limited balanced label indicates that the computationally limited time and memory-limited time are on the same order of magnitude and are both significantly greater than the inter-chip synchronization time penalty, and the operator performance is limited by both computing power and bandwidth; the computationally and synchronously competitive label indicates that the computationally limited time and the inter-chip synchronization time penalty are on the same order of magnitude, and the operator performance is in competition between computational intensity and synchronization waiting.

[0043] In some embodiments, the determination of relative size is based on a preset ratio threshold. Specifically, the maximum value among the three factors—calculation-constrained latency, memory access-constrained latency, and inter-chip synchronization latency penalty—is used as the benchmark. The ratios of the other two factors to the benchmark are then calculated. If the ratio of any factor to the benchmark is greater than or equal to the preset ratio threshold, the factor is determined to be in the same performance constraint dimension as the benchmark. If the ratio is less than the preset ratio threshold, the factor is determined not to constitute the current primary bottleneck. The preset ratio threshold is typically set to a fixed value between 0.5 and 0.8 to distinguish between primary and secondary bottlenecks.

[0044] In some embodiments, the process of attaching bottleneck classification labels includes: First, identifying the maximum value among computation-limited time, memory access-limited time, and inter-segment synchronization time penalty items; Second, if the maximum value is unique and is computation-limited time, and both memory access-limited time and inter-segment synchronization time penalty items are less than a preset proportion threshold for computation-limited time, then attaching a computation-limited label; if the maximum value is unique and is memory access-limited time, and the other two are less than a preset proportion threshold for memory access-limited time, then attaching a memory access-limited label; if the maximum value is unique and is an inter-segment synchronization time penalty item, and the other two are less than a preset proportion threshold for the inter-segment synchronization time penalty item, then attaching a synchronization-limited label; if both computation-limited time and memory access-limited time are greater than or equal to each other's preset proportion thresholds, and the inter-segment synchronization time penalty item is less than both of their preset proportion thresholds, then attaching a computation-memory balance label; if both computation-limited time and inter-segment synchronization time penalty items are greater than or equal to each other's preset proportion thresholds, and the memory access-limited time is less than both of their preset proportion thresholds, then attaching a computation synchronization contention label.

[0045] Preferably, step S300 includes steps S301 and S302.

[0046] S301: According to the target instruction set, call the corresponding target backend template from the preset backend template library; wherein, the backend template library includes backend templates corresponding to various heterogeneous instruction sets, and each backend template has a built-in parameterizable optimization knob.

[0047] The backend template library is a pre-built collection of code generation templates for different heterogeneous instruction sets, used to automatically map the intermediate representation model of operators into low-level code that can be executed on specific hardware instruction sets. Optionally, the backend template library includes CUDA templates, NEON templates, HVX templates, AVX-512 templates, and custom instruction set templates. Each backend template has a built-in parameterizable optimization knob, which refers to a set of configuration parameters used to adjust the values ​​of various compilation optimization strategies during code generation.

[0048] In some embodiments, the target instruction set refers to the set of underlying machine instructions supported by the target chip in the AI ​​inference engine, such as the CUDA instruction set, ARM NEON instruction set, Qualcomm HVX instruction set, or x86 AVX-512 instruction set. The target backend template refers to a code generation template that matches the target instruction set and contains specific instruction encoding rules and register allocation conventions. Optimization knobs include a data partitioning granularity knob, a loop unrolling factor knob, a vectorization width knob, a register allocation strategy knob, and a memory alignment mode knob.

[0049] In some embodiments, the process of calling the corresponding target backend template from a preset backend template library includes: parsing the identifier of the target instruction set, which is specified by the operator developer when submitting the operator generation request, or automatically identified by the AI ​​inference engine based on the model of the currently deployed chip; retrieving the target backend template that matches the identifier in the backend template library; and then loading the default value of the optimization knob built into the target backend template, and exposing the value space of the optimization knob to the subsequent optimization strategy combination determination step.

[0050] S302: Based on the computation graph characteristics of the intermediate representation model of the operator, determine the current optimization strategy combination from the value space of the optimization knob to generate the candidate operator code.

[0051] Among them, computation graph features refer to structured information extracted from the intermediate representation model of the operator, used to characterize the computational structural attributes and resource requirements of the operator. Optionally, computation graph features include tensor dimension features, operation type features, data dependency pattern features, and memory access locality features. Data dependency pattern features refer to the computational association topology between each element in the operator's output tensor and each element in the input tensor, including element-independent, row-dependent, column-dependent, or fully connected types; memory access locality features refer to the access step size and cache line alignment relationship of each dimension of the input tensor during operator execution. Optimization strategy combination refers to a set of specific parameter configurations selected for the current operator from the value space of each optimization knob.

[0052] In some embodiments, the process of determining the current optimization strategy combination from the value space of the optimization knob includes: First, traversing the computation nodes of the intermediate representation model of the operator, extracting tensor dimension features and operation type features, and identifying whether the current operator is computationally intensive or memory-intensive; Second, determining the data partitioning dimension and parallel mapping strategy based on the operation type features and data dependency pattern features. For example, for the matrix multiplication operator, the two-dimensional distribution of the thread grid is determined based on the row and column dimensions of the output tensor; Third, calculating the effective reuse length of the shared memory cache and the register temporary storage requirement based on the tensor dimension features and the memory access locality features, thereby determining the values ​​of the data partitioning granularity knob and the register allocation strategy knob from the value space; Fourth, determining the values ​​of the loop unrolling factor knob and the vectorization width knob based on the computational complexity in the operation type features.

[0053] The value range of the optimization knob is determined by the parameterized configuration of the target backend template.

[0054] In this context, parameterized configuration refers to the set of constraint rules preset by the target backend template during the construction phase for specific instruction set hardware characteristics and compiler toolchain constraints, used to limit the legal value boundaries of each optimization knob. Optionally, parameterized configuration includes hardware resource constraint configuration, instruction set semantic constraint configuration, and compiler frontend constraint configuration. Hardware resource constraint configuration refers to the upper limit of optimization knob values ​​determined based on the upper limit of the target chip's physical resources; instruction set semantic constraint configuration refers to the set of optimization knob values ​​determined based on the instruction encoding rules and operand type restrictions of the target instruction set; compiler frontend constraint configuration refers to the optimization knob value types determined based on the syntax support range of the code generation toolchain.

[0055] In some embodiments, the value space of the optimization knob is not an unlimited continuous range of values, but rather a set of discrete legal values ​​that are clipped during the template initialization phase by the parameterized configuration of the target backend template. When the operator code generation process starts, the target backend template loads its parameterized configuration into the code generation context, and the value space of each optimization knob is then instantiated into a legal subset that strictly matches the target instruction set. Subsequent optimization strategy combination determination steps can only select parameter values ​​within this legal subset.

[0056] Preferably, before determining the current combination of optimization strategies, steps S030 and S031 are also included.

[0057] S030: Based on the hardware topology descriptor of the AI ​​inference engine, determine the cross-chip data splitting axis and inter-chip synchronization strategy of the operator on the target instruction set.

[0058] In one implementation, the cross-chip data partitioning axis refers to a dimension identifier that uses a certain dimension of the operator input or output tensor as a dividing boundary, partitioning the tensor along that dimension into multiple sub-tensors and assigning them to different chips for execution. Optionally, the cross-chip data partitioning axis includes batch dimension partitioning axis, spatial dimension partitioning axis, channel dimension partitioning axis, and feature dimension partitioning axis. Inter-chip synchronization strategy refers to the collaborative control method adopted by each chip after completing local computation to maintain global data consistency or execute subsequent dependent operations. Optionally, inter-chip synchronization strategies include full reduction synchronization strategy, full collection synchronization strategy, point-to-point synchronization strategy, and barrier synchronization strategy.

[0059] In some embodiments, the hardware topology descriptor includes the number of chips, the chip interconnect topology, the inter-chip interconnect bandwidth, and the inter-chip communication latency.

[0060] In some embodiments, the process of determining the cross-chip data splitting axis includes: First, parsing the input tensor dimension and output tensor dimension of the operator intermediate representation model, and identifying the size and data dependency of each dimension; Second, based on the number of chips in the hardware topology descriptor, selecting candidate dimensions that are divisible by the number of chips and do not destroy the mathematical equivalence of the operator after splitting; Third, estimating the amount of inter-chip communication data corresponding to each candidate dimension based on the size of the sub-tensor after splitting each candidate dimension and the interconnection bandwidth between chips; Fourth, selecting the candidate dimension with the smallest amount of inter-chip communication data as the cross-chip data splitting axis.

[0061] In some embodiments, the process of determining the inter-chip synchronization strategy includes: First, based on the computation graph characteristics of the operator intermediate representation model, identifying whether there is cross-chip data dependency or gradient aggregation requirement during operator execution; if the operator is an independent computation node in forward propagation and the outputs of each chip do not need to be merged immediately, then the inter-chip synchronization strategy is determined to be a barrier synchronization strategy, which only needs to perform global step alignment once after each chip completes local computation; if the operator is a gradient aggregation node in backpropagation or needs to aggregate intermediate results across chips, then the inter-chip synchronization strategy is determined to be a full reduction synchronization strategy or a full collection synchronization strategy, so that the local results of each chip are merged into a globally consistent tensor according to the operator semantics.

[0062] S031: While keeping the computation logic of the operator intermediate representation model unchanged, the operator intermediate representation model is reconstructed into a distributed computation graph according to the cross-chip data splitting axis and inter-chip synchronization strategy, and the reconstructed operator intermediate representation model is input into the target backend template to generate candidate operator code that is compatible with the AI ​​inference engine.

[0063] Distributed computation graph reconstruction refers to transforming a computation graph originally designed for execution in contiguous memory on a single node into one designed for collaborative execution across multiple chips, without altering the semantics of the operator's mathematical operations. Optionally, the reconstructed computation graph includes data partitioning nodes, local computation nodes, inter-chip communication nodes, synchronization nodes, and data aggregation nodes. Data partitioning nodes are computation nodes that split the global input tensor into multiple sub-tensors along a cross-chip data partitioning axis and distribute them to each chip. Local computation nodes are computation nodes where each chip executes local variants of the original operator based on its locally held sub-tensors, maintaining the same internal computational logic as the original operator computation nodes. Inter-chip communication nodes are computation nodes that perform data migration between the local storage of different chips. Synchronization nodes are computation nodes that perform pace alignment or state negotiation between chips. Data aggregation nodes are computation nodes that merge the local output sub-tensors generated by each chip into a global output tensor.

[0064] In some embodiments, keeping the computational logic unchanged means that after all the reconstructed distributed computation graphs have been executed, the value of each element of the output tensor is completely consistent with the value of the corresponding element of the output tensor of the original single-node computation graph under the same input. That is, the reconstruction only changes the physical distribution of the computation and the execution timing, without changing the mathematical mapping relationship from input to output. The reconstruction process includes: First, inserting a data splitting node after the input tensor node representing the model in the middle of the operator, mapping the global input tensor into multiple sub-tensors according to the cross-chip data splitting axis; Second, copying the original operator computation node into multiple local computation nodes, each local computation node is bound to a chip and receives the sub-tensors output by the corresponding data splitting node; Third, inserting corresponding synchronization nodes between or after the local computation nodes according to the inter-chip synchronization strategy. If the inter-chip synchronization strategy is a full reduction synchronization strategy, a full reduction synchronization node is inserted after the local computation nodes, so that the local results of each chip are merged into a globally consistent result according to the operator semantics; if the inter-chip synchronization strategy is a barrier synchronization strategy, a barrier synchronization node is inserted after the local computation nodes, requiring only that each chip confirms that it is ready after completing the local computation; Fifth, inserting a data aggregation node after the synchronization node, concatenating the local output sub-tensors of each chip into a global output tensor according to the inverse operation of the splitting axis.

[0065] Preferably, step S400 includes steps S401-S404.

[0066] S401: The measured results include the measured output results and the measured time.

[0067] S402: The accuracy of the measured output result is evaluated based on the output of the operator gold reference module to obtain the accuracy evaluation index.

[0068] The operator gold reference module refers to an operator reference implementation independently implemented in a high-precision numerical format on a general-purpose computing platform. It provides a benchmark output result that is unaffected by the hardware precision limitations and optimization strategies of the target chip. Optionally, the operator gold reference module performs the same mathematical operations as the candidate operator code in double-precision floating-point or high-precision fixed-point format, and its output result serves as the absolute benchmark for judging the numerical correctness of the candidate operator code.

[0069] In some embodiments, accuracy evaluation metrics include absolute error metrics, relative error metrics, element-level maximum deviation metrics, and tensor-level structural similarity metrics.

[0070] In some embodiments, the process of accuracy evaluation includes: First, aligning the measured output results with the output tensor dimensions of the operator's golden reference module to ensure that their shapes, data types, and element arrangement order are consistent; Second, based on the actual numerical execution format of the candidate operator code on the target chip, reducing the precision of the output tensor of the operator's golden reference module to the same data format as the measured output results; wherein, the data format includes double-precision floating-point format, single-precision floating-point format, half-precision floating-point format, unidirectional floating-point format, or fixed-point format; Third, calculating the deviation between the measured output results and the reduced-precision golden reference output element by element; and obtaining the accuracy evaluation index from the element-by-element deviation according to the preset accuracy evaluation index type.

[0071] S403: The measured time is evaluated based on the theoretical lower bound time to obtain performance evaluation indicators; the performance evaluation indicators include the deviation between the measured time and the theoretical lower bound time.

[0072] Optionally, the measured execution time is collected through the hardware performance counters or driver-level event logging mechanisms built into the target chip. The collection granularity includes operator-level end-to-end execution time, kernel-level execution time, or thread-level scheduling time. Deviation refers to the relative proportion of the measured execution time exceeding the theoretical lower bound, used to quantify the gap between the implementation efficiency of the candidate operator code and the physical limits of the hardware.

[0073] S404: Generate the evaluation result based on the accuracy evaluation index and the performance evaluation index.

[0074] The evaluation results refer to structured judgment information that comprehensively characterizes whether candidate operator code meets the inclusion criteria in both numerical correctness and execution efficiency. Optionally, the evaluation results include accuracy evaluation status, performance evaluation status, comprehensive judgment status, accuracy deviation summary, performance deviation summary, and optimization suggestion summary. Accuracy evaluation status refers to the numerical correctness judgment of the candidate operator code's measured output result relative to the output of the operator's golden reference module; performance evaluation status refers to the efficiency judgment of the candidate operator code's measured time consumption relative to the theoretical lower bound time consumption; comprehensive judgment status refers to the final inclusion judgment obtained from the joint logical operation of accuracy evaluation status and performance evaluation status; accuracy deviation summary refers to the specific deviation type and location information recorded when the accuracy evaluation status is not passed; performance deviation summary refers to the deviation value and bottleneck classification information recorded when the performance evaluation status is not passed; optimization suggestion summary refers to the targeted optimization hints generated based on the performance deviation summary and bottleneck classification labels, which are read by the reimplementation strategy engine.

[0075] In some embodiments, the process of generating evaluation results includes: First, comparing the relative error index in the accuracy evaluation metrics with a preset accuracy threshold. If the relative error index is less than the preset accuracy threshold and the maximum deviation index at the element level is less than a preset peak deviation threshold, the accuracy evaluation status is marked as passed; otherwise, it is marked as failed, and the accuracy deviation summary is recorded as a structured list containing the coordinates of the failed element, the measured value, the reference value, and the absolute error. Second, comparing the deviation in the performance evaluation metrics with a preset deviation threshold. If the deviation is less than or equal to the preset deviation threshold, the performance evaluation status is marked as passed; otherwise, it is marked as failed. The performance deviation summary is recorded as a structured dictionary containing the deviation percentage, theoretical lower bound time, measured time, and bottleneck classification label. Then, a logical AND operation is performed on the accuracy evaluation status and performance evaluation status. If both pass, the overall judgment status is marked as meeting the preset conditions; otherwise, it is marked as not meeting the preset conditions. Finally, if the overall judgment status is not meeting the preset conditions, an optimization suggestion summary is generated based on the bottleneck classification label in the performance deviation summary. For example, when the bottleneck classification label is computationally constrained and the deviation is greater than the preset deviation threshold, the optimization suggestion summary indicates that the optimization strategy combination of data splitting dimension and instruction parallelism dimension should be adjusted first.

[0076] Preferably, the evaluation includes steps S410-S412.

[0077] S410: Obtain the measured output results and hardware counter data obtained from multiple rounds of execution on the target chip.

[0078] In some embodiments, hardware counter data includes clock cycle counts, instruction issue counts, cache access counts, memory transaction counts, and compute unit utilization counts. Clock cycle counts refer to the total number of internal clock cycles consumed by the target chip from operator core startup to execution completion; instruction issue counts refer to the total number of instructions successfully issued to the execution pipeline during operator execution; cache access counts refer to the number of cache hits and misses at each level; memory transaction counts refer to the number of data transfer transactions completed between the target chip and external memory; and compute unit utilization counts refer to the percentage of time the compute unit is in an active state. Multi-round execution refers to continuously executing the candidate operator code on the target chip a preset number of times.

[0079] S411: Identify abnormal execution rounds during the execution process based on the hardware counter data, and remove the entire round of measured output results corresponding to the abnormal execution rounds in the accuracy evaluation.

[0080] Among them, abnormal execution rounds refer to individual execution rounds in which the hardware counter data of the candidate operator code deviates significantly from the normal statistical distribution due to interference from factors such as operating system scheduling preemption, dynamic frequency adjustment, heat dissipation power limitation, or bus resource contention during multiple rounds of repeated execution. Complete round removal means completely removing the measured output results corresponding to abnormal execution rounds from the comparison samples for accuracy evaluation, excluding them from subsequent element-by-element deviation calculations with the operator's golden reference module output.

[0081] In some embodiments, the process of identifying abnormal execution rounds includes: extracting clock cycle counts from hardware counter data corresponding to multiple execution rounds to construct a clock cycle sequence; calculating the statistical median and absolute median difference of the clock cycle sequence; determining the upper and lower boundaries of the normal fluctuation range based on a preset abnormality judgment threshold (typically 2 to 3 times the absolute median difference); and marking rounds exceeding the normal fluctuation boundaries as abnormal execution rounds. The absolute median difference refers to the median of the absolute values ​​of the differences between the clock cycle counts and the statistical median for each round.

[0082] In some embodiments, the process of performing full-round elimination in accuracy evaluation includes: establishing an abnormal round index list based on the aforementioned marking results; deleting the measured output tensor corresponding to the abnormal round index from the multi-round measured output result set before comparing the measured output results with the output of the operator's golden reference module element by element; and then retaining only the measured output parameter of the normal execution round.

[0083] S412: Compare the accuracy of the statistical median of the measured output results of the remaining rounds after elimination with the output of the golden reference module of the operator, and compare the performance of the statistical mean of the measured time consumption of the remaining rounds with the theoretical lower bound time consumption.

[0084] The statistical median is the value at the middle position after sorting the remaining test outputs from multiple rounds (excluding abnormal rounds) according to their corresponding element values. It is used to characterize the typical numerical performance of candidate operator code output under stable hardware conditions. The statistical mean is the arithmetic average of the test execution times from multiple rounds (excluding abnormal rounds). It is used to characterize the typical execution time of candidate operator code under stable hardware conditions.

[0085] In some embodiments, the process of calculating the statistical median includes: for each element position of the output tensor, collecting the measured output values ​​at that position for each remaining round after removing outliers, forming a numerical sequence for that position; sorting the numerical sequence in ascending order; if the number of remaining rounds is odd, taking the value at the middle position after sorting as the statistical median for that position; if the number of remaining rounds is even, taking the arithmetic mean of the two middle values ​​after sorting as the statistical median for that position; and then reorganizing the statistical medians of all element positions according to the original tensor shape to obtain the statistical median tensor. Optionally, when the total number of elements in the output tensor exceeds a preset sampling threshold, a hierarchical random sampling strategy is adopted, performing position-by-position median calculation only on a preset proportion of element positions, and using the statistical mean for the remaining element positions.

[0086] Preferably, the step of returning candidate operator codes that do not meet the preset conditions for optimization includes the reimplementation strategy engine jointly judging the accuracy evaluation and the performance evaluation, and generating a reimplementation decision instruction, including steps S501-S503.

[0087] S501: If the candidate operator fails the accuracy evaluation, a first reimplementation decision instruction is generated to switch the implementation strategy in the strategy space according to the accuracy deviation information in the evaluation result, and to re-trigger the target instruction set code generation and evaluation.

[0088] The first re-realization decision instruction refers to the control instruction output by the re-realization strategy engine when it determines that the numerical correctness of the candidate operator code does not meet the standard. This instruction is used to terminate the current optimization path and initiate an alternative realization path. Optionally, the first re-realization decision instruction includes an accuracy deviation type identifier, deviation location distribution characteristics, current realization strategy identifier, and target switching strategy identifier. Accuracy deviation information refers to the specific numerical deviation details recorded in the evaluation results, including the coordinates of the deviation element, the measured value, the reference value, and the absolute error.

[0089] In some embodiments, the deviation types in the precision deviation information include cumulative error, truncation error, denormalized number processing difference, and instruction reordering error. Cumulative error refers to error amplification caused by different accumulation orders of intermediate results from a large number of operations; truncation error refers to the loss of low-order information due to insufficient data type bit width or improper rounding mode; denormalized number processing difference refers to deviation caused by inconsistencies between the target chip's processing behavior for subnormal floating-point numbers and the operator's golden reference module; instruction reordering error refers to deviation caused by the compiler or backend template reordering the order of operation instructions to improve performance, thus violating the original mathematical associative law. Switching implementation strategies refers to replacing at least one optimization knob value in the current optimization strategy combination with another valid value.

[0090] In some embodiments, the process of generating the first reimplementation decision instruction includes: parsing the accuracy deviation summary in the accuracy evaluation results and extracting a multi-dimensional deviation feature vector; the deviation feature vector includes the error magnitude distribution, error spatial distribution pattern, correlation between error and data position, correlation between error and thread index, and stability of error and operation rounds; further, the main deviation types are identified by comprehensively analyzing the deviation feature vectors. If the deviation elements are concentrated in the edge region of the tensor and the error magnitude increases with the accumulation length, it is inferred to be the reduction accumulation error at the data split boundary; if the deviation elements are randomly distributed and the error magnitude is related to the operation order, it is inferred to be the operation order rearrangement error caused by instruction-level parallelism; if the deviation elements are concentrated in a specific numerical range and coincide with the range of the subnormal number. If the deviation is strong and the error is strongly correlated with the thread index, it is inferred to be a non-normalized number processing difference error. If the deviation element shows a striped distribution, it is inferred to be a read error caused by asynchronous memory contention or shared memory conflict. Then, according to the identified deviation type, the corresponding alternative implementation strategy is retrieved in the policy space. For example, for instruction reordering error, the loop unrolling factor in the current optimization strategy combination is switched from 4 to 1 to disable instruction reordering optimization. For non-normalized number processing difference error, the numerical flush-to-zero mode in the current optimization strategy combination is switched to the disabled state. The current implementation strategy identifier and the target switching strategy identifier are encapsulated into a first reimplementation decision instruction, and the code generation process from the operator intermediate representation model to the target instruction set and the subsequent evaluation process are triggered.

[0091] S502: If the accuracy of the candidate operator is acceptable but the measured time deviates from the theoretical lower bound by more than a preset threshold, a second reimplementation decision instruction is generated to adjust and optimize the strategy combination in the strategy space according to the bottleneck classification label carried by the theoretical lower bound time, and the candidate operator code is regenerated.

[0092] Optionally, the second reimplementation decision instruction includes the current deviation value, bottleneck classification label, dimension identifier of the optimization strategy to be adjusted, adjustment direction identifier, and adjustment step size. The preset threshold refers to the upper limit boundary value of the deviation for determining whether the measured time is acceptable, usually set to 10% to 30%. Targeted adjustment refers to modifying the parameter values ​​of the corresponding optimization strategy dimension in the strategy space based on the performance-limited dimension indicated by the bottleneck classification label, rather than randomly traversing or globally resetting all optimization knobs.

[0093] In some embodiments, the process of generating a second reimplementation decision instruction includes: reading the performance evaluation status in the evaluation results, confirming that the accuracy evaluation status is passed and the performance evaluation status is failed; extracting the deviation value and bottleneck classification label from the performance deviation summary, comparing the deviation value with a preset threshold, and if the deviation is greater than the preset threshold, triggering the generation of a second reimplementation decision instruction; further, querying a preset optimization strategy dimension mapping table according to the bottleneck classification label to determine the optimization strategy dimension and adjustment direction that need to be adjusted.

[0094] In some embodiments, the optimization strategy dimension mapping table is a structured configuration pre-built in the reimplementation strategy engine that records the correspondence between each bottleneck category label and the optimization strategy dimension to be adjusted. Specifically, when the bottleneck category label is computationally constrained, the mapping table indicates that the optimization strategy dimensions to be adjusted include data partitioning dimension and instruction parallelism dimension, and the adjustment direction is to increase the computation density within a single thread block and improve instruction issue saturation; when the bottleneck category label is memory access constrained, the mapping table indicates that the optimization strategy dimensions to be adjusted include storage access dimension and data partitioning dimension, and the adjustment direction is to improve cache reuse rate and reduce the proportion of non-merged memory access; when the bottleneck category label is synchronization constrained, the mapping table indicates that the optimization strategy dimensions to be adjusted include synchronization control dimension and data partitioning dimension, and the adjustment direction is to reduce synchronization frequency or use asynchronous pipelines to mask synchronization latency; when the bottleneck category label is computation synchronization contention, the mapping table indicates that the optimization strategy dimensions to be adjusted include synchronization control dimension and instruction parallelism dimension, and the adjustment direction is to rebalance resource allocation between computation density and synchronization overhead.

[0095] S503: If the preset conditions cannot be met after N consecutive iterations, a termination decision instruction is generated, and the current best candidate is marked as suboptimal and available.

[0096] Wherein, the policy space is the set of all optimization policy combinations that generate candidate operator codes; N is an integer from 3 to 20.

[0097] The termination decision instruction is a control command output by the reimplementation strategy engine after reaching the preset maximum number of iterations. It terminates the current operator development and optimization loop and transitions to the downgraded storage process. Optionally, the termination decision instruction includes the cumulative number of iterations, the deviation sequence for each iteration, the optimal candidate iteration identifier, and a suboptimal availability flag. Suboptimal availability means that although the candidate operator code has not brought the performance evaluation metrics within the preset deviation threshold, its deviation is the smallest among the explored optimization strategy combinations, and its accuracy evaluation continues to pass. Therefore, it is allowed to be stored but a downgrade flag is attached. N refers to the maximum number of reimplementation iterations allowed by the reimplementation strategy engine for the same operator intermediate representation model and the same target instruction set. This is used to prevent infinite loop optimization caused by local optimum traps or insurmountable hardware physical limits in the strategy space. The strategy space refers to the set of optimization strategy combinations that generate candidate operator code, and each optimization strategy combination corresponds to a complete set of code generation parameter configurations.

[0098] In some embodiments, the process of determining that the preset conditions cannot be met even after N consecutive iterations includes: maintaining an iteration counter, incrementing the iteration counter by one after each generation of a first reimplementation decision instruction or a second reimplementation decision instruction and re-triggering code generation and evaluation; checking the comprehensive judgment status after each evaluation, and if the comprehensive judgment status does not meet the preset conditions, recording the deviation and accuracy evaluation status of the current round; and terminating the current optimization loop and no longer triggering a new code generation process when the iteration counter reaches the preset N value.

[0099] In some embodiments, the process of marking the current best candidate as suboptimal available includes: backtracking all candidate operator codes in the N iterations that have a comprehensive judgment state that does not meet the preset conditions but a precision evaluation state that has passed; comparing the deviation of each passed candidate and selecting the one with the smallest deviation as the current best candidate; attaching a suboptimal available flag to the best candidate, the flag recording its deviation value, the identifier of the optimization strategy combination used to reach the deviation, and the specific dimension that does not meet the preset conditions; outputting the suboptimal available candidate to the operator library, but placing its priority after the candidates that meet the preset conditions, and calling it only when there are no better candidates available during operator library scheduling.

[0100] Preferably, the reimplementation strategy engine adjusts the optimization strategy dimension corresponding to the bottleneck classification label in the strategy space according to the second reimplementation decision instruction, and regenerates candidate operator code; wherein, the optimization strategy dimension includes data partitioning dimension, storage access dimension, instruction parallelism dimension, synchronization control dimension, register allocation dimension, tensor memory layout dimension and numerical precision dimension, and the activation state of each dimension in the strategy space is determined by the bottleneck classification label through a preset mapping relationship table.

[0101] Among them, the data partitioning dimension refers to the set of parameters that control the granularity, partitioning order, and sub-tensor shape matching method of input or output tensors between various computational units; the storage access dimension refers to the set of parameters that control the data transport path, cache reuse strategy, memory alignment method, and load instruction width between different storage levels; the instruction parallelism dimension refers to the set of parameters that control the number of concurrent independent computation instructions, loop unrolling depth, and pipeline fill density in a single instruction stream; the synchronization control dimension refers to the set of parameters that control the timing of synchronization alignment between multiple computational units, the type of synchronization primitive, and the synchronization frequency; and the register allocation dimension refers to the set of parameters that control the upper limit of the number of registers allocated by the compiler to a single thread and the register overflow handling strategy.

[0102] In some embodiments, a pre-defined mapping table between bottleneck classification labels and optimization strategy dimensions is implemented in the strategy engine. When the bottleneck classification label is computationally constrained, the mapping table indicates the activation of the data partitioning dimension and the instruction parallelism dimension to alleviate the bottleneck by increasing the computation density within a single thread block and improving instruction issue saturation. When the bottleneck classification label is memory access constrained, the mapping table indicates the activation of the memory access dimension and the data partitioning dimension to alleviate the bottleneck by improving cache line reuse and optimizing memory access patterns. When the bottleneck classification label is synchronization constrained, the mapping table indicates the activation of the synchronization control dimension and the data partitioning dimension to reduce synchronization frequency or mask synchronization latency using asynchronous pipelines. When the bottleneck classification label is computational synchronization contention, the mapping table indicates the simultaneous activation of the instruction parallelism dimension, the synchronization control dimension, and the register allocation dimension to rebalance resource allocation between computational density and synchronization overhead.

[0103] Preferably, the present invention also provides an apparatus for developing operators based on an AI inference engine, which stores a computer program. The apparatus includes at least an AI inference engine, which includes a main control chip and a computing module composed of several ClusterUnits. Each ClusterUnit includes at least one AI processing chip and is hard-connected via a high-speed inter-chip interconnect bus. The apparatus also includes a processor and a memory, which is coupled to the processor. The memory is used to store computer program code, which includes computer instructions. When the processor reads the computer instructions from the memory, the apparatus executes the method described above.

[0104] It is important to note that, in terms of deployment, the method and apparatus can run independently on an offline host machine as a software toolchain, or they can be integrated into the AI ​​inference engine's SDK / CI toolchain and provided externally. For example... Figure 2As shown, in terms of hardware architecture, the offline host and AI inference engine work collaboratively through an encrypted communication link: the offline host undertakes heavy offline computing tasks such as operator modeling, theoretical time calculation, cross-instruction set code generation, and evaluation strategy decision-making; the AI ​​inference engine includes a main control chip and a computing module composed of several Cluster Units. Each Cluster Unit contains at least one AI processing chip and is hard-connected through a high-speed inter-board interconnect bus. The main control chip receives candidate operator code from the offline host via the encrypted communication link and controls the AI ​​processing chip to execute it. After execution, the AI ​​processing chip sends the test results back to the offline host via the same encrypted communication link. This architecture utilizes the general computing resources of the offline host to complete complex compilation optimization while ensuring that the actual execution of operators occurs in a real hardware environment, guaranteeing the consistency between the evaluation conclusions and the final deployment environment. It should be understood that the above dual-sided deployment architecture and functional division are merely exemplary descriptions of the present invention and are not intended to limit the scope of protection of the present invention; those skilled in the art can make adaptive adjustments to the deployment location, communication link form, and interaction method of each module according to actual needs, and this application does not impose any restrictions in this regard.

[0105] Therefore, the embodiments should be considered as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0106] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for developing operators based on AI inference engines, characterized in that, Includes the following steps: Obtain the microarchitecture description table of the operator golden reference module and the target chip, and generate the operator intermediate representation model based on the operator golden reference module; Based on the operator intermediate representation model and the microarchitecture description table, calculate the theoretical lower bound time consumption of the operator on the target instruction set; Based on the operator intermediate representation model and the target instruction set, candidate operator codes are generated; The candidate operator code is deployed to the AI ​​inference engine for execution, and the test results are obtained. The evaluation is then performed based on the test results, the time consumption of the operator's golden reference module, and the theoretical lower bound. Based on the evaluation results, candidate operator codes that meet the preset conditions are output to the operator library, or candidate operator codes that do not meet the preset conditions are returned for optimization.

2. The method for developing operators based on AI inference engines according to claim 1, characterized in that, Before calculating the time taken for the theoretical lower bound, the following steps are also included: Based on the hardware topology descriptor of the AI ​​inference engine, determine the inter-chip data transfer amount and inter-chip synchronization overhead of the operator executed across computing cluster units on the target instruction set; Based on the inter-chip data transfer volume and inter-chip synchronization overhead, the operator computation volume and data transfer volume corresponding to the operator intermediate representation model are corrected to obtain the corrected operator computation volume and data transfer volume.

3. The method for developing operators based on AI inference engines according to claim 2, characterized in that, Based on the operator intermediate representation model and the microarchitecture description table, the theoretical lower bound time consumption of the operator on the target instruction set is calculated, including: Based on the corrected operator computation and data transfer volume, and combined with the peak computing power and storage bandwidth in the microarchitecture description table, the initial theoretical latency is calculated based on the Roofline model; wherein, the initial theoretical latency is the larger of the computation-limited latency and the memory access-limited latency, the computation-limited latency is obtained by dividing the corrected operator computation by the peak computing power, and the memory access-limited latency is obtained by dividing the corrected data transfer volume by the storage bandwidth; Calculate the inter-chip synchronization time penalty term based on the inter-chip synchronization overhead and the typical synchronization primitive overhead in the microarchitecture description table; The initial theoretical time consumption is combined with the inter-chip synchronization time consumption penalty term to obtain the theoretical lower bound time consumption; Based on the relative magnitudes of the computationally limited time, the memory access limited time, and the inter-slice synchronization time penalty term, a bottleneck classification label is added to the theoretical lower bound time.

4. The method for developing operators based on AI inference engines according to claim 1, characterized in that, The step of generating candidate operator code based on the operator intermediate representation model and the target instruction set includes: According to the target instruction set, the corresponding target backend template is called from the preset backend template library; wherein, the backend template library includes backend templates corresponding to various heterogeneous instruction sets, and each backend template has a built-in parameterizable optimization knob; Based on the computation graph characteristics of the intermediate representation model of the operator, the current combination of optimization strategies is determined from the value space of the optimization knob to generate the candidate operator code; The value range of the optimization knob is determined by the parameterized configuration of the target backend template.

5. The method for developing operators based on AI inference engines according to claim 4, characterized in that, Before determining the current combination of optimization strategies, the following steps are also included: Based on the hardware topology descriptor of the AI ​​inference engine, determine the cross-chip data splitting axis and inter-chip synchronization strategy of the operator on the target instruction set; While keeping the computational logic of the operator intermediate representation model unchanged, the operator intermediate representation model is reconstructed into a distributed computation graph according to the cross-chip data splitting axis and inter-chip synchronization strategy, and the reconstructed operator intermediate representation model is input into the target backend template to generate candidate operator code adapted to the AI ​​inference engine.

6. The method for developing operators based on AI inference engines according to claim 1, characterized in that, The evaluation is based on the measured results, the time consumption of the operator's golden reference module, and the theoretical lower bound. include: The measured results include the measured output and the measured time. The accuracy of the measured output result is evaluated based on the output of the operator gold reference module to obtain the accuracy evaluation index. The measured time is evaluated based on the theoretical lower bound time to obtain performance evaluation indicators; the performance evaluation indicators include the deviation between the measured time and the theoretical lower bound time. The evaluation results are generated based on the accuracy evaluation indicators and the performance evaluation indicators.

7. The method for developing operators based on AI inference engines according to claim 6, characterized in that, The evaluation includes: Obtain the measured output results and hardware counter data obtained from multiple rounds of execution on the target chip; Based on the hardware counter data, abnormal execution rounds are identified during the execution process, and the measured output results corresponding to the abnormal execution rounds are completely removed in the accuracy evaluation. The accuracy is compared between the statistical median of the measured output results of the remaining rounds after elimination and the output of the golden reference module of the operator. At the same time, the performance is compared between the statistical mean of the measured time consumption of the remaining rounds and the theoretical lower bound time consumption.

8. The method for developing operators based on AI inference engines according to claim 1, characterized in that, The step of returning candidate operator codes that do not meet the preset conditions for optimization includes the reimplementation strategy engine jointly determining the accuracy evaluation and the performance evaluation, and generating reimplementation decision instructions, including: If the candidate operator fails the accuracy evaluation, a first reimplementation decision instruction is generated to switch the implementation strategy in the strategy space based on the accuracy deviation information in the evaluation result, and to re-trigger the target instruction set code generation and evaluation. If the accuracy of the candidate operator is acceptable but the measured time deviates from the theoretical lower bound by more than a preset threshold, a second reimplementation decision instruction is generated to adjust and optimize the strategy combination in the strategy space according to the bottleneck classification label carried by the theoretical lower bound time, and to regenerate the candidate operator code. If the preset conditions cannot be met after N consecutive iterations, a termination decision instruction is generated, and the current best candidate is marked as suboptimal and available. Wherein, the policy space is the set of all optimization policy combinations that generate candidate operator codes; N is an integer from 3 to 20.

9. The method for developing operators based on AI inference engines according to claim 8, characterized in that, The reimplementation strategy engine adjusts the optimization strategy dimensions corresponding to the bottleneck classification labels in the strategy space according to the second reimplementation decision instruction, and regenerates candidate operator code; wherein, the optimization strategy dimensions include data partitioning dimension, storage access dimension, instruction parallelism dimension, synchronization control dimension, register allocation dimension, tensor memory layout dimension and numerical precision dimension, and the activation state of each dimension in the strategy space is determined by the bottleneck classification labels through a preset mapping relationship table.

10. A device for developing operators based on AI inference engines, characterized in that, The device includes at least an AI inference engine, which includes a main control chip and a computing module composed of several cluster units. Each cluster unit includes at least one AI processing chip and is hard-connected via a high-speed inter-chip interconnect bus. The device also includes a processor and a memory, which is coupled to the processor. The memory is used to store computer program code, which includes computer instructions. When the processor reads the computer instructions from the memory, the device executes the method described above.

Images