Ai accelerator construction method and related apparatus

Abstract

The application relates to the field of artificial intelligence, in particular to an AI accelerator construction method and related device. The method comprises the following steps: importing machine learning models of different AI frameworks through a multi-framework adaptation interface, and converting the machine learning models into linear algebra dialect representations by using an MLIR dialect conversion chain; performing target-independent optimization processing, retaining core calculation semantics in the linear algebra dialect, and reconstructing the linear algebra dialect representations into optimized intermediate representations with efficient data flow structures; identifying target hardware characteristics of a target hardware; adaptively performing hardware optimization processing matched with the target hardware characteristics on the optimized intermediate representations; converting the target intermediate representations into hardware operation units executable by the target hardware, loading the hardware operation units into the target hardware to obtain an AI accelerator, so that the hardware operation units are run on the target hardware through the AI accelerator, the corresponding calculation resources in the target hardware are scheduled, and hardware acceleration of the machine learning models is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an AI accelerator construction method and related device. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the complexity of machine learning models increases exponentially, and the demand for computing resources also increases rapidly. This trend has driven the rapid development and popularization of special hardware accelerators, such as graphics processing units (GPUs), tensor processing units (TPUs), and field programmable gate arrays (FPGAs). Such hardware, with its high parallelism and high computing power density, has become the core infrastructure supporting the efficient operation of complex models. Currently, AI accelerator construction and adaptation schemes in the prior art are mainly implemented using XLA, TVM, and IREE. However, existing technical solutions are usually deeply bound to a single machine learning framework, making it difficult to achieve unified import and compilation of models in multiple frameworks such as TensorFlow, PyTorch, and JAX, resulting in the need for additional development of adaptation tools for cross-framework deployment, increasing development costs and deployment complexity. The existing technical solutions lack coordination in the intermediate representation (IR) transformation and optimization stages, making it difficult to form a coherent optimization chain from high-level models to hardware instructions, and unable to implement fine-grained optimization based on hardware characteristics such as parallelism and memory hierarchy. In addition, the existing technical solutions lack a tensor shape adjustment mechanism for hardware characteristics, which cannot efficiently adapt to reduce additional computational overhead, and lack efficient allocation and reuse strategies for on-chip memory, resulting in low utilization of parallel resources and inability to fully utilize the parallel computing capabilities of hardware. In summary, there is an urgent need for a technical solution to overcome at least one of the technical problems of high model deployment complexity, inability to fine-tune, low adaptation efficiency, and low parallel resource utilization in the prior art. SUMMARY

[0003] The present application addresses the technical problems in the prior art and provides an AI accelerator construction method and related device to solve the technical problems of high model deployment complexity, inability to fine-tune, low adaptation efficiency, and low parallel resource utilization in related technologies.

[0004] In a first aspect, an AI accelerator construction method is provided, which includes:

[0005] Importing machine learning models of different AI frameworks through a multi-framework adaptation interface, parsing the model representation of the machine learning model, and converting the model representation into a formally unified linear algebra dialect representation using an MLIR dialect conversion chain;

[0006] In the target-independent optimization stage, through target-independent optimization processing, core computing semantics in the linear algebra dialect are retained, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization;

[0007] In the hardware adaptive optimization stage, target hardware characteristics of the target hardware are identified, the target hardware characteristics including: hardware type, hardware structure, and parallelism degree;

[0008] The optimized intermediate representation is adaptively executed with hardware optimization processing matched with the target hardware characteristics to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion;

[0009] The target intermediate representation is converted into a hardware operation unit executable by the target hardware, and the hardware operation unit is loaded into the target hardware to construct an AI accelerator;

[0010] In the AI accelerator, the hardware operation unit is run on the target hardware through a hardware abstraction layer dialect to link the IR at the compilation time and the runtime, and the corresponding computing resources in the target hardware are scheduled to complete hardware acceleration of the machine learning model.

[0011] In a second aspect, an AI accelerator construction device is provided, and the device includes the following units, wherein,

[0012] The import unit is configured to import a machine learning model of different AI frameworks through a multi-framework adaptation interface, parse a model representation of the machine learning model, and convert the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain;

[0013] The optimization unit is configured to, in a target-independent optimization stage, retain core computing semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; in a hardware adaptive optimization stage, target hardware characteristics of the target hardware are identified, the target hardware characteristics including: hardware type, hardware structure, and parallelism degree; and the optimized intermediate representation is adaptively executed with hardware optimization processing matched with the target hardware characteristics to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion;

[0014] a building unit configured to convert the target intermediate representation into hardware operation units executable by the target hardware, load the hardware operation units into the target hardware, and build an AI accelerator;

[0015] the AI accelerator configured to bridge the compile-time IR and the runtime by a hardware abstraction layer dialect to run the hardware operation units on the target hardware, schedule corresponding computing resources in the target hardware, and complete hardware acceleration of the machine learning model.

[0016] In a third aspect, an electronic device is provided, and the electronic device includes:

[0017] at least one processor, a memory, and an input-output unit;

[0018] The memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to execute the AI accelerator building method of the first aspect.

[0019] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to execute the AI accelerator building method of the first aspect.

[0020] The AI accelerator building method and related devices are provided. In the technical solution, first, a machine learning model of different AI frameworks is imported through a multi-framework adaptation interface, a model representation of the machine learning model is parsed, and the model representation is converted into a linear algebra dialect representation with a formal uniform by using an MLIR dialect conversion chain. Then, in a target-independent optimization stage, core computing semantics in the linear algebra dialect are retained by target-independent optimization processing, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure. The target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization. Then, in a hardware adaptive optimization stage, target hardware characteristics of the target hardware are identified, the target hardware characteristics include a hardware type, a hardware structure, and a parallelism degree. The optimized intermediate representation is adaptively executed by hardware optimization processing matched with the target hardware characteristics to obtain a target intermediate representation. The hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion. Finally, the target intermediate representation is converted into hardware operation units executable by the target hardware, the hardware operation units are loaded into the target hardware, and an AI accelerator is built. In the AI accelerator, the hardware operation units are run on the target hardware by a hardware abstraction layer dialect to bridge the compile-time IR and the runtime, corresponding computing resources in the target hardware are scheduled, and hardware acceleration of the machine learning model is completed.

[0021] The technical scheme of the present application, through the target-independent optimization stage processing and adaptive hardware optimization, constructs an AI accelerator containing personalized hardware operation units for machine learning models of different AI architectures, thereby realizing flexible allocation of computing resources in the target hardware through the AI accelerator, effectively improving the model deployment efficiency and running efficiency, improving the adaptation of machine learning models and hardware resources, helping fine optimization from machine learning models to hardware resources, and simultaneously improving the parallel resource utilization rate, avoiding waste of accelerator resources. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flow diagram of an AI accelerator construction method according to an embodiment of the present application;

[0023] Figure 2 is a structural diagram of an AI accelerator construction device according to an embodiment of the present application;

[0024] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present application;

[0025] Figure 4 is a structural diagram of a medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] An AI accelerator is a hardware unit specially designed for artificial intelligence applications, aiming to accelerate the execution of artificial intelligence algorithms and improve computing efficiency and performance. AI accelerators usually adopt a special hardware architecture and instruction set, which is optimized for the characteristics of artificial intelligence algorithms to realize efficient parallel computing.

[0028] Machine learning models, such as deep learning neural networks, are becoming increasingly complex, with an exponential increase in demand for computing resources. For example, the large number of matrix multiplications, Reduction operations, and vector transformations in the model, general-purpose CPUs are difficult to efficiently adapt due to the generalization of architecture (need to consider various tasks), resulting in low execution efficiency and high energy consumption. In this context, AI accelerators have emerged, with customized structures at the hardware level to complete the core computing needs of AI tasks, becoming the key hardware foundation to support complex AI models (such as large language model inference and computer vision training). Currently, existing AI accelerator construction and adaptation schemes mainly use XLA, TVM, and IREE to achieve this. However, this approach has the following technical problems:

[0029] First, existing technical solutions usually deeply bind a single machine learning framework, making it difficult to implement unified import and compilation of multiple framework models such as TensorFlow, PyTorch, and JAX, resulting in the need for additional development of adaptation tools for cross-framework deployment, increasing development costs and deployment complexity. At the same time, existing technical solutions do not fully support emerging customized hardware such as domestic TPU, lacking complete software support solutions for such hardware characteristics, and failing to fully utilize the computing advantages of customized hardware, resulting in ineffective release of hardware computing power.

[0030] Second, the intermediate representation (IR) transformation and optimization steps in existing technical solutions are loosely connected, with target-independent optimization (such as improving code generality and reducing unnecessary recalculation) and target-specific optimization (such as adapting hardware parallelism and memory hierarchy) lacking coordination, making it difficult to form a coherent optimization chain from high-level models to hardware instructions, and failing to achieve fine-grained optimization for hardware parallelism, memory hierarchy structure, and other characteristics.

[0031] Third, existing technical solutions lack a tensor shape adjustment mechanism for hardware characteristics. For tensor shapes that do not meet the hardware computing definition (such as memory transfer, fixed vector length, and other limitations), it is difficult to efficiently adapt to reduce additional computational overhead. Additionally, there is a lack of flat processing of high-dimensional memory access, efficient allocation and reuse strategies for on-chip memory, and optimization of multi-device collaboration (such as data interaction between different hardware) and asynchronous task scheduling, resulting in low utilization of parallel resources and failing to fully utilize the parallel computing capabilities of hardware.

[0032] To solve at least one technical problem in the prior art, the embodiments of the present application provide an AI accelerator construction method and related devices. In the technical solution provided by the present application, through target-independent optimization stage processing and adaptive hardware optimization, an AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures, so that the flexible allocation of computing resources in the target hardware is realized through the AI accelerator, the model deployment efficiency and the running efficiency are effectively improved, the adaptation degree of the machine learning model and the hardware resources is improved, which helps fine optimization from the machine learning model to the hardware resources, the parallel resource utilization rate is simultaneously improved, and the accelerator resource waste is avoided.

[0033] The AI accelerator construction scheme provided by the embodiments of the present application can also be executed by an electronic device, which can be a server, a server cluster, a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a special-purpose device (such as a special-purpose terminal device with an AI accelerator construction method system, etc.). The chips introduced in the above embodiments can also be carried in these electronic devices. Alternatively, these electronic devices can also install a service program for executing the AI accelerator construction scheme.

[0034] Figure 1 A flowchart of an AI accelerator construction method provided by the embodiments of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0035] 101, importing machine learning models of different AI frameworks through a multi-framework adaptation interface, parsing to obtain a model representation of the machine learning model, and converting the model representation into a linear algebra dialect representation in a form unified by using an MLIR dialect conversion chain;

[0036] 102, in the target-independent optimization stage, through target-independent optimization processing, the core computing semantics in the linear algebra dialect are retained, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with a high-efficiency data flow structure;

[0037] 103, in the hardware adaptive optimization stage, identifying the target hardware characteristics of the target hardware;

[0038] 104, adaptively performing hardware optimization processing matched with the target hardware characteristics on the optimized intermediate representation to obtain a target intermediate representation;

[0039] 105, converting the target intermediate representation into hardware operation units executable by the target hardware, loading the hardware operation units into the target hardware, and constructing an AI accelerator;

[0040] ​106, in the AI accelerator, the compiled-time IR is connected with the runtime to run the hardware operation unit on the target hardware through the hardware abstraction layer dialect, the corresponding computing resources in the target hardware are scheduled, and hardware acceleration of the machine learning model is completed.

[0041] Through steps 101 to 106, the AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures through the target-independent optimization stage processing and adaptive hardware optimization, so that the flexible allocation of computing resources in the target hardware is realized through the AI accelerator, the model deployment efficiency and the running efficiency are effectively improved, the adaptation degree of the machine learning model and the hardware resources is improved, which helps fine optimization from the machine learning model to the hardware resources, the parallel resource utilization rate is simultaneously improved, and the accelerator resource waste is avoided.

[0042] In the embodiments of the present application, the machine learning models of different AI frameworks can be various deep learning and machine learning models constructed under mainstream AI frameworks such as TensorFlow, PyTorch, JAX, covering AI task scenarios such as image classification, feature extraction, data prediction, etc.

[0043] Among them, the model of the PyTorch framework can include a lightweight neural network constructed based on PyTorch 2.0, for example, an image classification model containing convolutional layers, batch normalization layers, ReLU activation layers and fully connected layers. When imported, such a model will trigger the Dynamo component processing through the torch.compile function of PyTorch 2.0, finally generate an FX computation graph and further convert it into a unified IR.

[0044] The model of the TensorFlow framework can include a sequential model or a functional model constructed using the Keras API, such as a recurrent neural network (RNN) model for text classification or a convolutional neural network (CNN) model for image segmentation. Such a model will parse its computation graph structure through the TensorFlow interface adapted by the system front end, and extract the corresponding operator sequence.

[0045] The model of the JAX framework can include a complex numerical calculation model built based on the automatic differentiation feature of JAX, for example, a prediction model for scientific computing combined with AI. Such a model can parse its functional calculation logic through a front-end JAX adaptation interface and be converted into an identifiable model representation. These models from different frameworks, regardless of the specific tasks they involve or the operators they contain (such as matrix multiplication, Reduction operators, convolution operators, activation operators, etc.), will ultimately be converted into an MLIR ModuleOp represented in the linear algebra dialect through the MLIR dialect conversion chain, ensuring that subsequent optimization and code generation processes can standardize the processing of various models and achieve efficient execution on target hardware such as TPU.

[0046] As an optional embodiment, in step 101, machine learning models of different AI frameworks are imported through a multi-framework adaptation interface, the model representation of the machine learning model is parsed, and the MLIR dialect conversion chain is used to convert the model representation into a formally unified linear algebra dialect representation, including:

[0047] If the imported AI framework is PyTorch, the Dynamo component is triggered to work through the torch.compile function of PyTorch 2.0. The Dynamo component dynamically modifies the Python bytecode by hooking the frame evaluation API of CPython, rewrites the bytecode to extract the sequence of Aten operators, and finally generates an FX computation graph. Further, the FX computation graph is traversed by the SHARK Turbine tool, the data types and operator names of the operators in the FX computation graph are parsed, and the Aten operators are mapped one by one to the Torch dialect operators defined in the Torch MLIR tool. Finally, the conversion interface of Torch MLIR is called to convert the Torch dialect operators into an MLIR ModuleOp represented in the linear algebra dialect, and the MLIR ModuleOp contains the complete calculation definition of the machine learning model.

[0048] Specifically, in the optional embodiments of the present application, for the import and unified IR conversion of PyTorch framework models, the cooperation of PyTorch ecological tools and MLIR dialect chain is needed to realize the complete conversion from Python code to linear algebra dialect representation, and the whole process needs to ensure that the operator semantics are not lost and adapt to the subsequent unified optimization process. When the imported AI framework is PyTorch, first start the model analysis and operator extraction process, and call the torch.compile function provided by PyTorch 2.0, which will automatically trigger the built-in Dynamo component of PyTorch. The core function of the Dynamo component is to break through the limitations of Python static analysis, and through the hooking of the frame evaluation API of the CPython interpreter, it dynamically intercepts and modifies Python bytecodes during model execution. For example, when processing a PyTorch image classification lightweight model containing a convolution layer and a ReLU activation layer, Dynamo will analyze the bytecodes line by line during the execution of the model, identify the instructions related to tensor operations, and rewrite those scattered, non-operatorized bytecode fragments into a continuous sequence of Aten operators. Aten is the core operator library at the bottom of PyTorch, covering basic operations such as matrix multiplication, convolution, and activation. Finally, an FX computation graph is generated. The FX computation graph clearly records the execution order of each Aten operator in the model, the input-output relationship, and the tensor dimension information. For example, in the above lightweight model, the FX computation graph records the calling relationship of aten::conv2d (convolution operator) and aten::relu (activation operator) in turn, as well as the input tensor shape and data type of each operator.

[0049] After the completion of the FX computation graph generation, the dialect mapping link from the PyTorch ecosystem to the MLIR ecosystem is entered, and at this time the SHARK Turbine tool needs to be used. This tool is a conversion bridge specially connecting PyTorch and MLIR, and can deeply analyze the structure and semantics of the FX computation graph. SHARK Turbine will traverse each node (i.e. Aten operator) in the FX computation graph, first analyze the key information of each operator, including but not limited to the data types of input and output tensors (such as float32, int64), operator names (such as aten::conv2d, aten::batch_norm), and attribute parameters of the operator (such as the convolution kernel size and step of the convolution operator), and then according to the preset operator mapping rule, each Aten operator is matched one by one to the Torch dialect operator predefined in the Torch MLIR tool. Taking the aten::conv2d operator in the above lightweight model as an example, SHARK Turbine will identify the convolution kernel size, input channel number and other attributes of the operator, and accurately map it to the Torch::conv2d operator defined in the Torch dialect of Torch MLIR, ensuring that the calculation logic and parameter information of the operator are completely corresponding, and avoiding semantic deviation caused by dialect conversion.

[0050] Finally, the unified IR generation link is entered, and the mapped Torch dialect operator is further converted into an MLIR ModuleOp represented by the linear algebra dialect (Linalg Dialect) by calling the conversion interface of Torch MLIR. Torch MLIR will first perform semantic analysis on the Torch dialect operator, convert the operator logic specific to the Torch dialect (such as the automatic broadcast mechanism specific to PyTorch) into general linear algebra operation logic, and then generate an operator representation conforming to the Linalg dialect specification. For example, the Torch::conv2d operator in the above lightweight model will be converted into the Linalg::conv_2d operator of the Linalg dialect, and the Torch::relu operator will be converted into the Linalg::relu operator, and these Linalg operators will be organized into an MLIR ModuleOp. ModuleOp is the basic unit of MLIR that represents a complete computing program, including all computing definitions of the lightweight model from receiving data at the input layer, through convolution and activation operations, to output classification results, and this linear algebra dialect representation is platform-independent and can be directly used as a unified input carrier for subsequent target-independent optimization and hardware adaptive optimization.

[0051] After obtaining the linear algebra dialect representation, in step 102, in the target-independent optimization stage, the core computing semantics in the linear algebra dialect are preserved and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure through target-independent optimization processing.

[0052] In the embodiments of the present application, the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization.

[0053] As an optional embodiment, in step 102, the scheduling creation layer traverses all blocks and operators in the blocks in the MLIR ModuleOp corresponding to the linear algebra dialect representation, identifies root operators in all blocks as optimization center points, and assigns a unique identifier kRootOpAttr to the root operators. Through the fusion group attribute kFusionGroupsAttr, the producer operators providing input to the root operators and the consumer operators using the output of the root operators are marked respectively, and the producer operators and the consumer operators are fused into the root operators. Finally, the linear algebra dialect operators that are not marked as the root operators and do not belong to linalg.fill operators or dequantization operators are distributed to independent scheduling sections. Memory management optimization, redundant computation elimination, and data access pattern optimization are performed synchronously to generate the optimized intermediate representation.

[0054] Specifically, in step 102, the core work of the scheduling creation layer is to perform operator relationship sorting and fusion optimization on the MLIR ModuleOp of the linear algebra dialect representation, and to complete general optimization to generate an optimized intermediate representation. The entire process needs to ensure that the operator semantics are complete and the data flow efficiency is improved.

[0055] First, the scheduling creation layer will completely traverse all blocks contained in the MLIR ModuleOp and the operators in each block. During the traversal process, the computing logic, input and output tensor association, and data dependency relationship of each operator are parsed line by line to determine which operators are suitable as root operators of the optimization center point. The selection of the root operator is usually based on the core position of the operator in the computing flow. Generally, it is an operator with large computation and significant impact on the overall execution efficiency, such as a two-dimensional convolution operator (Linalg::conv_2d) in a convolutional neural network model for image classification, which is selected as the root operator. Because the convolution operation often occupies most of the computing resources of the model, fusion around it can maximize the reduction of data transmission overhead between operators. After determining the root operator, the scheduling creation layer assigns a unique kRootOpAttr identifier to each root operator. This identifier is used to distinguish root operators from other operators in the subsequent fusion process to avoid confusion about the ownership of operators.

[0056] Subsequently, the operator fusion link is entered, and the scheduling creation layer establishes the association of the root operator and other operators through the fusion group attribute kFusionGroupsAttr. First, the producer operator that provides the input data for the root operator is identified according to the data dependency relationship, and then the consumer operator that directly uses the output data of the root operator is found. Then, the two types of operators are gradually fused into the calculation unit where the root operator is located. Taking the two-dimensional convolution operator mentioned just now as the root operator as an example, if the input data of the convolution operator comes from the output of the batch normalization operator (Linalg::batch_norm), then the batch normalization operator is the producer operator. The output data of the convolution operator will be transmitted to the ReLU activation operator (Linalg::relu) for nonlinear transformation, and the ReLU activation operator is the consumer operator. The scheduling creation layer will mark the batch normalization operator and the ReLU activation operator as belonging to the same fusion group as the convolution operator through kFusionGroupsAttr. After that, the mean and variance calculation of the batch normalization, the feature extraction calculation of the convolution, and the nonlinear activation calculation of the ReLU are integrated together to form a fused composite operator. In this way, the intermediate tensors that need to be transmitted between three independent operators (batch normalization output to convolution input, convolution output to ReLU input) can be directly transmitted within the fused operator, without the need for frequent reading and writing of intermediate data in memory, greatly reducing the data transfer time and memory access overhead between operators.

[0057] After the fusion of the root operator and the related operators is completed, the scheduling creation layer will process the linear algebra dialect operators that are not marked as root operators. For these operators, the linalg.fill operator and the dequantization operator are first excluded, because the linalg.fill operator is usually used to initialize tensor data (such as filling boundary values for the input feature matrix of the convolution layer or initializing zero values for the weight matrix of the fully connected layer), and the dequantization operator is used to restore the low-precision tensor data (such as INT8) quantized and compressed in the training stage to high-precision data (such as FP32) required for inference. The calculation logic of these two types of operators is relatively simple, and the calculation amount is small. Separately allocating a scheduling section for them will increase the scheduling management overhead, which is not conducive to overall efficiency improvement, and therefore it is more suitable to be executed together with the associated operators before and after. For the remaining operators that are not marked as root operators and do not belong to the above two types, such as the max pooling operator (Linalg::max_pool_2d) used for downsampling in the model, the scheduling creation layer will allocate them to independent scheduling sections, and each independent scheduling section corresponds to a separate calculation task unit, ensuring that the execution of these types of operators will not be disturbed by other fused operators, and also facilitating subsequent differentiated optimization for the characteristics of different scheduling sections.

[0058] Finally, while completing the operator fusion and scheduling section allocation, the scheduling creation layer synchronously performs memory management optimization, redundant computation elimination, and data access pattern optimization, and finally generates an optimized intermediate representation. Memory management optimization adjusts the storage layout and access order of data in memory according to the memory access requirements of different operators, such as adjusting the frequently accessed feature tensor of the fused convolution, batch normalization, and ReLU composite operator to a continuous storage mode, so that the hardware cache can more efficiently read data and reduce the probability of cache miss failure. Redundant computation elimination identifies repeated computation steps in the model by analyzing the entire ModuleOp computation process, such as a fixed weight matrix repeatedly participating in matrix multiplication in multiple loop iterations, and only computing the weight matrix once and caching the result, and directly reusing the cached data in subsequent iterations to avoid resource waste caused by repeated computation. Data access pattern optimization sorts the data dependency order between different scheduling sections, arranges operators with close data dependencies in adjacent execution steps, such as directly obtaining the input data of the max pooling operator in the independent scheduling section from the output results of the previous fused operator, reducing the number of data migrations between different memory areas and delays. Taking the entire image classification model as an example, after these optimizations, the optimized intermediate representation not only retains the core computation semantics of the linear algebra dialect, but also significantly improves the data flow efficiency through operator fusion and general optimization, providing an efficient and unified input carrier for subsequent adaptive optimization for target hardware such as TPU, GPU, etc.

[0059] Further optionally, in the step of fusing the producer operator and the consumer operator into the root operator, the fuseRootsWithConsumers function is called to fuse the consumer operator into the root operator. It can be understood that, in actual application, in the AI accelerator code generation system based on MLIR and IREE, the process of calling the fuseRootsWithConsumers function to fuse the consumer operator into the root operator is implemented by relying on the accurate analysis of operator data dependency and the dynamic adjustment of IR structure by the scheduling creation layer, and the core goal is to eliminate the intermediate data transmission overhead between the root operator and the consumer operator, while ensuring the integrity of the calculation logic after fusion. Before execution, the function has completed the identification and marking of the root operator (assigned kRootOpAttr) through the previous process, and the output tensor association relationship of the root operator is clear. Therefore, the function first takes the output tensor of the root operator as the tracking basis, traverses the operator dependency link in the corresponding block in the MLIR ModuleOp, and locates all consumer operators that directly use the output of the root operator as input. For example, when processing a neural network model containing a convolution layer and an activation layer, if the root operator is a two-dimensional convolution operator in the linear algebra dialect (Linalg::conv_2d), the output tensor will be directly passed to the ReLU activation operator (Linalg::relu), and at this time the fuseRootsWithConsumers function will accurately identify that the ReLU operator is the consumer operator that needs to be fused through the tensor usage relationship graph.

[0060] After locating the consumer operator, the fuseRootsWithConsumers function further analyzes the calculation semantics and parameter attributes of the root operator and the consumer operator, and judges the compatibility of their fusion. The key is to check whether the input of the consumer operator is only from the output of the root operator (without other external data dependency), and whether the calculation logic of the consumer operator can be embedded in the execution flow of the root operator (such as activation operators, pooling operators, etc. Light calculation operators usually have such compatibility). If the fusion condition is met, the function will start the reconstruction of the IR structure: first, the calculation logic of the consumer operator is disassembled into the basic operation steps supported by the linear algebra dialect, and then these steps are sequentially embedded at the end of the calculation flow of the root operator to form a unified composite calculation unit. For example, the ReLU activation operator, whose calculation logic is to retain the original value if greater than 0 and set 0 if less than 0, will be disassembled into comparison and selection operations, directly connected after the feature extraction calculation of the convolution operator, so that the convolution output tensor can directly participate in the ReLU operation without writing to the memory, saving the redundant step of writing the output of the root operator to the memory and reading the data from the memory by the consumer operator.

[0061] Meanwhile, the fuseRootsWithConsumers function synchronously processes the updates of the operator attributes and the scheduling markers. The attribute parameters of the consumer operator (such as the activation threshold of the ReLU operator) are integrated into the attribute set of the root operator, ensuring that the calculation parameters are not lost after fusion; and by updating the kFusionGroupsAttr marker, the consumer operator is formally incorporated into the fusion group to which the root operator belongs, and the association between the two is made clear. For independent consumer operator nodes that are no longer needed after fusion, the function triggers the cleaning operation of IR to remove these redundant nodes to maintain the simplicity of the MLIR ModuleOp structure and avoid invalid operators interfering in the subsequent optimization process. In addition, the function also coordinates with the memory management related modules to adjust the memory access mode of the fused operator. For example, the reading operation of the original consumer operator on the output tensor of the root operator is changed to directly reference the calculation result of the root operator at the register level, further reducing the memory access delay.

[0062] Taking the image feature extraction task as an example, if the root operator is a convolution operator for processing the input feature map, and the consumer operator is a subsequent batch normalization operator (Linalg::batch_norm), the fuseRootsWithConsumers function will first confirm that only the feature map data comes from the convolution operator output in the input mean and variance parameters of the batch normalization operator, without other external dependencies, and then embed the mean calculation and variance adjustment steps of the batch normalization into the execution process of the convolution operator, so that the feature map generated by the convolution is directly used for batch normalization processing without the need for intermediate transfer in memory. At the same time, the kFusionGroupsAttr marker of the root operator is updated to include the batch normalization operator in the fusion group, and the original independent batch normalization operator node is cleaned up. Through a series of operations, the function finally realizes the deep fusion of the root operator and the consumer operator, forms a linear algebra dialect operator with convolution and batch normalization composite functions, not only retains the original calculation semantics, but also significantly optimizes the data flow efficiency, providing a more efficient IR basis for subsequent memory management, redundant calculation elimination and other steps in target-independent optimization.

[0063] In step 103, in the hardware adaptive optimization stage, the target hardware characteristics of the target hardware are identified.

[0064] In the embodiments of the present application, the target hardware characteristics include: hardware type, hardware structure, and parallelism. Further optionally, the hardware optimization processing matched with the target hardware characteristics at least includes one of: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion.

[0065] It can be understood that the identification of the target hardware characteristics relies on the interaction interface of the system and the hardware and the preset hardware information library, and provides accurate hardware characteristic basis for subsequent hardware optimization processing through multi-dimensional information collection and analysis. The identification of the hardware type is mainly realized through the interaction of the hardware abstraction layer dialect of the system and the target hardware driver: the system calls the hardware backend driver interface (such as the CUDA driver of GPU, the OpenMP driver of CPU, and the exclusive driver of TPU), obtains the device identification information of the hardware, such as the hardware model and the architecture code returned by the driver, to distinguish whether the target hardware is CPU, GPU or TPU; at the same time, the preset hardware type mapping table is combined to confirm the specific category of the hardware and the corresponding optimization adaptation rule, such as identifying the hardware model as NVIDIA A100, determining it as a GPU type supporting Tensor Cores, and identifying the TPU as its exclusive systolic array optimization strategy, to ensure that the subsequent optimization direction is completely aligned with the hardware type.

[0066] The identification of the hardware structure focuses on the memory level and the computing core layout of the hardware. The system will obtain the memory distribution information and the computing core characteristics of the target hardware through the driver API or the hardware specification query interface. In terms of memory structure, the multi-level memory composition of the hardware will be determined, such as whether it contains on-chip private memory, on-chip shared memory and off-chip high-speed memory (such as HBM), and the capacity, access speed and data transmission bandwidth of each level of memory will be obtained, for example, when identifying GPU, the size of the on-chip shared memory and the bandwidth of the off-chip HBM will be confirmed, and when identifying TPU, the maximum capacity and access delay of the on-chip private memory will be obtained. In terms of computing core structure, the core type and layout of the hardware will be analyzed, such as whether the GPU is equipped with Tensor Cores, whether the TPU adopts the systolic array structure, and whether the core of the CPU supports the vector instruction set (such as AVX-512). These structure information directly determines the direction of subsequent memory optimization (such as on-chip memory allocation) and computing division (such as sub-matrix operation adapted to Tensor Cores).

[0067] The identification of parallelism mainly relies on the concurrent execution capability supported by hardware. Through driving query or hardware performance interface, the parallel computing parameters of hardware are obtained, including but not limited to the maximum number of workgroups supported by hardware, the number of threads that each workgroup can accommodate, the total number of computing cores, etc. For example, when identifying GPU, the maximum number of thread blocks supported by GPU and the maximum number of threads in each thread block are obtained. When identifying TPU, the size of the computing unit array and the number of parallel processable subtasks are obtained. At the same time, the task scheduling mechanism of hardware is also parsed, such as whether it supports multi-stream asynchronous execution and whether it supports dynamic load balancing of tasks. These parallelism information will directly guide the task division granularity in the subsequent multi-layer tiling optimization, such as how to divide the number of workgroups in the parallel allocation layer and how to cut the data to match the parallel processing capability of hardware in the BatchTile layer, to ensure that the optimized computing task can fully utilize the concurrent resources of hardware and avoid parallel capability waste or overload.

[0068] In the identification process, the system integrates the collected hardware types, hardware structures and parallelism information into standardized hardware feature descriptions and stores them in the attribute set of the hardware abstraction layer dialect. Subsequent hardware optimization processes such as tensor shape adaptation, multi-layer tiling, memory optimization and vector conversion will directly read these feature descriptions to ensure that each optimization operation accurately matches the actual characteristics of the target hardware. For example, according to the identified GPU Tensor Cores structure, the KernelTile layer will divide the matrix operation into adaptive submatrix calculation. According to the identified on-chip memory size, the BatchTile layer will adjust the tiling parameters to ensure that the data can be completely placed in the on-chip memory, and finally realize the deep adaptation of hardware optimization and target hardware.

[0069] As an optional embodiment, in step 104, the adaptive execution of the optimization intermediate representation and the hardware optimization processing matched with the target hardware characteristics are performed to obtain a target intermediate representation, including: after identifying the target hardware characteristics, the null value padding layer takes the optimization intermediate representation as input and traverses all linear algebra operator input tensors and output tensors. For tensors that do not meet the target hardware characteristics, the tensor.pad operation is used to fill 0. For the filled tensors or tensors that meet the target hardware characteristics, the tensor.extract_slice operation is used to extract valid data, which aligns the tensor shape with the target hardware computing method while keeping the output result of the machine learning model unchanged.

[0070] Specifically, in step 104, the core role of the null value padding layer is to adjust the tensor shape so that the linear algebra operators in the optimized intermediate representation can adapt to the calculation logic of the target hardware, and to achieve accurate alignment between the tensor and the hardware characteristics without changing the model output result. The null value padding layer takes the optimization intermediate representation that has undergone target-independent optimization as the processing object. First, it will comprehensively traverse all the input and output tensors of the linear algebra operators in the optimization intermediate representation. This traversal process will combine the target hardware characteristics identified in the early stage (such as the memory transfer rules of the hardware, the fixed vector length limit, the tensor dimension specification supported by the calculation core, etc.), and check whether the shape of each tensor meets the hardware requirements one by one. For example, if the target hardware is a GPU that supports fixed vector length instructions, its vector operation unit can only efficiently process tensors with a dimension of a specific value. When the input tensor dimension of a linear algebra operator is less than this specific value, it is determined that the tensor does not meet the target hardware characteristics. If the target hardware is a TPU, its memory transfer mechanism requires the tensor dimension to be an integer multiple of a certain value. When the dimension of a tensor cannot be divided by this value, it is also determined as not meeting the characteristics.

[0071] For tensors that are determined not to meet the target hardware characteristics, the null value padding layer will perform padding operations to supplement the specified dimensions of the tensor with the value 0, so that the overall shape of the tensor is adjusted to meet the hardware requirements. The padding operation will determine the position and quantity of padding according to the hardware characteristics. For example, if the hardware requires the tensor channel dimension to be an integer multiple of 8, and the current tensor channel dimension is 5, 3 0s will be padded at the end or beginning of this dimension to make the dimension 8. If the hardware memory access requires the tensor height dimension to be 16, and the current height is 12, 4 0s will be padded to make the height 16. For tensors that already meet the target hardware characteristics, or tensors that meet the requirements after padding, the null value padding layer will further perform effective data extraction operations to locate the original effective data area in the tensor, and remove the 0 values supplemented in the padding process, and only keep the effective data required for calculation to participate in subsequent operations. For example, after padding, the channel tensor with a dimension of 5 becomes 8. After completing the calculation of the corresponding linear algebra operator, the first 5 dimensions of the effective data will be extracted, and the last 3 dimensions of the padded 0 values will be discarded. The tensor with a height of 12 padded to 16 will extract the first 12 heights of the effective data after calculation to ensure that the final output result is completely consistent with that before padding.

[0072] During the whole process, the null value padding layer closely connects the calculation logic of the linear algebra operator, ensuring that the padding and extraction operations are seamlessly integrated into the execution flow of the operator, that is, the padding operation is completed before the operator reads the tensor data, providing the operator with tensor input that meets the hardware requirements. The extraction operation is executed after the operator completes the calculation, and the original valid data is filtered out from the calculation results as the operator output. For example, when processing a convolution operator, if the width dimension of the convolution input tensor does not meet the requirements of the GPU vector instruction, the null value padding layer will first pad 0 in the width dimension to make the dimension meet the requirements. The convolution operator completes the feature extraction calculation based on the padded tensor, and after the calculation is completed, the padding part in the width dimension is removed through the extraction operation to obtain the convolution output result that matches the original tensor width. Through such processing, the execution efficiency problem caused by the mismatch between the tensor shape and the hardware characteristics is solved, and the calculation accuracy of the machine learning model is guaranteed through effective data extraction, providing a shape-adapted tensor basis for subsequent hardware optimization steps such as multi-layer tiling and memory optimization.

[0073] As an optional embodiment, in step 104, the adaptive execution of the optimization intermediate representation and the hardware optimization processing matched with the target hardware characteristics are performed to obtain the target intermediate representation, including: based on the TilingInterface interface, the iteration domain of the operator in the optimization intermediate representation is obtained in the parallel allocation layer, the tiling size is calculated and the outer loop is generated, the tiling operation implementation is generated in the innermost loop, and the operator is divided and allocated to multiple work groups after cutting, so that the amount of tensor data processed by each work group meets the memory limit of the target hardware. After the parallel allocation layer is executed, the BatchTile layer combines the maximum private memory of the target hardware to calculate the tiling size set, ensuring that each dimension vector of each tensor is divided by the corresponding tiling size value in the tiling size set and the total memory occupancy after cutting is less than the maximum private memory. The scf::tileUsingSCF interface is called to complete tiling and introduce For control flow to realize on-chip memory reuse and process the thread-in data access of the Reduction operator. After the BatchTile layer is executed, the KernelTile layer splits the complex calculation into sub-calculations matched with the hardware core according to the core calculation characteristics of the target hardware; the calculation definition and memory transmission are segmented by using respective control flows, and the time overhead of the calculation definition and memory transmission is balanced through the memory optimization of the BatchTile layer.

[0074] Specifically, in step 104, the multi-layer tiling optimization, as the core part of hardware adaptive optimization, needs to be executed in the progressive order of the parallel allocation layer, the BatchTile layer, and the KernelTile layer. Each layer adjusts the calculation division strategy around the target hardware characteristics to ensure that the operator processing is accurately adapted to the hardware memory limit, parallel capability, and core computing characteristics. The parallel allocation layer first takes the optimized intermediate representation as input and relies on the TilingInterface interface to implement the preliminary division of the linear algebra operator. This interface can automatically analyze the iteration domain of the operator, clearly define the calculation range and data dependency of the operator in each dimension, and then calculate the reasonable tiling size based on the basic memory limit of the target hardware (such as the maximum data volume that a single workgroup can carry). Based on this size, the outer loop structure is generated. These outer loops correspond to the iteration logic of different tiling blocks, and the specific execution logic of the tiling operation is constructed in the innermost loop, which divides the original complete operator into multiple smaller sub-operators. After the division is completed, the parallel allocation layer distributes these sub-operators to multiple workgroups of the target hardware, and each workgroup only processes the corresponding sub-operator and the associated small-size tensor data, ensuring that the memory occupancy of each workgroup does not exceed the hardware's basic limit. For example, when processing a large-size matrix multiplication operator, the parallel allocation layer divides the matrix into multiple sub-matrix blocks, and each sub-matrix block is calculated by a workgroup, avoiding memory overflow caused by excessive data volume in a single workgroup.

[0075] After the parallel allocation layer is executed, the BatchTile layer further optimizes based on the division results of the former, with the core focusing on the on-chip private memory characteristics of the target hardware. The BatchTile layer first obtains the maximum private memory value of the target hardware through the hardware abstraction layer dialect, and then calculates the adaptive tiling size set in combination with the dimension information of each tensor in the current scheduling task. This set needs to satisfy two key conditions: first, each dimension of each tensor can be divided by the corresponding tiling size in the set, ensuring that there is no remaining data after division; second, the total occupancy of all tensors in the on-chip private memory after division does not exceed the maximum private memory value, avoiding frequent access to off-chip memory due to insufficient on-chip memory. After determining the tiling size set, the BatchTile layer calls the pre-set interface to complete the final tiling and introduces the For control flow statement to manage data access logic. This control flow allows the calculation process within the same workgroup to reuse data in the on-chip private memory, reducing the overhead of repeated data loading. Most importantly, the BatchTile layer can effectively handle Reduction operators (such as sum and average operators that require cumulative previous results), as it handles on-chip memory access of internal threads in a workgroup and does not have data isolation problems between different threads, allowing it to directly obtain the previous calculation results required for accumulation. The parallel allocation layer cannot handle such operators due to cross-thread data interaction, and the BatchTile layer compensates for this limitation.

[0076] After the BatchTile layer processing is completed, the KernelTile layer, as the last step of multi-layer tiling, further refines the granularity of the split to match the core computing characteristics of the target hardware. The KernelTile layer first analyzes the computing core specifications of the target hardware. For example, if the target hardware is a GPU equipped with Tensor Cores, its cores are good at processing submatrix operations of a specific size. If it is a TPU, its computing cores are adapted to tensor operations of a specific structure. The KernelTile layer will accordingly split the sub-computations output by the BatchTile layer into smaller sub-tasks, ensuring that each sub-task can be directly executed by the hardware core without additional adjustments. At the same time, the KernelTile layer separates the computation definition and memory transfer through control flow design, allowing the data transfer from off-chip memory to on-chip memory to be independently scheduled from the computation execution in on-chip memory. Combined with the on-chip memory optimization already completed by the BatchTile layer, the time overhead of data transfer and computation gradually tends to balance, avoiding the idle of computing cores due to data transfer waiting, or the insufficient data supply due to too fast computation, for example, when processing complex matrix multiplication, the KernelTile layer will split the computation into submatrix multiplication that the hardware core is adapted to, and through control flow, the data transfer and submatrix multiplication are alternately performed, so that the hardware core is in a high-efficiency operation state, and finally the target intermediate representation fully adapted to the target hardware is output.

[0077] In an optional embodiment of step 104, further optionally, the computation definition and memory transfer are separated by their respective control flows, and after the time overhead of computation definition and memory transfer is balanced through the memory optimization of the BatchTile layer, the memory allocation layer further includes: after the multi-layer tiling execution is completed, the SubSpanToAllocaPattern rewriting mode is defined, and the memory optimization is performed on the optimized intermediate representation to convert the complex memory reference in the optimized intermediate representation into a memref::AllocaOp operation to apply for on-chip private memory of the target hardware; according to whether the operator has a vector user, the memory space attribute and alignment mode are set, and the memref::CopyOp instruction is inserted to realize the data copy from off-chip memory to on-chip private memory. Further, after the memory allocation layer is executed, the memory offset flattening layer calls the IREEExpandStridedMetadata Pass and ExpandStridedMetadata Pass in sequence to convert the high-dimensional memory access expression into one-dimensional through affine transformation, and establish the memory copy mapping relationship from high-dimensional to low-dimensional.

[0078] Specifically, after the multi-layer tiling optimization is completed, the memory optimization will focus on the efficient use of on-chip memory and the simplification of memory access logic. Through the collaborative operation of the memory allocation layer and the memory offset flattening layer, the hardware memory access delay is further reduced, and the memory addressing rules of the target hardware are adapted. After the multi-layer tiling is completed, the memory allocation layer first starts the memory operation optimization. The core is to define the SubSpanToAllocaPattern rewrite mode to sort out the memory reference relationship in the optimized intermediate representation. This rewrite mode can accurately identify complex memory reference forms, such as sub-view memory references generated by tensor splitting. Such references often contain multiple layers of nested address calculation logic, which directly increases the complexity and delay of hardware memory access. The memory allocation layer will convert these complex memory references to memref::AllocaOp operations, which will apply for independent on-chip private memory space to the target hardware. On-chip private memory can directly interact with hardware registers, and its access speed is much faster than off-chip memory, which can greatly improve data read-write efficiency. When applying for on-chip private memory, the memory allocation layer will first determine whether the current operator has a vector user. If the operator needs to participate in vector operations in the future, such as vector addition and vector multiplication handled by the vectorization layer, it will set the high-speed vector memory space attribute and the alignment mode matching the vector instruction to ensure that the vector operation can efficiently read memory data. If there is no need for vector operation, the standard memory space and basic alignment mode are used. At the same time, the memory allocation layer will insert the memref::CopyOp instruction to copy the data originally stored in the off-chip memory to the newly applied on-chip private memory. For example, when processing feature map data of a convolutional neural network, the feature map sub-blocks after multi-layer tiling will be transferred from the off-chip HBM to the on-chip private memory of the GPU through this copy instruction. The subsequent convolution calculation directly reads data from the on-chip memory, avoiding the delay caused by frequent access to off-chip memory.

[0079] After the on-chip memory application and data copy are completed by the memory allocation layer, the memory offset flattening layer will continue to process the dimension adaptation problem of memory access. This layer will call IREEExpandStridedMetadata Pass and ExpandStridedMetadata Pass in turn. These two Passes convert high-dimensional memory access expressions in the optimized intermediate representation into one-dimensional form through affine transformation technology. High-dimensional memory access, such as three-dimensional tensor channel, height, and width dimension access, although conforms to the model calculation logic, the memory addressing of the target hardware is usually implemented in one-dimensional address form. Directly using high-dimensional access will increase the address calculation overhead of the hardware. Through affine transformation, the memory offset flattening layer will establish a mapping relationship between high-dimensional dimensions and one-dimensional addresses, accurately corresponding each data position in the high-dimensional space to a unique address in the one-dimensional memory. For example, when processing the memory access of a three-dimensional feature map, the channel index is multiplied by the height and then by the width, and the height index is multiplied by the width, and then the width index is added to the high-dimensional offset calculation. Through affine transformation, this is simplified to a single one-dimensional offset value. This conversion not only makes the memory access logic more consistent with the addressing method of the hardware, but also avoids the hardware from frequently performing real-time calculation of multi-dimensional to one-dimensional during execution, reducing hardware resource consumption. Taking the TPU processing of fully connected layer data as an example, the weight matrix of the fully connected layer is usually two-dimensional in structure. The memory offset flattening layer will convert the two-dimensional input neuron and output neuron memory access into one-dimensional address access through affine transformation, so that the memory controller of the TPU can directly locate the weight data according to the one-dimensional address, cooperate with the data storage of the on-chip memory in the early stage, further improve the overall efficiency of the fully connected layer calculation, and ultimately provide low-latency and high-adaptation memory access basis for subsequent vector conversion and code generation.

[0080] In an optional embodiment of step 104, further optionally, the memory allocation layer defines a SubSpanToAllocaPattern rewrite pattern, performs memory optimization on the optimized intermediate representation, and converts complex memory reference in the optimized intermediate representation into a memref::AllocaOp operation to apply on-chip private memory of the target hardware, including: initializing the SubSpanToAllocaPattern rewrite pattern; the rewrite pattern inherits from Rewrite Pattern, adapts to all operation types, and sets the optimization level. Check whether the operation to be processed in the optimized intermediate representation has a CopySource attribute and does not have a CopySourceDone attribute, and return a failure indication if not satisfied; if satisfied, check whether the User of the operation has executed a memref::CopyOp, and if so, set the CopySourceDone attribute for the operation and remove the CopySource attribute, and return a success indication. Check whether there is a memref::CollapseShapeOp in the User of the operation to be processed, so as to determine whether the complex memory reference operation in the optimized intermediate representation needs to be replaced by a memref::AllocaOp, and update the operation pointer op synchronously. Set the memory space attribute and the alignment according to whether the operation to be processed has a vector user hasVectorUser; if there is a vector user, use the high-speed vector memory space obtained by getVectorMemoryAttr; if not, use the standard memory space. Create a new memref::AllocaOp to replace the original memory operation, and configure the memory type and the alignment of the memref::AllocaOp; insert a memref::CopyOp according to the read-only state of the memref::AllocaOp: if it is a read-only state, only insert a copy instruction from the result memory to the memref::AllocaOp; if it is a non-read-only state, in addition to the above copy instruction, additionally insert an instruction at the end of the operation to copy back from the memref::AllocaOp to the result memory. Replace all uses of the original memory operation with the updated memref::AllocaOp, and ignore the User of the memref::AssumeAlignmentOp and the memref::CopyOp; if the updated memref::AllocaOp has a sub-view User, update the sub-view operation according to the parameters of the memref::AllocaOp.

[0081] It can be understood that in the optional embodiment of step 104, when the memory allocation layer performs memory optimization through the SubSpanToAllocaPattern rewrite mode, the initialization of the rewrite mode will be completed first. Since the rewrite mode needs to adapt to various memory-related operations in the optimization intermediate representation, it will inherit from the system-based Rewrite Pattern to ensure that all operation types can be processed; at the same time, the corresponding optimization level will be set to let the system prioritize the processing of memory references when performing multiple rounds of optimization, so as to avoid the influence of other low-priority optimizations on the optimization effect of memory operations. For example, when processing the optimization intermediate representation containing multiple operators such as convolution and pooling, the rewrite mode will be called first to ensure that the optimization at the memory level is completed before other detailed optimizations. After initialization, the rewrite mode will perform attribute checking on the operation to be processed in the optimization intermediate representation, the core of which is to judge whether the operation has the CopySource attribute and does not have the CopySourceDone attribute. These two attributes are the key to the system marking whether the memory reference needs to be optimized. The CopySource attribute indicates that the memory reference corresponding to the operation needs to be further processed, and the CopySourceDone attribute indicates that the optimization has been completed. If the condition of having CopySource and not having CopySourceDone is not met, it means that the operation either does not need to be optimized or has been optimized, at which time the failure indication information will be returned to terminate the subsequent processing of the operation. If the condition is met, the User (i.e., the subsequent operation that depends on the output of the operation) of the operation will be further checked to see if it has executed the memref::CopyOp. If it has, it means that the data copying process has been completed, and there is no need to adjust the current memory reference. At this time, the CopySourceDone attribute will be set for the operation and the CopySource attribute will be removed, and the success indication information will be returned to avoid repeated optimization. For example, when processing the memory reference of the weight tensor in the fully connected layer, if it is found that the subsequent User has executed the memref::CopyOp, the memory reference optimization will be marked as completed, and no further processing will be performed.

[0082] Subsequently, the rewriting mode determines whether the complex memory reference in the optimized intermediate representation needs to be converted into a memref::AllocaOp, specifically by checking whether there is a memref::CollapseShapeOp in the User of the operation to be processed, which is commonly used to compress high-dimensional memory structures into low-dimensional ones. The memory reference dependent on it often contains multiple layers of sub-view nesting and belongs to a complex memory reference. If the operation exists, it means that the current memory reference needs to be converted into a more concise memref::AllocaOp to reduce the access complexity, and the operation pointer op will be updated synchronously to ensure that subsequent operations can accurately locate the memory reference that needs to be replaced. For example, when processing a feature map sub-view after multiple layers of tiling, if there is a memref::CollapseShapeOp for dimension compression afterwards, it will be determined that the complex memory reference corresponding to the sub-view needs to be converted into a memref::AllocaOp.

[0083] Once a conversion is determined, the memory space attributes and alignment are configured based on whether a vector user (hasVectorUser) exists for the operation to be processed. If a vector user exists, it means that the memory reference will subsequently participate in vector operations at the vectorization layer. Since vector operations have high requirements for memory access speed, the high-speed vector memory space of the target hardware is obtained through getVectorMemoryAttr, and an alignment matching the vector instructions is set to ensure that vector operations can efficiently read memory data. If no vector user exists, it means that the memory reference is only used for ordinary scalar operations. In this case, the standard memory space and basic alignment of the target hardware are used to balance memory resource usage and access efficiency. For example, when processing memory references corresponding to vector multiplication operators on the GPU, because a vector user exists, the GPU's high-speed vector memory is allocated for it and aligned according to the vector instruction requirements; when processing memory references corresponding to ordinary scalar addition, the GPU's standard memory space is used. Next, a new `memref::AllocaOp` will be created to replace the original complex memory reference operation. During the creation process, the memory type (such as matching the data type and dimension information of the tensor) and alignment of `memref::AllocaOp` will be configured according to the memory requirements of the operation to be processed, ensuring that the newly allocated memory can accurately inherit the functionality of the original memory reference. At the same time, `memref::CopyOp` will be inserted according to the read-only state of `memref::AllocaOp`: If the `AllocaOp` is in a read-only state, it means that subsequent calculations only need to read data from external sources and do not require data modification. In this case, only a copy instruction from the result memory (usually off-chip memory) to the `AllocaOp` (corresponding to on-chip private memory) will be inserted to load the data onto the chip; if it is in a non-read-only state, it means that subsequent calculations will modify the data in this memory. In addition to inserting a copy instruction from the result memory to the `AllocaOp`, an instruction to copy from the `AllocaOp` back to the result memory will also be inserted at the end of the operation to ensure that the modified data can be synchronized to the off-chip memory and avoid data loss. For example, when processing memory references of the input feature map of a convolution operator, if only the feature map data needs to be read for convolution calculation, AllocaOp is set to read-only, and only an off-chip to on-chip copy is inserted; when processing memory references of the output feature map, because the convolution result needs to be stored, AllocaOp is set to non-read-only, and an on-chip to off-chip copy will be inserted.

[0084] Finally, all uses of the original memory operation are replaced with the newly created memref::AllocaOp, and the User of the memref::AssumeAlignmentOp and the memref::CopyOp are ignored in the replacement process. The memref::AssumeAlignmentOp is only used to declare memory alignment information and does not depend on the specific form of the memory reference, and the memref::CopyOp has been adapted in the previous steps, so there is no need to replace the User of these two types of operations. If the new memref::AllocaOp has a subview User (i.e., subsequent operations access this memory through a subview), the access range and address calculation logic of the subview operation are updated according to the memory address, dimension, and other parameters of the AllocaOp to ensure that the subview can correctly locate the data in the newly allocated on-chip private memory. For example, when processing the subview of the convolution output feature map that depends on the pooling operator, after being replaced with the AllocaOp, the address parameters of the subview are updated to allow the pooling operator to accurately read the feature map data in the on-chip memory corresponding to the AllocaOp, ensuring the correctness of subsequent calculations.

[0085] As an optional embodiment, in step 104, the hardware optimization processing matching the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation, including: a vectorization layer taking the optimized intermediate representation as input, calling an linalg::vectorize interface to identify parallelizable scalar operations therein, and converting the scalar operations into vector operations; synchronously rewriting the memory access mode, converting scalar memory loading into vector memory loading, and converting scalar memory storage operations into vector memory storage operations; if the target hardware does not support vector operations, calling a LinalgToLoops method to convert linear algebra operators in the optimized intermediate representation into multiple loops; after the vectorization execution is completed, a vector flattening layer splits high-dimensional vector operators into multiple one-dimensional vector operators, so that the obtained to-be-processed intermediate representation conforms to an LLVM vector type, which is a compile-time abstraction of a target hardware vector register; and a software pipeline layer takes the vector-flattened to-be-processed intermediate representation as input, rearranges instructions in a loop body, and obtains the target intermediate representation, so that multiple different iteration instructions are executed in parallel in each loop iteration.

[0086] In particular, in the optional embodiment of step 104, the vectorization layer, the vector flattening layer and the software pipeline layer process the optimized intermediate representation in a progressive order, optimize around the vector processing capabilities and instruction level parallelism requirements of the target hardware, and finally generate a target intermediate representation adapted for hardware execution. The vectorization layer first takes as input the optimized intermediate representation that has been tiled, memory-optimized, and the like through multiple layers, and the core task is to convert scalar operations into vector operations that can take advantage of hardware vector instructions. It calls the linalg::vectorize interface, parses the execution logic of linear algebra operators in the optimized intermediate representation line by line, and identifies parallelizable scalar operations therein. Such operations are usually independent iterations in a loop, such as the ReLU activation calculation of the output feature map of the convolution layer in an image classification model, where the activation judgment of each pixel point (retaining the original value if greater than 0, or setting to 0 if less than 0) is independent of each other and belongs to parallelizable scalar operations. The interface packs and integrates these independent scalar operations into corresponding vector operations, such as combining the scalar activation operations of multiple pixel points into a vector activation operation. At the same time, the vectorization layer simultaneously rewrites the memory access pattern. The memory load operation originally directed at a single scalar data is changed to a vector memory load operation that loads multiple scalar data at a time. The memory storage operation directed at a single scalar result is also changed to a vector memory storage operation that stores multiple vector results at a time, ensuring that the memory access and the processing rhythm of the vector operation match. If the target hardware does not support vector operations, such as some low-end CPUs that can only handle scalar operations, the vectorization layer calls the LinalgToLoops method to disassemble the linear algebra operators in the optimized intermediate representation into multiple nested loops, so that each loop iteration handles a scalar operation, adapting to the scalar computing capabilities of the hardware, such as splitting the matrix multiplication operator of the fully connected layer into multiple loops that traverse the input neurons in the outer loop and the output neurons in the inner loop, to ensure that the operator can execute normally on a low-end CPU.

[0087] After the vectorization execution is completed, the vector flattening layer will continue to process the adaptation problem of the vector dimension. Since the vector operations generated by the vectorization layer can contain high-dimensional structures, the LLVM vector type is an abstraction of the target hardware vector register at compile time, and usually only supports one-dimensional form. If a high-dimensional vector operator is used directly, it will cause subsequent LLVM dialect conversion and hardware register allocation to fail. Therefore, the vector flattening layer will traverse the vectorized intermediate representation and split the high-dimensional vector operator into multiple one-dimensional vector operators. For example, when processing a three-dimensional vector convolution operation on a TPU, the operation originally contains vector calculations in the channel, height, and width dimensions. The vector flattening layer will split it into three independent one-dimensional vector operators corresponding to the vector calculations in the channel, height, and width dimensions, so that the split intermediate representation to be processed fully complies with the specifications of the LLVM vector type, ensuring that it can be successfully mapped to the vector registers of the TPU.

[0088] Finally, the software pipeline layer takes the vector flattened intermediate representation as input, focusing on the instruction level parallel optimization of the loop code. It analyzes the loop structure in the intermediate representation to be processed, identifies the key instructions in the loop body (such as memory load, calculation, and storage instructions), and then rearranges the order of instructions to break the serial execution constraints of instructions within a single loop iteration, allowing multiple instructions from different iterations to be executed in parallel in each loop iteration. For example, when processing the weight calculation of the fully connected layer, the software pipeline layer will move the instruction to load the next weight data to the calculation of the current weight instruction during the current iteration, allowing the calculation of the current iteration to be parallel to the memory load of the next iteration. At the same time, the instruction to store the current calculation result is delayed to the calculation period of the next iteration to form a pipeline execution mode. This optimization maximizes the utilization of the target hardware resources and reduces the idle time caused by waiting for memory loading or calculation to complete. For example, after executing this optimization on a GPU, the GPU's compute cores and memory controllers can work simultaneously during each loop iteration, significantly improving the overall execution efficiency of the fully connected layer. Finally, the target intermediate representation is generated to adapt to the vector characteristics and instruction level parallel requirements of the target hardware.

[0089] As an optional embodiment, before obtaining the target intermediate representation in step 104, the tensor in the optimization intermediate representation can also be allocated to the corresponding target hardware device according to the computing power of the target hardware and the amount of tensor data through the device scheduling layer; after entering the hardware adaptive optimization phase, a stream bound to the device is created according to the target hardware device where the tensor is located; the stream bound to the GPU adapts to the CUDA backend, and the stream bound to the CPU adapts to the CPU backend; the hardware optimization processing in each stream and the generation of subsequent hardware operation units are matched with the backend characteristics of the corresponding device; the streams pass the tensor data through the data copy operation containing memref::CopyOp, ensuring the correctness of cross-device data interaction.

[0090] In an optional embodiment of step 104, the device scheduling layer plays a core role in tensor, hardware matching and cross-device task coordination, which operates between target hardware characteristic identification and hardware adaptive optimization, and defines a clear device execution range for subsequent optimization processing. The device scheduling layer first analyzes the computing power of each target hardware and the data volume of the tensors in the optimization intermediate representation in combination with the target hardware characteristics identified in the early stage: the computing power of the target hardware is reflected in the number of parallel tasks it supports, the type of computing core (such as Tensor Cores of GPU, vector core of CPU), etc., and the data volume of the tensor is related to data storage demand and transmission cost. Based on these two factors, the device scheduling layer allocates different tensors in the optimization intermediate representation to the corresponding target hardware devices. For example, when processing an image segmentation model, large-size feature map tensors in the model used for feature extraction (large data volume, computation-intensive) are allocated to GPU, because it has a large number of parallel computing cores and can efficiently process large-scale tensor operations. Small-size parameter tensors in the model used for result post-processing (small data volume, light computation) are allocated to CPU to avoid resource waste caused by occupying GPU resources with small data volume, and to reduce the frequency of cross-device data transmission.

[0091] After completing the tensor allocation, when entering the hardware adaptive optimization phase, the device scheduling layer will create a stream directly bound to the device according to the target hardware device where each tensor is located. As a logical channel for hardware task execution, the stream will form a fixed association with the hardware backend. If the device bound to the stream is GPU, the stream will automatically adapt to the CUDA backend, and all subsequent computing tasks in the stream will follow the task scheduling rules and memory model of CUDA; if the device bound to the stream is CPU, the stream will adapt to the CPU backend, adapting to the thread scheduling mechanism and vector instruction set of CPU. This binding relationship ensures that the optimization processing in the stream can accurately match the hardware bottom characteristics, for example, the stream bound to GPU will give priority to the shared memory usage, thread block division, etc. of GPU in subsequent optimization. The CPU-bound stream will focus on the cache utilization, vectorization instruction adaptation, etc. of CPU optimization direction.

[0092] The hardware optimization processing (such as tensor shape adaptation, multi-layer tiling, vectorization, etc.) in each stream and the generation of subsequent hardware operation units strictly match the backend characteristics of the corresponding device. Taking the GPU-bound stream as an example, when performing multi-layer tiling optimization, the tiling parameters are determined according to the thread block size supported by the CUDA backend and the calculation specifications of the Tensor Cores, the large matrix operation is divided into sub-matrix calculation that adapts to the Tensor Cores, and the generated hardware operation unit is also a binary instruction that conforms to the CUDA instruction specification. When the CPU-bound stream performs vectorization optimization, it will convert scalar operations into corresponding vector operations based on the vector instruction set (such as the AVX series) supported by the CPU backend, and the generated hardware operation unit adapts to the vector registers and instruction execution logic of the CPU.

[0093] When there is a data dependency between tasks in different streams, the streams will pass tensor data through data copy operations containing memref::CopyOp to ensure the correctness of cross-device data interaction. For example, in an image segmentation model, the feature map tensor processed by the GPU stream needs to be passed to the CPU stream for final segmentation result output processing, which triggers the memref::CopyOp operation. This operation first reads the storage address and data format of the feature map tensor in the GPU stream, and then transfers the data from the GPU's off-chip memory (such as HBM) to the CPU's main memory according to the memory access rules of the CPU backend. During the transmission process, the memory layout differences between the two ends are automatically adapted to avoid calculation errors caused by incompatible data formats. Through this data transfer method, the streams on different devices can cooperate to complete the entire model's calculation task, fully leveraging the computing advantages of each hardware while ensuring the continuity of cross-device execution.

[0094] In the embodiments of the present application, the hardware operation unit is the core output product of the code generation stage, which is a binary instruction set that adapts to the computing characteristics of the target hardware (such as GPU, CPU, domestic TPU, etc.) and can be directly recognized and executed by the target hardware. Its essence is to map the target intermediate representation after hardware self-adaptive optimization to the specific execution carrier of the target hardware's native instruction through the dialect conversion and compilation process.

[0095] As an optional embodiment, in step 105, the target intermediate representation is converted into a hardware operation unit executable by the target hardware, and the hardware operation unit is loaded into the target hardware to construct an AI accelerator. Specifically, converting the target intermediate representation into a hardware operation unit executable by the target hardware can be implemented as follows: taking the target intermediate representation after hardware optimization as input, a dialect step-by-step conversion process is executed; starting from the linear algebra dialect in the target intermediate representation, the following processes are sequentially performed: tensor dialect, one-time buffering, memory reference dialect, vectorization, vector dialect, LLVM dialect conversion, LLVM dialect; the LLVM dialect file is converted into an assembly file corresponding to the target hardware by an LLVM compiler, and the assembly file is used to construct a binary form hardware language file and hardware operation instructions; the hardware language file and hardware operation instructions are fused into the hardware operation unit by combining the configuration information and hardware attribute information of the target hardware.

[0096] Understandably, in the optional embodiment of step 105, the process of converting the target intermediate representation into a hardware operation unit and building an AI accelerator revolves around dialect step-by-step conversion, LLVM compilation chain processing, and hardware configuration integration to ensure that the final output can be directly recognized and loaded by the target hardware. First, the dialect step-by-step conversion process is initiated. This process takes the target intermediate representation after hardware optimization (such as multi-layer tiling, memory optimization, and vector transformation) as input, and always starts with the linear algebra dialect. The linear algebra dialect, as a hardware-independent computational definition carrier, contains all the core computational logic in the model (such as convolution, matrix multiplication, activation operators, etc.). Subsequent conversions need to gradually adapt to the underlying hardware representation while preserving these computational semantics. During the conversion process, the tensor dialect stage is entered first. This stage mainly organizes the basic morphological description of the tensor data to prepare for subsequent memory operations. Next, a one-time buffering process is performed, converting the abstract memory representation of the tensor into a form that can be directly mapped to hardware physical memory. This solves the adaptation problem between tensors and hardware memory models. For example, when processing convolution operators in image classification models, the convolution input tensor is converted from an abstract tensor form into a buffer representation that can be associated with hardware memory addresses. Following this, the memory reference dialect stage is entered. This stage explicitly describes the data access patterns in hardware memory (such as memory addresses, data spans, storage order, etc.). For example, a contiguous memory storage order is set for the weight tensors of the convolution operator to reduce the probability of cache hit failures during hardware access. Then, scalar operations are converted into vector operations through vectorization, and the vector dialect standardizes the representation of vector operations to ensure compatibility with hardware vector register abstractions (LLVM vector types). Finally, through the LLVM dialect conversion step, the vector dialect representation is mapped to the LLVM dialect at the MLIR level, completing the transition from a high-level MLIR dialect to an LLVM-compatible representation.

[0097] The LLVM dialect conversion layer plays a key role in this process, as it defines a series of patterns that serve as conversion devices to accurately convert accelerator-specific dialect operators (such as Tensor Cores operators for GPUs or systolic array operators for TPUs) into LLVM IR. To ensure that the converted LLVM IR matches the hardware instruction set, on the one hand, the IntrinsicsXXX.td file (XXX is the name of the target hardware platform, such as IntrinsicsTPU.td for TPU) is defined to define the instruction primitives of the target accelerator. These primitives are abstract descriptions of hardware-level instructions at compile time, such as defining a matrix multiplication instruction primitive for the tensor operation core of TPU or a mixed-precision calculation instruction primitive for Tensor Cores of GPU. On the other hand, existing general operations in LLVM IR are used to avoid repeated definitions, such as directly reusing standard arithmetic operations in LLVM IR for hardware-specific scalar operations like addition and multiplication. Only complex operations specific to hardware (such as systolic array data transfer for TPU or shared memory synchronization for GPU) are handled through custom patterns and instruction primitives.

[0098] After completing the dialect conversion to obtain the LLVM dialect, the mlir-translate tool is used to generate the.ll file (LLVM IR text file). Subsequently, the LLVM compiler chain is used to process it. First, the.ll file is converted into the corresponding assembly file for the target hardware by tools such as clang and llc. The assembly file will retain the execution logic of the hardware instructions, such as matrix operation instructions for Tensor Cores and shared memory access instructions for GPUs, or vector register operation instructions for CPUs. Then, the LLVM compiler further compiles the assembly file into binary hardware language files (such as.ptx files for GPUs or.o files for CPUs) and corresponding hardware operation instructions. These binary files have the format of hardware executable instructions, but still need to be combined with the configuration information and attribute information of the target hardware to form a complete hardware operation unit.

[0099] Finally, the configuration information (such as the memory specification, the number of computing cores, the supported precision type, the data transmission bandwidth, etc.) and the attribute information (such as the instruction set version of the hardware, the memory access delay threshold, etc.) of the target hardware are read, and these information is fused with the binary hardware language file and the hardware operation instruction. For example, for a TPU, the on-chip private memory capacity and the systolic array size are embedded into the hardware operation unit to ensure that the memory allocation instruction in the operation unit is adapted to the memory limit of the TPU. For a GPU, the maximum thread block size and the shared memory size are fused to ensure that the thread scheduling instruction in the operation unit conforms to the parallel execution rules of the GPU. After the fusion, the hardware operation unit obtained contains all the instruction logic required for model calculation and is adapted to the hardware characteristics and configuration parameters of the target hardware. When the hardware operation unit is loaded into the target hardware, the construction of the AI accelerator is completed. For example, after the hardware operation unit adapted to the TPU is loaded into the TPU chip, the TPU can directly call the computing core to execute the operator instruction in the hardware operation unit to achieve hardware acceleration of the deep learning model.

[0100] In step 106, in the AI accelerator, the hardware abstraction layer dialect is used to connect the IR at the compilation time and the hardware operation unit running on the target hardware at the runtime to schedule the corresponding computing resources in the target hardware and complete the hardware acceleration of the machine learning model. Specifically, in step 106, the hardware abstraction layer dialect provides two core functions: at the compilation time, a complete IR and instruction describing the program execution process are provided. At the runtime, a Runtime API (written in C language) corresponding to the compilation time IR and instruction is provided. Then, according to the target hardware type (CPU / GPU / TPU), the Runtime API is implemented through the driving API of the target hardware, and the Stream Dialect is converted into the hardware abstraction layer dialect. Further, the binary file of the hardware operation unit is combined with the converted hardware abstraction layer dialect to form a complete neural network executable program. The runtime component completes the resource management (including on-chip memory allocation management, register scheduling, computing core task allocation) of the target hardware, task scheduling synchronization, and interaction with the host code through the Runtime API, drives the hardware operation unit to execute on the target hardware, and realizes the hardware acceleration of the machine learning model.

[0101] In the embodiments of the present application, through the target-independent optimization stage processing and adaptive hardware optimization, an AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures, so that the flexible allocation of computing resources in the target hardware is realized through the AI accelerator, the model deployment efficiency and running efficiency are effectively improved, the adaptation degree of the machine learning model and the hardware resources is improved, which helps fine optimization from the machine learning model to the hardware resources, and the parallel resource utilization rate is simultaneously improved, and the accelerator resource waste is avoided.

[0102] In still another embodiment of the present application, an AI accelerator construction device is also provided. Referring to Figure 2 As described above, the device comprises the following units:

[0103] The import unit is configured to import machine learning models of different AI frameworks through a multi-framework adaptation interface, parse the model representation of the machine learning model, and convert the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain;

[0104] The optimization unit is configured to, in the target-independent optimization stage, retain the core computing semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; in the hardware adaptive optimization stage, the target hardware characteristics of the target hardware are identified, the target hardware characteristics include: hardware type, hardware structure, and parallelism; the optimized intermediate representation is adaptively executed with hardware optimization processing matched with the target hardware characteristics to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion;

[0105] The construction unit is configured to convert the target intermediate representation into hardware operation units executable by the target hardware, load the hardware operation units into the target hardware, and construct an AI accelerator;

[0106] The AI accelerator is configured to connect the hardware abstraction layer dialect to the compile-time IR and the runtime to run the hardware operation units on the target hardware, schedule the corresponding computing resources in the target hardware, and complete the hardware acceleration of the machine learning model.

[0107] Further optionally, the importing unit is configured to, if the imported AI framework is PyTorch, trigger a Dynamo component to work through a torch.compile function of PyTorch 2.0, the Dynamo component dynamically modifies Python bytecode through a hook CPython frameevaluation API, rewrites the bytecode to extract an Aten operator sequence, and finally generates an FX computation graph; traverse the FX computation graph through a SHARK Turbine tool, parse data types and operator names of operators in the FX computation graph, and map the Aten operators to Torch dialect operators defined in a Torch MLIR tool one by one; and call a conversion interface of the Torch MLIR to convert the Torch dialect operators into an MLIRModuleOp in a linear algebra dialect representation, the MLIRModuleOp containing complete computation definitions of the machine learning model.

[0108] Further optionally, the optimization unit is configured to, in the target-independent optimization stage, retain core computation semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure, and is configured to: a scheduling creation layer traverses all blocks and block-in operators in an MLIR ModuleOp corresponding to the linear algebra dialect representation, identifies root operators in all blocks as optimization center points, and assigns a unique identifier kRootOpAttr to the root operators; through a fusion group attribute kFusionGroupsAttr, marks producer operators providing input for the root operators and consumer operators using output of the root operators, and fuses the producer operators and the consumer operators into the root operators; linear algebra dialect operators that are not marked as the root operators and do not belong to linalg.fill operators or dequantization operators are allocated to independent scheduling sections; and memory management optimization, redundant computation elimination, and data access pattern optimization are synchronously performed to generate the optimized intermediate representation.

[0109] Further optionally, the optimization unit is configured to fuse the producer operators and the consumer operators into the root operators by calling a fuseRootsWithConsumers function.

[0110] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to, after identifying the target hardware characteristics, fill the null value padding layer with the optimization intermediate representation as input, and traverse all linear algebra operator input tensors and output tensors therein: for tensors not meeting the target hardware characteristics, fill 0 using a tensor.pad operation; and for the filled tensors or tensors meeting the target hardware characteristics, extract valid data using a tensor.extract_slice operation, so that the tensor shape is aligned with the target hardware calculation mode while keeping the machine learning model output result unchanged.

[0111] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to, based on a TilingInterface interface, obtain an iteration domain of an operator in the optimization intermediate representation, calculate a tiling size and generate an external loop, generate a tiling operation implementation in an innermost loop, and allocate the cut operator to a plurality of work groups so that the amount of tensor data processed by each work group meets the memory limit of the target hardware; after the parallel allocation layer is executed, the BatchTile layer combines a private memory maximum value of the target hardware to calculate a tiling size set, so as to ensure that each dimension vector of each tensor is divided by the corresponding tiling size value in the tiling size set and the total memory occupancy after cutting is less than the private memory maximum value; the scf::tileUsingSCF interface is called to complete tiling and introduce For control flow to realize on-chip memory reuse and process thread-in data access of the Reduction operator; after the BatchTile layer is executed, the KernelTile layer splits complex calculations into sub-calculations matched with the hardware core according to the core calculation characteristics of the target hardware; the calculation definition and the memory transmission are segmented by using respective control flows, and the time overhead of the calculation definition and the memory transmission is balanced by the memory optimization of the BatchTile layer.

[0112] Further optionally, the optimization unit, after the memory allocation layer defines the SubSpanToAllocaPattern rewriting pattern, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and applies on-chip private memory of the target hardware, is configured to: according to whether the vector user exists, set the memory space attribute and the alignment mode, and insert the memref::CopyOp instruction to realize the data copy from the off-chip memory to the on-chip private memory; after the memory allocation layer is executed, the memory offset flattening layer calls the IREEExpandStridedMetadata Pass and the ExpandStridedMetadata Pass in turn, converts the high-dimensional memory access expression into one dimension through affine transformation, and establishes the memory copy mapping relationship from high dimension to low dimension.

[0113] Further optionally, the optimization unit, the memory allocation layer defines the SubSpanToAllocaPattern rewriting pattern, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and applies on-chip private memory of the target hardware, is configured to:

[0114] The initialization SubSpanToAllocaPattern rewriting mode; the rewriting mode inherits from RewritePattern, adapts to all operation types, and sets the optimization level; check if the operation to be processed in the optimization intermediate representation has the CopySource attribute and does not have the CopySourceDone attribute, if not, return a failure indication; if yes, check if the User of the operation has performed memref::CopyOp, if yes, set the CopySourceDone attribute for the operation and remove the CopySource attribute, return a success indication; check if there is a memref::CollapseShapeOp in the User of the operation to be processed, so as to judge whether the complex memory reference operation in the optimization intermediate representation needs to be replaced by a memref::AllocaOp, and update the operation pointer op synchronously; set the memory space attribute and the alignment according to whether the operation to be processed has a vector user hasVectorUser; if there is a vector user, use the high-speed vector memory space obtained by getVectorMemoryAttr; if not, use the standard memory space; create a new memref::AllocaOp to replace the original memory operation, and configure the memory type and the alignment of the memref::AllocaOp; insert a memref::CopyOp according to the read-only state of the memref::AllocaOp: if it is a read-only state, only insert a copy instruction from the result memory to the memref::AllocaOp; if it is a non-read-only state, in addition to the above copy instruction, additionally insert an instruction at the end of the operation to copy back from the memref::AllocaOp to the result memory; replace all uses of the original memory operation with the updated memref::AllocaOp, and ignore the User of the memref::AssumeAlignmentOp and the memref::CopyOp; if the updated memref::AllocaOp has a sub-view User, update the sub-view operation according to the parameters of the memref::AllocaOp.

[0115] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to: take the optimization intermediate representation as input to a vectorization layer, call an linalg::vectorize interface to identify parallelizable scalar operations therein, and convert the scalar operations into vector operations; rewrite memory access patterns to convert scalar memory loading into vector memory loading and convert scalar memory storage operations into vector memory storage operations; if the target hardware does not support vector operations, call an LinalgToLoops method to convert linear algebra operators in the optimization intermediate representation into multi-layer loops; after vectorization execution is completed, a vector flattening layer splits high-dimensional vector operators into multiple one-dimensional vector operators to make the obtained to-be-processed intermediate representation conform to an LLVM vector type, which is a compile-time abstraction of a target hardware vector register; and a software pipeline layer takes the vector-flattened to-be-processed intermediate representation as input, rearranges instructions in a loop body for a loop code, and obtains the target intermediate representation to make multiple different iteration instructions be executed in parallel in each loop iteration.

[0116] Further optionally, the construction unit is configured to convert the target intermediate representation into a hardware operation unit executable by the target hardware, and is configured to: take the target intermediate representation after the hardware optimization processing is completed as input, and perform a dialect step-by-step conversion process; take a linear algebra dialect in the target intermediate representation as a starting point, and sequentially pass through the following processing: a tensor dialect, one-time bufferization, a memory reference dialect, vectorization, a vector dialect, LLVM dialect conversion, and an LLVM dialect; convert the LLVM dialect file into an assembly file corresponding to the target hardware through an LLVM compiler, and construct the assembly file into a binary-form hardware language file and hardware operation instructions; and fuse the hardware language file and the hardware operation instructions into the hardware operation unit in combination with configuration information and hardware attribute information of the target hardware.

[0117] Further optionally, before the optimization unit adaptively performs hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, the optimization unit is further configured to: a device scheduling layer allocates tensors in the optimization intermediate representation to corresponding target hardware devices according to computing capabilities of the target hardware and tensor data volumes; and after entering a hardware adaptive optimization stage, create a stream bound to a device according to a target hardware device where a tensor is located; a stream bound to a GPU adapts a CUDA back end, and a stream bound to a CPU adapts a CPU back end; hardware optimization processing in each stream and generation of a subsequent hardware operation unit are matched with back end characteristics of the corresponding device; and the streams pass tensor data through a data copy operation containing a memref::CopyOp to ensure correctness of cross-device data interaction. The system can implement various steps in the method embodiments described above, which are not described here.

[0118] In the embodiments of the present application, the AI accelerator is constructed, and the AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures through target-independent optimization stage processing and adaptive hardware optimization, so as to flexibly deploy computing resources, effectively improve model deployment efficiency and running efficiency, improve the adaptation of machine learning models and hardware resources, help fine optimization from machine learning models to hardware resources, and simultaneously improve parallel resource utilization and avoid waste of accelerator resources.

[0119] Please refer to Figure 3 , Figure 3 The embodiment of the electronic device provided by the embodiments of the present application is shown. As shown in Figure 3 , the embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. The processor 520 implements each step in the above embodiments when executing the computer program 511. Please refer to Figure 4 , Figure 4 The embodiment of the computer-readable storage medium provided by the embodiments of the present application is shown. As shown in Figure 4 , the present embodiment provides a computer-readable storage medium 600, which stores a computer program 611. The computer program 611 is executed by a processor to implement each step in the above embodiments. In the above embodiments, the description of each embodiment is focused on different aspects. The parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0120] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they understand the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0121] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and variations.

Claims

1. A method for constructing an AI accelerator, characterized in that, The method includes: Machine learning models from different AI frameworks are imported through a multi-framework adaptation interface, the model representation of the machine learning model is obtained by parsing, and the model representation is converted into a linear algebra dialect representation with a unified form by using an MLIR dialect conversion chain. In the objective-independent optimization stage, the core computational semantics of the linear algebraic dialect are preserved through objective-independent optimization processing, and the linear algebraic dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure. The objective-independent optimization processing includes at least operator fusion, memory management optimization, reduction of redundant computation, and general data flow optimization. In the hardware adaptive optimization phase, the target hardware characteristics are identified, including: hardware type, hardware structure, and parallelism. The optimized intermediate representation is adaptively subjected to hardware optimization processing that matches the target hardware characteristics to obtain the target intermediate representation. The hardware optimization processing that matches the target hardware characteristics includes at least one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector transformation. The target intermediate representation is converted into a hardware operation unit executable by the target hardware, and the hardware operation unit is loaded into the target hardware to construct an AI accelerator. In the AI ​​accelerator, the hardware abstraction layer dialect provides compile-time IRs and instructions describing the program execution process, and runtime APIs that correspond one-to-one with the IRs and instructions at the runtime level. The runtime component drives the hardware operation unit to run on the target hardware through the runtime API, schedules the corresponding computing resources in the target hardware, and completes the hardware acceleration of the machine learning model. The process of adaptively performing hardware optimization processing on the optimized intermediate representation to match the characteristics of the target hardware, and obtaining the target intermediate representation, involves the memory allocation layer defining a SubSpanToAllocaPattern rewrite mode after the multi-layer tiling execution is completed. This mode performs memory optimization on the optimized intermediate representation, converting complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and allocating on-chip private memory for the target hardware. Depending on whether the operator has a vector user, memory space attributes and alignment are set, and memref::CopyOp instructions are inserted to implement data copying from off-chip memory to on-chip private memory. After the memory allocation layer completes its execution, the memory offset flattening layer calls IREEExpandStridedMetadataPass and ExpandStridedMetadataPass sequentially to convert the high-dimensional memory access expression into one dimension through affine transformation, establishing a memory copy mapping relationship from high dimension to low dimension.

2. The AI ​​accelerator construction method according to claim 1, characterized in that, The process involves importing machine learning models from different AI frameworks through a multi-framework adaptation interface, parsing the model representations of these models, and then using an MLIR dialect conversion chain to convert these model representations into a unified linear algebraic dialect representation. This includes: If the imported AI framework is PyTorch, the Dynamo component is triggered to work through the torch.compile function of PyTorch 2.

0. The Dynamo component dynamically modifies the Python bytecode by hooking the CPython frame evaluation API, rewrites the Python bytecode to extract the Aten operator sequence, and generates the FX computation graph. The FX computation graph is traversed using the SHARK Turbine tool to parse the data types and operator names of the operators in the FX computation graph, and the Aten operators are mapped one by one to the Torch dialect operators defined in the Torch MLIR tool. The Torch MLIR conversion interface is invoked to convert the Torch dialect operator into an MLIRModuleOp representing a linear algebra dialect. The MLIRModuleOp contains the complete computational definition of the machine learning model.

3. The AI ​​accelerator construction method according to claim 1, characterized in that, In the objective-independent optimization stage, the core computational semantics of the linear algebraic dialect are preserved through objective-independent optimization processing, and the linear algebraic dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure, including: The scheduling creation layer traverses all blocks and intra-block operators within the MLIR ModuleOp corresponding to the linear algebra dialect representation, identifies the root operator in all blocks that serves as the optimization center point, and assigns a unique identifier kRootOpAttr to the root operator. By using the fusion group attribute kFusionGroupsAttr, the producer operator that provides input to the root operator and the consumer operator that uses the output of the root operator are respectively marked, and the producer operator and the consumer operator are fused into the root operator; Linear algebraic dialect operators that are not marked as the root operator and do not belong to the linalg.fill operator or dequantization operator are assigned to independent scheduling segments; memory management optimization, redundant computation elimination and data access mode optimization are performed synchronously to generate the optimized intermediate representation.

4. The AI ​​accelerator construction method according to claim 3, characterized in that, The step of fusing the producer operator and the consumer operator into the root operator includes: The function fuseRootsWithConsumers is called to merge the consumer operators into the root operators.

5. The AI ​​accelerator construction method according to claim 1, characterized in that, The adaptive execution of hardware optimization processing on the optimized intermediate representation to match the target hardware characteristics, to obtain the target intermediate representation, includes: After identifying the target hardware characteristics, the null-filling layer takes the optimized intermediate representation as input and iterates through the input and output tensors of all linear algebra operators therein: For tensors that do not conform to the characteristics of the target hardware, the tensor.pad operation is used to pad them with zeros; for tensors that are padded or conform to the characteristics of the target hardware, the tensor.extract_slice operation is used to extract valid data, so that the shape of the tensor is aligned with the computation method of the target hardware while keeping the output of the machine learning model unchanged.

6. The AI ​​accelerator construction method according to claim 1, characterized in that, The adaptive execution of hardware optimization processing on the optimized intermediate representation to match the target hardware characteristics, to obtain the target intermediate representation, includes: Based on the TilingInterface interface, the iteration domain of the operator in the optimized intermediate representation is obtained in the parallel allocation layer. The tiling size is calculated and an outer loop is generated. The tiling operation is generated in the innermost loop. The operator is divided and allocated to multiple workgroups so that the amount of tensor data processed by each workgroup meets the memory limit of the target hardware. After the parallel allocation layer is completed, the BatchTile layer calculates the tile size set by combining the maximum private memory of the target hardware, ensuring that each dimension vector of each tensor is divisible by the corresponding tile size value in the tile size set and that the total memory usage after splitting is less than the maximum private memory. The scf::tileUsingSCF interface is called to complete the tiling and a For control flow is introduced to realize on-chip memory reuse, and the in-thread data access of the Reduction operator is processed. After the BatchTile layer is executed, the KernelTile layer breaks down complex calculations into sub-computations that match the core computing characteristics of the target hardware. The calculation definition and memory transfer are separated using their respective control flows. Through memory optimization in the BatchTile layer, the time overhead of calculation definition and memory transfer tends to be balanced.

7. The AI ​​accelerator construction method according to claim 1, characterized in that, The adaptive execution of hardware optimization processing on the optimized intermediate representation to match the target hardware characteristics, to obtain the target intermediate representation, includes: The vectorization layer takes the optimized intermediate representation as input, calls the linalg::vectorize interface to identify parallelizable scalar operations, and converts the scalar operations into vector operations; it synchronously rewrites the memory access mode, converting scalar memory loading into vector memory loading, and scalar memory storage operations into vector memory storage operations; if the target hardware does not support vector operations, it calls the LinalgToLoops method to convert the linear algebra operators in the optimized intermediate representation into multi-level loops; After vectorization is completed, the vector flattening layer splits the high-dimensional vector operator into multiple one-dimensional vector operators, so that the resulting intermediate representation to be processed conforms to the LLVM vector type. The LLVM vector type is abstracted at compile time from the target hardware vector register. The software pipeline layer takes the vector-flattened intermediate representation to be processed as input, rearranges the instructions in the loop body for the loop code, and obtains the target intermediate representation, so that the instructions of multiple different iterations are executed in parallel in each loop iteration.

8. The AI ​​accelerator construction method according to claim 1, characterized in that, The hardware operation unit that converts the target intermediate representation into a target hardware executable includes: Taking the target intermediate representation after hardware optimization as input, a dialect step-by-step conversion process is executed. Starting with the linear algebra dialect in the target intermediate representation, the following processes are performed in sequence: tensor dialect, one-time buffering, memory reference dialect, vectorization, vector dialect, LLVM dialect conversion, and LLVM dialect. The LLVM dialect file is converted into an assembly file corresponding to the target hardware using the LLVM compiler. The assembly file is then used to construct a hardware language file and hardware operation instructions in binary form. The hardware language file and hardware operation instructions are then combined with the configuration information and hardware attribute information of the target hardware to form the hardware operation unit.

9. The AI ​​accelerator construction method according to claim 1, characterized in that, Before adaptively performing hardware optimization processing to match the target hardware characteristics on the optimized intermediate representation to obtain the target intermediate representation, the process further includes: The device scheduling layer allocates the tensors in the optimized intermediate representation to the corresponding target hardware devices based on the computing power and tensor data volume of the target hardware. After entering the hardware adaptive optimization phase, a stream bound to the target hardware device where the tensor is located is created; if the stream is bound to the GPU, it is adapted to the CUDA backend, and if the stream is bound to the CPU, it is adapted to the CPU backend; the hardware optimization processing within each stream and the generation of subsequent hardware operation units are matched with the backend characteristics of the corresponding device. Tensor data is passed between streams via a data copy operation that includes memref::CopyOp, ensuring the correctness of cross-device data interaction.

10. An AI accelerator construction device, characterized in that, The device includes the following units, wherein, The import unit is configured to import machine learning models from different AI frameworks through a multi-framework adaptation interface, parse the model representation of the machine learning model, and use the MLIR dialect conversion chain to convert the model representation into a linear algebra dialect representation with a unified form. The optimization unit is configured to, in the objective-independent optimization phase, retain the core computational semantics of the linear algebraic dialect through objective-independent optimization processing, and reconstruct the linear algebraic dialect representation into an optimized intermediate representation with an efficient data flow structure; the objective-independent optimization processing includes at least operator fusion, memory management optimization, reduction of redundant computation, and general data flow optimization; in the hardware adaptive optimization phase, the target hardware characteristics are identified, including: hardware type, hardware structure, and parallelism; the optimized intermediate representation is adaptively subjected to hardware optimization processing that matches the target hardware characteristics to obtain the target intermediate representation, and the hardware optimization processing that matches the target hardware characteristics includes at least one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector transformation; The building unit is configured to convert the target intermediate representation into a hardware operation unit executable by the target hardware, load the hardware operation unit into the target hardware, and build an AI accelerator. The AI ​​accelerator is configured to provide a compile-time IR and instructions describing the program execution process through a hardware abstraction layer dialect, and a runtime API that corresponds one-to-one with the IR and instructions at the runtime level. The runtime component drives the hardware operation unit to run on the target hardware through the runtime API, schedules the corresponding computing resources in the target hardware, and completes the hardware acceleration of the machine learning model. The optimization unit is also used to adaptively perform hardware optimization processing that matches the characteristics of the target hardware on the optimized intermediate representation. After obtaining the target intermediate representation, after the multi-layer tiling execution is completed, the memory allocation layer defines the SubSpanToAllocaPattern rewrite mode to perform memory optimization on the optimized intermediate representation, converting the complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and requesting on-chip private memory of the target hardware. According to whether the operator has a vector user, the memory space attributes and alignment are set, and memref::CopyOp instructions are inserted to implement data copying from off-chip memory to on-chip private memory. After the memory allocation layer is executed, the memory offset flattening layer calls IREEExpandStridedMetadata Pass and ExpandStridedMetadata Pass in sequence to convert the high-dimensional memory access expression into one dimension through affine transformation, and establish a memory copy mapping relationship from high dimension to low dimension.

11. An electronic device, characterized in that, include: Memory, used to store computer software programs; A processor for reading and executing the computer software program, thereby implementing the AI ​​accelerator construction method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer software program, which, when executed by a processor, implements the AI ​​accelerator construction method as described in any one of claims 1-9.

13. A chip, characterized in that, The chip is loaded with computer software programs and / or the chip is provided with hardware units, the computer software programs and / or hardware units being used to implement the AI ​​accelerator construction method as described in any one of claims 1-9.

Images