AI accelerator construction method and related device

Abstract

The invention relates to the field of artificial intelligence, in particular to an AI accelerator construction method and a related device. The method comprises the following steps: importing machine learning models of different AI frames through a multi-frame adaptation interface, and converting the machine learning models into linear algebraic dialect representation by adopting an MMIR dialect conversion chain; through target-independent optimization processing, core calculation semantics in linear algebraic dialects are reserved, and linear algebraic dialect representation is reconstructed into optimization intermediate representation with an efficient data flow structure; identifying target hardware characteristics of the target hardware; adaptively executing hardware optimization processing matched with target hardware characteristics on the optimization intermediate representation; and converting the target intermediate representation into a hardware operation unit executable by the target hardware, loading the hardware operation unit into the target hardware to obtain an AI accelerator, thereby running the hardware operation unit on the target hardware through the AI accelerator, scheduling corresponding computing resources in the target hardware, and completing hardware acceleration of the machine learning model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an AI accelerator construction method and related device. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the complexity of machine learning models is exponentially increasing, and the demand for computing resources is also increasing. This trend has driven the rapid development and popularization of special hardware accelerators, such as graphics processing units (GPUs), tensor processing units (TPUs), and field programmable gate arrays (FPGAs). Such hardware, with its high parallelism and high computing power density, has become the core infrastructure supporting the efficient operation of complex models. Currently, AI accelerator construction and adaptation schemes in the prior art are mainly implemented using XLA, TVM, and IREE. However, existing technical solutions are usually deeply bound to a single machine learning framework, making it difficult to achieve unified import and compilation of models in multiple frameworks such as TensorFlow, PyTorch, and JAX, resulting in the need for additional development of adaptation tools for cross-framework deployment, increasing development costs and deployment complexity. The existing technical solutions lack coordination in the intermediate representation (IR) transformation and optimization stages, making it difficult to form a coherent optimization chain from high-level models to hardware instructions, and unable to implement fine-grained optimization based on hardware characteristics such as parallelism and memory hierarchy. In addition, the existing technical solutions lack a tensor shape adjustment mechanism for hardware characteristics, which cannot efficiently adapt to reduce additional computational overhead, and lack efficient allocation and reuse strategies for on-chip memory, resulting in low utilization of parallel resources and inability to fully utilize the parallel computing capabilities of hardware. In summary, there is an urgent need for a technical solution to overcome at least one of the technical problems of high model deployment complexity, inability to fine-tune, low adaptation efficiency, and low parallel resource utilization in the prior art. SUMMARY

[0003] The present application addresses the technical problems in the prior art and provides an AI accelerator construction method and related device to solve the technical problems of high model deployment complexity, inability to fine-tune, low adaptation efficiency, and low parallel resource utilization in related technologies.

[0004] In a first aspect, an AI accelerator construction method is provided, which includes: Importing machine learning models of different AI frameworks through a multi-framework adaptation interface, parsing the model representation of the machine learning model, and converting the model representation into a formally unified linear algebra dialect representation using an MLIR dialect conversion chain. In the target-independent optimization stage, through target-independent optimization processing, core computing semantics in the linear algebra dialect are retained, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization. In the hardware adaptive optimization stage, target hardware characteristics of the target hardware are identified, the target hardware characteristics including: hardware type, hardware structure, and parallelism degree; The hardware optimization processing matched with the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion; The target intermediate representation is converted into a hardware operation unit executable by the target hardware, and the hardware operation unit is loaded into the target hardware to construct an AI accelerator; In the AI accelerator, the hardware operation unit is run on the target hardware through a hardware abstraction layer dialect to connect the IR at the compilation time and the runtime, and the corresponding computing resources in the target hardware are scheduled to complete hardware acceleration of the machine learning model.

[0005] In a second aspect, an AI accelerator construction device is provided, and the device includes the following units, wherein, The import unit is configured to import a machine learning model of different AI frameworks through a multi-framework adaptation interface, parse a model representation of the machine learning model, and convert the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain; The optimization unit is configured to, in a target-independent optimization stage, retain core computing semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; in a hardware adaptive optimization stage, target hardware characteristics of the target hardware are identified, the target hardware characteristics including: hardware type, hardware structure, and parallelism degree; and the hardware optimization processing matched with the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion; The construction unit is configured to convert the target intermediate representation into a hardware operation unit executable by the target hardware, load the hardware operation unit into the target hardware, and construct an AI accelerator; The AI accelerator is configured to connect the compile-time IR and the runtime by a hardware abstraction layer dialect, run the hardware operation unit on the target hardware, schedule corresponding computing resources in the target hardware, and complete hardware acceleration of the machine learning model.

[0006] In a third aspect, an electronic device is provided, and the electronic device includes: at least one processor, a memory, and an input / output unit; The memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to execute the AI accelerator construction method of the first aspect.

[0007] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to execute the AI accelerator construction method of the first aspect.

[0008] The beneficial effects of the present application are as follows: an AI accelerator construction method and related devices are provided. In the technical solution, first, a machine learning model of different AI frameworks is imported through a multi-framework adaptation interface, a model representation of the machine learning model is parsed, and the model representation is converted into a formal unified linear algebra dialect representation by using an MLIR dialect conversion chain. Then, in the target-independent optimization stage, the core computing semantics in the linear algebra dialect are retained by target-independent optimization processing, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure. The target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization. Then, in the hardware adaptive optimization stage, the target hardware characteristics of the target hardware are identified, and the target hardware characteristics include: hardware type, hardware structure, and parallelism. The optimized intermediate representation is adaptively executed by hardware optimization processing that matches the target hardware characteristics to obtain a target intermediate representation. The hardware optimization processing that matches the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion. Finally, the target intermediate representation is converted into a hardware operation unit executable by the target hardware, the hardware operation unit is loaded into the target hardware, and an AI accelerator is constructed. In the AI accelerator, the compile-time IR and the runtime are connected by a hardware abstraction layer dialect, the hardware operation unit is run on the target hardware, the corresponding computing resources in the target hardware are scheduled, and hardware acceleration of the machine learning model is completed.

[0009] The technical scheme of the present application constructs an AI accelerator containing personalized hardware operation units for machine learning models of different AI architectures through target-independent optimization stage processing and adaptive hardware optimization, thereby realizing flexible allocation of computing resources in the target hardware through the AI accelerator, effectively improving model deployment efficiency and running efficiency, improving the adaptation of machine learning models and hardware resources, helping fine optimization from machine learning models to hardware resources, and simultaneously improving parallel resource utilization and avoiding accelerator resource waste. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of an AI accelerator construction method according to an embodiment of the present application; Figure 2 is a structural diagram of an AI accelerator construction device according to an embodiment of the present application; Figure 3 is a structural diagram of an electronic device according to an embodiment of the present application; Figure 4 is a structural diagram of a medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0012] An AI accelerator is a hardware unit specially designed for artificial intelligence applications, aiming to accelerate the execution of artificial intelligence algorithms and improve computing efficiency and performance. AI accelerators usually adopt a special hardware architecture and instruction set, which is optimized for the characteristics of artificial intelligence algorithms to realize efficient parallel computing.

[0013] Machine learning models (such as deep learning neural networks) are increasingly complex, and the demand for computing resources is growing exponentially. For example, a large number of matrix multiplication, Reduction reduction operations, and vector transformation operations in the model, general-purpose CPUs are difficult to adapt efficiently due to the generalization of architecture (need to consider various tasks), resulting in low execution efficiency and high energy consumption. In this context, AI accelerators have emerged, which complete the core computing needs of AI tasks through customized structures at the hardware level, becoming the key hardware foundation for supporting complex AI model landing (such as large language model inference, computer vision training). Currently, the AI accelerator construction and adaptation schemes in the prior art mainly use XLA, TVM, and IREE to realize this scheme, which mainly has the following technical problems: Firstly, the prior art solution is usually deeply bound to a single machine learning framework, making it difficult to implement unified import and compilation of multi-framework models such as TensorFlow, PyTorch, and JAX, resulting in the need for additional development of adaptation tools for cross-framework deployment, increasing development costs and deployment complexity. At the same time, the prior art solution does not support emerging customized hardware such as domestic TPU, lacks complete software support solutions for the characteristics of such hardware, and cannot fully utilize the computing advantages of customized hardware, resulting in ineffective release of hardware computing power.

[0014] Secondly, the intermediate representation (IR) transformation and optimization link of the prior art solution is loose, and target-independent optimization (such as improving code generality and reducing unnecessary recalculation) and target-specific optimization (such as adapting hardware parallelism and memory hierarchy) lack coordination, making it difficult to form a coherent optimization chain from high-level models to hardware instructions, and unable to achieve fine-grained optimization for hardware parallelism, memory hierarchy structure, and other characteristics.

[0015] Thirdly, the prior art solution lacks a tensor shape adjustment mechanism for hardware characteristics. For tensor shapes that do not meet the hardware computing definition (such as memory transfer and fixed vector length restrictions), it is difficult to efficiently adapt to reduce additional computational overhead, and there is a lack of flattening processing of high-dimensional memory access, efficient allocation and reuse strategies of on-chip memory, which can easily cause memory bandwidth bottlenecks. Moreover, multi-device collaboration (such as data interaction between different hardware) and asynchronous task scheduling optimization are insufficient, resulting in low utilization of parallel resources and inability to fully utilize the parallel computing capabilities of hardware.

[0016] To solve at least one of the technical problems in the prior art, the embodiments of the present application provide an AI accelerator construction method and related devices. In the technical solution provided by the present application, the target-independent optimization stage processing and adaptive hardware optimization are used to adaptively construct an AI accelerator containing personalized hardware operation units for machine learning models of different AI architectures, so as to realize flexible allocation of computing resources in the target hardware through the AI accelerator, effectively improve the model deployment efficiency and running efficiency, improve the adaptation degree of machine learning models and hardware resources, help fine optimization from machine learning models to hardware resources, and simultaneously improve the utilization rate of parallel resources, avoiding waste of accelerator resources.

[0017] The AI accelerator construction scheme provided by the embodiments of the present application can also be executed by an electronic device, which can be a server, a server cluster, a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a special-purpose device (such as a special-purpose terminal device with an AI accelerator construction method system, etc.). The above-mentioned chip can also be mounted in these electronic devices. Alternatively, these electronic devices can also install a service program for executing the AI accelerator construction scheme.

[0018] Figure 1 A flowchart of an AI accelerator construction method provided in an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1 101, importing machine learning models of different AI frameworks through a multi-framework adaptation interface, parsing to obtain a model representation of the machine learning model, and converting the model representation into a linear algebra dialect representation in a form uniform by using an MLIR dialect conversion chain; 102, in a target-independent optimization stage, performing target-independent optimization processing, retaining core computing semantics in the linear algebra dialect, and reconstructing the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; 103, in a hardware adaptive optimization stage, identifying target hardware characteristics of the target hardware; 104, performing hardware optimization processing matching the target hardware characteristics on the optimized intermediate representation to obtain a target intermediate representation; 105, converting the target intermediate representation into hardware operation units executable by the target hardware, loading the hardware operation units into the target hardware, and constructing an AI accelerator; 106, in the AI accelerator, connecting the hardware abstraction layer dialect to the IR at compile time and the hardware operation units running on the target hardware at runtime to schedule corresponding computing resources in the target hardware, and completing hardware acceleration of the machine learning model.

[0019] Through steps 101 to 106, the target-independent optimization stage processing and adaptive hardware optimization are used to adaptively construct an AI accelerator containing personalized hardware operation units for machine learning models of different AI architectures, so that the AI accelerator realizes flexible allocation of computing resources in the target hardware, effectively improves the model deployment efficiency and running efficiency, improves the adaptation of machine learning models and hardware resources, helps fine optimization from machine learning models to hardware resources, and simultaneously improves the parallel resource utilization rate, avoiding waste of accelerator resources.

[0020] In an embodiment of the present application, the machine learning models of different AI frameworks can be various deep learning and machine learning models constructed under mainstream AI frameworks such as TensorFlow, PyTorch, JAX, etc., covering AI task scenarios such as image classification, feature extraction, data prediction, etc.

[0021] ​Among them, the model of the PyTorch framework can include a lightweight neural network built based on PyTorch 2.0, such as an image classification model containing convolutional layers, batch normalization layers, ReLU activation layers, and fully connected layers. When imported, such a model will trigger the Dynamo component processing through the torch.compile function of PyTorch 2.0, and finally generate an FX computation graph and further convert it to a unified IR.

[0022] The model of the TensorFlow framework can include a sequential model or a functional model built using the Keras API, such as a recurrent neural network (RNN) model for text classification or a convolutional neural network (CNN) model for image segmentation. Such a model will parse its computation graph structure through the TensorFlow interface adapted by the system front end, and extract the corresponding operator sequence.

[0023] The model of the JAX framework can include a complex numerical calculation model built based on the automatic differentiation feature of JAX, such as a prediction model for scientific computing combined with AI. Such a model will parse its functional calculation logic through the front-end JAX adaptation interface and convert it into a recognizable model representation. These models from different frameworks, regardless of the specific tasks they involve or the operators they contain (such as matrix multiplication, Reduction operators, convolution operators, activation operators, etc.), will ultimately be converted into MLIR ModuleOp represented by the linear algebra dialect through the MLIR dialect conversion chain. This ensures that subsequent optimization and code generation processes can standardize the processing of various models, and achieve efficient execution on target hardware such as TPU.

[0024] As an optional embodiment, in step 101, the machine learning model of different AI frameworks is imported through the multi-framework adaptation interface, the model representation of the machine learning model is parsed, and the model representation is converted into a formally unified linear algebra dialect representation using the MLIR dialect conversion chain, including: If the imported AI framework is PyTorch, the Dynamo component is triggered to work through the torch.compile function of PyTorch 2.0, and the Dynamo component dynamically modifies the Python bytecode by hooking the frame evaluation API of CPython, rewrites the bytecode to extract the Aten operator sequence, and finally generates the FX computation graph. Further, the FX computation graph is traversed by the SHARK Turbine tool to parse the data types and operator names of the operators in the FX computation graph, and the Aten operators are mapped one by one to the Torch dialect operators defined in the Torch MLIR tool. Finally, the conversion interface of Torch MLIR is called to convert the Torch dialect operators into MLIR ModuleOps represented by linear algebra dialect, which contains the complete computational definition of the machine learning model.

[0025] Specifically, in the optional embodiments of the present application, for the import and unified IR conversion of the PyTorch framework model, it is necessary to rely on the cooperation of PyTorch ecological tools and MLIR dialect chain to realize the complete conversion from Python code to linear algebra dialect representation, and the whole process needs to ensure that the operator semantics are not lost and are adapted to the subsequent unified optimization process. When the imported AI framework is PyTorch, first start the model parsing and operator extraction process, and call the torch.compile function provided by PyTorch 2.0, which will automatically trigger the built-in Dynamo component of PyTorch to work. The core function of the Dynamo component is to break through the limitations of Python static analysis, and dynamically intercept and modify Python bytecode by hooking the frame evaluation API of the CPython interpreter during model execution. For example, when processing a PyTorch image classification lightweight model containing a convolution layer and a ReLU activation layer, Dynamo will analyze the bytecode line by line during the execution of the model, identify the instructions related to tensor operations, and rewrite the scattered and non-operatorized bytecode fragments to integrate them into a continuous Aten operator sequence. Aten is the core operator library at the bottom of PyTorch, which covers basic operations such as matrix multiplication, convolution, and activation. Finally, the FX computation graph is generated. The FX computation graph clearly records the execution order of each Aten operator in the model, the input-output relationship, and the tensor dimension information, such as in the above lightweight model, the FX computation graph records the calling relationship of aten::conv2d (convolution operator) and aten::relu (activation operator) in turn, as well as the input tensor shape and data type of each operator.

[0026] After the completion of the FX computation graph generation, the dialect mapping link from the PyTorch ecosystem to the MLIR ecosystem is entered, and at this time the SHARK Turbine tool needs to be used. This tool is a conversion bridge specially connecting PyTorch and MLIR, and can deeply analyze the structure and semantics of the FX computation graph. SHARK Turbine will traverse each node (i.e. Aten operator) in the FX computation graph, first analyze the key information of each operator, including but not limited to the data types of input and output tensors (such as float32, int64), operator names (such as aten::conv2d, aten::batch_norm), and attribute parameters of the operator (such as the convolution kernel size and step of the convolution operator), and then according to the pre-set operator mapping rule, each Aten operator is matched one by one to the Torch dialect operator predefined in the Torch MLIR tool. Taking the aten::conv2d operator in the above lightweight model as an example, SHARK Turbine will identify the convolution kernel size, input channel number and other attributes of the operator, and accurately map it to the Torch::conv2d operator defined in the Torch dialect of Torch MLIR, ensuring that the calculation logic and parameter information of the operator are completely corresponding, and avoiding semantic deviation caused by dialect conversion.

[0027] Finally, the unified IR generation link is entered, and the mapped Torch dialect operator is further converted into an MLIR ModuleOp represented by the linear algebra dialect (Linalg Dialect) by calling the conversion interface of Torch MLIR. Torch MLIR will first perform semantic analysis on the Torch dialect operator, convert the operator logic specific to the Torch dialect (such as the automatic broadcast mechanism specific to PyTorch) into general linear algebra operation logic, and then generate an operator representation conforming to the Linalg dialect specification. For example, the Torch::conv2d operator in the above lightweight model will be converted into the Linalg::conv_2d operator of the Linalg dialect, and the Torch::relu operator will be converted into the Linalg::relu operator, and these Linalg operators will be organized into an MLIR ModuleOp. ModuleOp is the basic unit of MLIR that represents a complete computing program, including all computing definitions of the lightweight model from receiving data at the input layer, through convolution and activation operations, to output classification results, and this linear algebra dialect representation is platform-independent and can be directly used as a unified input carrier for subsequent target-independent optimization and hardware adaptive optimization.

[0028] After obtaining the linear algebraic dialect representation, in step 102, during the objective-independent optimization stage, the core computational semantics of the linear algebraic dialect are preserved through objective-independent optimization processing, and the linear algebraic dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure.

[0029] In the embodiments of this application, the target-independent optimization process includes at least operator fusion, memory management optimization, reduction of redundant calculations, and general data flow optimization.

[0030] As an optional embodiment, in step 102, the scheduling creation layer traverses all blocks and intra-block operators within the MLIR ModuleOp corresponding to the linear algebraic dialect representation, identifies the root operator in all blocks as the optimization center point, and assigns a unique identifier kRootOpAttr to the root operator. Using the fusion group attribute kFusionGroupsAttr, producer operators providing input to the root operator and consumer operators using the output of the root operator are marked respectively, and the producer and consumer operators are merged into the root operator. Finally, linear algebraic dialect operators that are not marked as root operators and do not belong to the linalg.fill or dequantization operators are assigned to independent scheduling segments. Simultaneously, memory management optimization, redundant computation elimination, and data access mode optimization are performed to generate the optimized intermediate representation.

[0031] Specifically, in step 102, the core task of the scheduling creation layer is to sort out and optimize the operator relations of the MLIRModuleOp of the linear algebra dialect representation, and at the same time complete the general optimization to generate the optimized intermediate representation. The whole process must ensure the semantic integrity of the operators and the improvement of data flow efficiency.

[0032] First, the scheduling creation layer completely traverses all blocks contained in the MLIR ModuleOp and the operators within each block. During the traversal, it analyzes the computational logic, input-output tensor relationships, and data dependencies of each operator line by line to determine which operators are suitable as root operators for optimization. The selection of root operators is usually based on their core position in the computation flow; they are generally operators with large computational loads and significant impacts on overall execution efficiency. For example, when processing convolutional neural network models for image classification, the two-dimensional convolution operator (Linalg::conv_2d) in the model is selected as the root operator because convolution operations often occupy most of the model's computational resources, and using it as the core for fusion can minimize the data transfer overhead between operators. After determining the root operators, the scheduling creation layer assigns a unique kRootOpAttr identifier to each root operator. This identifier is used to clearly distinguish the root operator from other operators and avoid confusion in operator attribution during the fusion process.

[0033] Subsequently, the operator fusion link is entered, and the scheduling creation layer establishes the association of the root operator and other operators through the fusion group attribute kFusionGroupsAttr. First, the producer operator that provides the input data for the root operator is identified according to the data dependency relationship, and then the consumer operator that directly uses the output data of the root operator is found. Then, the two types of operators are gradually fused into the calculation unit where the root operator is located. Taking the two-dimensional convolution operator mentioned just now as the root operator as an example, if the input data of the convolution operator comes from the output of the batch normalization operator (Linalg::batch_norm), then the batch normalization operator is the producer operator. The output data of the convolution operator will be transmitted to the ReLU activation operator (Linalg::relu) for nonlinear transformation, and the ReLU activation operator is the consumer operator. The scheduling creation layer will mark the batch normalization operator and the ReLU activation operator as belonging to the same fusion group as the convolution operator through kFusionGroupsAttr. After that, the mean and variance calculation of the batch normalization, the feature extraction calculation of the convolution, and the nonlinear activation calculation of the ReLU are integrated together to form a fused composite operator. In this way, the intermediate tensors that need to be transmitted between three independent operators (batch normalization output to convolution input, convolution output to ReLU input) can be directly transmitted within the fused operator, without the need for frequent reading and writing of intermediate data in memory, greatly reducing the data transfer time and memory access overhead between operators.

[0034] After the fusion of the root operator and the related operators is completed, the scheduling creation layer will process the linear algebra dialect operators that are not marked as root operators. For these operators, the linalg.fill operator and the dequantization operator are first excluded, because the linalg.fill operator is usually used to initialize tensor data (such as filling boundary values for the input feature matrix of the convolution layer or initializing zero values for the weight matrix of the fully connected layer), and the dequantization operator is used to restore the low-precision tensor data (such as INT8) quantized and compressed in the training stage to high-precision data (such as FP32) required for inference. The calculation logic of these two types of operators is relatively simple, and the calculation amount is small. Separately allocating a scheduling section for them will increase the scheduling management overhead, which is not conducive to overall efficiency improvement, and therefore it is more suitable to be executed together with the associated operators before and after. For the remaining operators that are not marked as root operators and do not belong to the above two types, such as the max pooling operator (Linalg::max_pool_2d) used for downsampling in the model, the scheduling creation layer will allocate them to independent scheduling sections, and each independent scheduling section corresponds to a separate calculation task unit, ensuring that the execution of these types of operators will not be disturbed by other fused operators, and also facilitating subsequent differentiated optimization for the characteristics of different scheduling sections.

[0035] Finally, while completing the operator fusion and scheduling section allocation, the scheduling creation layer synchronously performs memory management optimization, redundant computation elimination, and data access pattern optimization, and finally generates an optimized intermediate representation. Memory management optimization adjusts the storage layout and access order of data in memory according to the memory access requirements of different operators, such as adjusting the frequently accessed feature tensor of the fused convolution, batch normalization, and ReLU composite operator to a continuous storage mode, so that the hardware cache can more efficiently read data and reduce the probability of cache miss failure. Redundant computation elimination identifies repeated computation steps in the model by analyzing the entire ModuleOp computation process, such as a fixed weight matrix repeatedly participating in matrix multiplication in multiple loop iterations, and only computing the weight matrix once and caching the result, and directly reusing the cached data in subsequent iterations to avoid resource waste caused by repeated computation. Data access pattern optimization sorts the data dependency order between different scheduling sections, arranges operators with close data dependencies in adjacent execution steps, such as directly obtaining the input data of the max pooling operator in the independent scheduling section from the output results of the previous fused operator, reducing the number of data migrations between different memory areas and delays. Taking the entire image classification model as an example, after these optimizations, the optimized intermediate representation not only retains the core computation semantics of the linear algebra dialect, but also significantly improves the data flow efficiency through operator fusion and general optimization, providing an efficient and unified input carrier for subsequent adaptive optimization for target hardware such as TPU, GPU, etc.

[0036] Further optionally, in the step of fusing the producer operator and the consumer operator into the root operator, the fuseRootsWithConsumers function is called to fuse the consumer operator into the root operator. It can be understood that, in actual application, in the AI accelerator code generation system based on MLIR and IREE, the process of calling the fuseRootsWithConsumers function to fuse the consumer operator into the root operator is implemented by relying on the accurate analysis of operator data dependency and the dynamic adjustment of IR structure by the scheduling creation layer, and the core goal is to eliminate the intermediate data transmission overhead between the root operator and the consumer operator, while ensuring the integrity of the calculation logic after fusion. Before execution, the function has completed the identification and marking of the root operator (assigned kRootOpAttr) through the previous process, and the output tensor association relationship of the root operator is clear. Therefore, the function first takes the output tensor of the root operator as the tracking basis, traverses the operator dependency link in the corresponding block in the MLIR ModuleOp, and locates all consumer operators that directly use the output of the root operator as input. For example, when processing a neural network model containing a convolution layer and an activation layer, if the root operator is a two-dimensional convolution operator in the linear algebra dialect (Linalg::conv_2d), the output tensor will be directly passed to the ReLU activation operator (Linalg::relu), and at this time the fuseRootsWithConsumers function will accurately identify that the ReLU operator is the consumer operator that needs to be fused through the tensor usage relationship graph.

[0037] After locating the consumer operator, the fuseRootsWithConsumers function further analyzes the calculation semantics and parameter attributes of the root operator and the consumer operator, and judges the compatibility of their fusion. The input of the consumer operator is checked to see if it only comes from the output of the root operator (without other external data dependency), and the calculation logic of the consumer operator is checked to see if it can be embedded in the execution flow of the root operator (such as activation operators, pooling operators, etc. Light calculation operators usually have such compatibility). If the fusion condition is met, the function will start the reconstruction of the IR structure: first, the calculation logic of the consumer operator is disassembled into the basic operation steps supported by the linear algebra dialect, and then these steps are sequentially embedded at the end of the calculation flow of the root operator to form a unified composite calculation unit. For example, the ReLU activation operator, whose calculation logic is to retain the original value if greater than 0 and set it to 0 if less than 0, will be disassembled into comparison and selection operations and directly connected after the feature extraction calculation of the convolution operator, so that the convolution output tensor can participate in the ReLU operation without writing to the memory, saving the redundant step of writing the output of the root operator to the memory and reading the data from the memory by the consumer operator.

[0038] Meanwhile, the fuseRootsWithConsumers function synchronously processes the updates of the operator attributes and the scheduling markers. The attribute parameters of the consumer operator (such as the activation threshold of the ReLU operator) are integrated into the attribute set of the root operator, ensuring that the calculation parameters are not lost after fusion; and by updating the kFusionGroupsAttr marker, the consumer operator is formally incorporated into the fusion group to which the root operator belongs, and the association between the two is made clear. For independent consumer operator nodes that are no longer needed after fusion, the function triggers the cleaning operation of IR to remove these redundant nodes to maintain the simplicity of the MLIR ModuleOp structure and avoid invalid operators interfering in the subsequent optimization process. In addition, the function also coordinates with the memory management related modules to adjust the memory access mode of the fused operator. For example, the reading operation of the original consumer operator on the output tensor of the root operator is changed to directly reference the calculation result of the root operator at the register level, further reducing the memory access delay.

[0039] Taking the image feature extraction task as an example, if the root operator is a convolution operator for processing the input feature map, and the consumer operator is a subsequent batch normalization operator (Linalg::batch_norm), the fuseRootsWithConsumers function will first confirm that only the feature map data comes from the convolution operator output in the input mean and variance parameters of the batch normalization operator, without other external dependencies, and then embed the mean calculation and variance adjustment steps of the batch normalization into the execution process of the convolution operator, so that the feature map generated by the convolution is directly used for batch normalization processing without the need for intermediate transfer in memory. At the same time, the kFusionGroupsAttr marker of the root operator is updated to include the batch normalization operator in the fusion group, and the original independent batch normalization operator node is cleaned up. Through a series of operations, the function finally realizes the deep fusion of the root operator and the consumer operator, forms a linear algebra dialect operator with convolution and batch normalization composite functions, not only retains the original calculation semantics, but also significantly optimizes the data flow efficiency, providing a more efficient IR basis for subsequent memory management, redundant calculation elimination and other steps in target-independent optimization.

[0040] In step 103, in the hardware adaptive optimization phase, the target hardware characteristics of the target hardware are identified.

[0041] In the embodiments of the present application, the target hardware characteristics include: hardware type, hardware structure, and parallelism. Further optionally, the hardware optimization processing matched with the target hardware characteristics at least includes one of: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion.

[0042] It can be understood that the identification of the target hardware characteristics relies on the interaction interface of the system and the hardware and the preset hardware information library, and provides accurate hardware characteristic basis for subsequent hardware optimization processing through multi-dimensional information collection and analysis. The identification of the hardware type is mainly realized through the interaction of the hardware abstraction layer dialect of the system and the target hardware driver: the system calls the hardware backend driver interface (such as the CUDA driver of GPU, the OpenMP driver of CPU, and the exclusive driver of TPU), obtains the device identification information of the hardware, such as the hardware model and the architecture code returned by the driver, to distinguish whether the target hardware is CPU, GPU or TPU; at the same time, the preset hardware type mapping table is combined to confirm the specific category of the hardware and the corresponding optimization adaptation rule, such as identifying the hardware model as NVIDIA A100, determining it as a GPU type supporting Tensor Cores, and identifying the TPU as its exclusive systolic array optimization strategy, to ensure that the subsequent optimization direction is completely aligned with the hardware type.

[0043] The identification of the hardware structure focuses on the memory level and the computing core layout of the hardware. The system will obtain the memory distribution information and the computing core characteristics of the target hardware through the driver API or the hardware specification query interface. In terms of memory structure, the multi-level memory composition of the hardware will be determined, such as whether it contains on-chip private memory, on-chip shared memory and off-chip high-speed memory (such as HBM), and the capacity, access speed and data transmission bandwidth of each level of memory will be obtained, for example, when identifying GPU, the size of the on-chip shared memory and the bandwidth of the off-chip HBM will be confirmed, and when identifying TPU, the maximum capacity and access delay of the on-chip private memory will be obtained. In terms of computing core structure, the core type and layout of the hardware will be analyzed, such as whether the GPU is equipped with Tensor Cores, whether the TPU adopts the systolic array structure, and whether the core of the CPU supports the vector instruction set (such as AVX-512). These structure information directly determines the direction of subsequent memory optimization (such as on-chip memory allocation) and computing division (such as sub-matrix operation adapted to Tensor Cores).

[0044] The identification of parallelism mainly relies on the concurrent execution capability supported by hardware. Through driving query or hardware performance interface, the parallel computing parameters of hardware are obtained, including but not limited to the maximum number of workgroups supported by hardware, the number of threads that each workgroup can accommodate, the total number of computing cores, etc. For example, when identifying GPU, the maximum number of thread blocks supported by GPU and the maximum number of threads in each thread block are obtained. When identifying TPU, the size of the computing unit array and the number of parallel processable subtasks are obtained. At the same time, the task scheduling mechanism of hardware is also parsed, such as whether it supports multi-stream asynchronous execution and whether it supports dynamic load balancing of tasks. These parallelism information will directly guide the task division granularity in the subsequent multi-layer tiling optimization, such as how to divide the number of workgroups in the parallel allocation layer and how to cut the data to match the parallel processing capability of hardware in the BatchTile layer, to ensure that the optimized computing task can fully utilize the concurrent resources of hardware and avoid parallel capability waste or overload.

[0045] In the identification process, the system integrates the collected hardware types, hardware structures and parallelism information into standardized hardware feature descriptions and stores them in the attribute set of the hardware abstraction layer dialect. Subsequent hardware optimization processes such as tensor shape adaptation, multi-layer tiling, memory optimization and vector conversion will directly read these feature descriptions to ensure that each optimization operation accurately matches the actual characteristics of the target hardware. For example, according to the identified GPU Tensor Cores structure, the KernelTile layer will divide the matrix operation into adaptive submatrix calculation. According to the identified on-chip memory size, the BatchTile layer will adjust the tiling parameters to ensure that the data can be completely placed in the on-chip memory, and finally realize the deep adaptation of hardware optimization and target hardware.

[0046] As an optional embodiment, in step 104, the adaptive execution of the optimization intermediate representation and the hardware optimization processing matched with the target hardware characteristics are performed to obtain a target intermediate representation, including: after identifying the target hardware characteristics, the null value padding layer takes the optimization intermediate representation as input and traverses all linear algebra operator input tensors and output tensors. For tensors that do not meet the target hardware characteristics, the tensor.pad operation is used to fill 0. For the filled tensors or tensors that meet the target hardware characteristics, the tensor.extract_slice operation is used to extract valid data, which aligns the tensor shape with the target hardware computing method while keeping the output result of the machine learning model unchanged.

[0047] Specifically, in step 104, the core role of the null value padding layer is to adjust the tensor shape so that the linear algebra operators in the optimized intermediate representation can adapt to the calculation logic of the target hardware, and to achieve accurate alignment between the tensor and the hardware characteristics without changing the model output result. The null value padding layer takes the optimization intermediate representation that has undergone target-independent optimization as the processing object. First, it will comprehensively traverse all the input and output tensors of the linear algebra operators in the optimization intermediate representation. This traversal process will combine the target hardware characteristics identified in the early stage (such as the memory transfer rules of the hardware, the fixed vector length limit, the tensor dimension specification supported by the calculation core, etc.), and check whether the shape of each tensor meets the hardware requirements one by one. For example, if the target hardware is a GPU that supports fixed vector length instructions, its vector operation unit can only efficiently process tensors with a dimension of a specific value. When the input tensor dimension of a linear algebra operator is less than this specific value, it is determined that the tensor does not meet the target hardware characteristics. If the target hardware is a TPU, its memory transfer mechanism requires the tensor dimension to be an integer multiple of a certain value. When the dimension of a tensor cannot be divided by this value, it is also determined as not meeting the characteristics.

[0048] For tensors that are determined not to meet the target hardware characteristics, the null value padding layer will perform padding operations to supplement the specified dimensions of the tensor with the value 0, so that the overall shape of the tensor is adjusted to meet the hardware requirements. The padding operation will determine the position and quantity of padding according to the hardware characteristics. For example, if the hardware requires the tensor channel dimension to be an integer multiple of 8, and the current tensor channel dimension is 5, 3 0s will be padded at the end or beginning of this dimension to make the dimension 8. If the hardware memory access requires the tensor height dimension to be 16, and the current height is 12, 4 0s will be padded to make the height 16. For tensors that already meet the target hardware characteristics, or tensors that meet the requirements after padding, the null value padding layer will further perform effective data extraction operations to locate the original effective data area in the tensor, and remove the 0 values supplemented in the padding process, and only keep the effective data required for calculation to participate in subsequent operations. For example, after padding, the channel tensor with a dimension of 5 becomes 8. After completing the calculation of the corresponding linear algebra operator, the first 5 dimensions of the effective data will be extracted, and the last 3 dimensions of the padded 0 values will be discarded. The tensor with a height of 12 padded to 16 will extract the first 12 heights of the effective data after calculation to ensure that the final output result is completely consistent with that before padding.

[0049] During the whole process, the null value padding layer closely connects the calculation logic of the linear algebra operator, ensuring that the padding and extraction operations are seamlessly integrated into the execution flow of the operator, that is, the padding operation is completed before the operator reads the tensor data, providing the operator with tensor input that meets the hardware requirements. The extraction operation is executed after the operator completes the calculation, and the original valid data is filtered out from the calculation results as the operator output. For example, when processing a convolution operator, if the width dimension of the convolution input tensor does not meet the requirements of the GPU vector instruction, the null value padding layer will first pad 0 in the width dimension to make the dimension meet the requirements. The convolution operator completes the feature extraction calculation based on the padded tensor, and after the calculation is completed, the padding part in the width dimension is removed through the extraction operation to obtain the convolution output result that matches the original tensor width. Through such processing, the execution efficiency problem caused by the mismatch between the tensor shape and the hardware characteristics is solved, and the calculation accuracy of the machine learning model is guaranteed through effective data extraction, providing a shape-adapted tensor basis for subsequent hardware optimization steps such as multi-layer tiling and memory optimization.

[0050] As an optional embodiment, in step 104, the adaptive execution of the optimization intermediate representation and the hardware optimization processing matched with the target hardware characteristics are performed to obtain the target intermediate representation, including: based on the TilingInterface interface, the iteration domain of the operator in the optimization intermediate representation is obtained in the parallel allocation layer, the tiling size is calculated and the outer loop is generated, the tiling operation implementation is generated in the innermost loop, and the operator is divided and allocated to multiple work groups after cutting, so that the amount of tensor data processed by each work group meets the memory limit of the target hardware. After the parallel allocation layer is executed, the BatchTile layer combines the maximum private memory of the target hardware to calculate the tiling size set, ensuring that each dimension vector of each tensor is divided by the corresponding tiling size value in the tiling size set and the total memory occupancy after cutting is less than the maximum private memory. The scf::tileUsingSCF interface is called to complete tiling and introduce For control flow to realize on-chip memory reuse and process the thread-in data access of the Reduction operator. After the BatchTile layer is executed, the KernelTile layer splits the complex calculation into sub-calculations matched with the hardware core according to the core calculation characteristics of the target hardware; the calculation definition and memory transmission are segmented by using respective control flows, and the time overhead of the calculation definition and memory transmission is balanced through the memory optimization of the BatchTile layer.

[0051] Specifically, in step 104, the multi-layer tiling optimization, as the core part of hardware adaptive optimization, needs to be executed in the progressive order of the parallel allocation layer, the BatchTile layer, and the KernelTile layer. Each layer adjusts the calculation division strategy around the target hardware characteristics to ensure that the operator processing is accurately adapted to the hardware memory limit, parallel capability, and core computing characteristics. The parallel allocation layer first takes the optimized intermediate representation as input and relies on the TilingInterface interface to implement the preliminary division of the linear algebra operator. This interface can automatically analyze the iteration domain of the operator, clearly define the calculation range and data dependency of the operator in each dimension, and then calculate the reasonable tiling size based on the basic memory limit of the target hardware (such as the maximum data volume that a single workgroup can carry). Based on this size, the outer loop structure is generated. These outer loops correspond to the iteration logic of different tiling blocks, and the specific execution logic of the tiling operation is constructed in the innermost loop, which divides the original complete operator into multiple smaller sub-operators. After the division is completed, the parallel allocation layer distributes these sub-operators to multiple workgroups of the target hardware, and each workgroup only processes the corresponding sub-operator and the associated small-size tensor data, ensuring that the memory occupancy of each workgroup does not exceed the hardware's basic limit. For example, when processing a large-size matrix multiplication operator, the parallel allocation layer divides the matrix into multiple sub-matrix blocks, and each sub-matrix block is calculated by a workgroup, avoiding memory overflow caused by excessive data volume in a single workgroup.

[0052] After the execution of the parallel allocation layer, the BatchTile layer further optimizes based on the division results of the former, with the core focusing on the on-chip private memory characteristics of the target hardware. The BatchTile layer first obtains the maximum private memory value of the target hardware through the hardware abstraction layer dialect, and then calculates the adaptive tiling size set in combination with the dimension information of each tensor in the current scheduling task. This set needs to satisfy two key conditions: first, each dimension of each tensor can be divided by the corresponding tiling size in the set, ensuring that there is no remaining data after division; second, the total occupancy of all tensors in the on-chip private memory after division does not exceed the maximum private memory value, avoiding frequent access to off-chip memory due to insufficient on-chip memory. After determining the tiling size set, the BatchTile layer calls the pre-set interface to complete the final tiling and introduces the For control flow statement to manage data access logic. This control flow allows the calculation process within the same workgroup to reuse data in the on-chip private memory, reducing the overhead of repeated data loading. Most importantly, the BatchTile layer can effectively handle Reduction operators (such as sum and average operators that require cumulative previous results). Since it handles the on-chip memory access of internal threads in a workgroup, there is no data isolation problem between different threads, and it can directly obtain the previous calculation results required for accumulation. The parallel allocation layer cannot handle such operators due to cross-thread data interaction, and the BatchTile layer compensates for this limitation.

[0053] After the BatchTile layer processing is completed, the KernelTile layer, as the last step of multi-layer tiling, further refines the granularity of the cut to match the core computing characteristics of the target hardware. The KernelTile layer first analyzes the computing core specifications of the target hardware. For example, if the target hardware is a GPU equipped with Tensor Cores, its core is good at processing submatrix operations of a specific size. If it is a TPU, its computing core is adapted to tensor operations of a specific structure. The KernelTile layer will accordingly split the sub-computations output by the BatchTile layer into smaller sub-tasks, ensuring that each sub-task can be directly executed by the hardware core without additional adjustments. At the same time, the KernelTile layer separates the computation definition and memory transfer through control flow design, allowing the data transfer from off-chip memory to on-chip memory to be independently scheduled from the computation execution in on-chip memory. Combined with the on-chip memory optimization already completed by the BatchTile layer, the time overhead of data transfer and computation gradually tends to balance, avoiding the idle of computing core due to data transfer waiting, or the insufficient data supply due to too fast computation, for example, when processing complex matrix multiplication, the KernelTile layer will split the computation into submatrix multiplication that the hardware core is adapted to, and through control flow, the data transfer and submatrix multiplication are alternately performed, so that the hardware core is in a high-efficiency operation state, and finally the target intermediate representation fully adapted to the target hardware is output.

[0054] In an optional embodiment of step 104, further optionally, the computation definition and memory transfer are separated by respective control flows, and after the time overhead of computation definition and memory transfer is balanced through the memory optimization of the BatchTile layer, further comprising: after the multi-layer tiling execution is completed, the memory allocation layer defines the SubSpanToAllocaPattern rewriting mode, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and applies on-chip private memory of the target hardware; according to whether the operator has a vector user, set the memory space attribute and alignment mode, and insert the memref::CopyOp instruction to realize the data copy from off-chip memory to on-chip private memory. Further, after the memory allocation layer is executed, the memory offset flattening layer calls IREEExpandStridedMetadata Pass and ExpandStridedMetadata Pass in sequence, converts high-dimensional memory access expressions into one-dimensional through affine transformation, and establishes the memory copy mapping relationship from high-dimensional to low-dimensional.

[0055] Specifically, after the multi-layer tiling optimization is completed, the memory optimization will focus on the efficient use of on-chip memory and the simplification of memory access logic. Through the collaborative operation of the memory allocation layer and the memory offset flattening layer, the hardware memory access delay is further reduced, and the memory addressing rules of the target hardware are adapted. After the multi-layer tiling is completed, the memory allocation layer first starts the memory operation optimization. The core is to define the SubSpanToAllocaPattern rewrite mode to sort out the memory reference relationship in the optimized intermediate representation. This rewrite mode can accurately identify complex memory reference forms, such as sub-view memory references generated by tensor splitting. Such references often contain multiple layers of nested address calculation logic, which directly increases the complexity and delay of hardware memory access. The memory allocation layer will convert these complex memory references to memref::AllocaOp operations, which will apply for independent on-chip private memory space to the target hardware. On-chip private memory can directly interact with hardware registers, and its access speed is much faster than off-chip memory, which can greatly improve data read-write efficiency. When applying for on-chip private memory, the memory allocation layer will first determine whether the current operator has a vector user. If the operator needs to participate in vector operations in the future, such as vector addition and vector multiplication handled by the vectorization layer, it will set the high-speed vector memory space attribute and the alignment mode matching the vector instruction to ensure that the vector operation can efficiently read memory data. If there is no need for vector operation, the standard memory space and basic alignment mode are used. At the same time, the memory allocation layer will insert the memref::CopyOp instruction to copy the data originally stored in the off-chip memory to the newly applied on-chip private memory. For example, when processing feature map data of a convolutional neural network, the feature map sub-blocks after multi-layer tiling will be transferred from the off-chip HBM to the on-chip private memory of the GPU through this copy instruction. The subsequent convolution calculation directly reads data from the on-chip memory, avoiding the delay caused by frequent access to off-chip memory.

[0056] After the on-chip memory application and data copy are completed by the memory allocation layer, the memory offset flattening layer will continue to handle the dimension adaptation problem of memory access. This layer will call IREEExpandStridedMetadata Pass and ExpandStridedMetadata Pass in turn. These two Passes convert high-dimensional memory access expressions in the optimized intermediate representation into one-dimensional form through affine transformation technology. High-dimensional memory access, such as three-dimensional tensor channel, height, and width dimension access, although conforms to the model calculation logic, the memory addressing of the target hardware is usually implemented in one-dimensional address form. Directly using high-dimensional access will increase the address calculation overhead of the hardware. Through affine transformation, the memory offset flattening layer will establish a mapping relationship between high-dimensional dimensions and one-dimensional addresses, accurately corresponding each data position in the high-dimensional space to a unique address in the one-dimensional memory. For example, when processing the memory access of a three-dimensional feature map, the channel index is multiplied by the height and then by the width, and the height index is multiplied by the width, and then the width index is added to the high-dimensional offset calculation. Through affine transformation, this is simplified to a single one-dimensional offset value. This conversion not only makes the memory access logic more consistent with the addressing method of the hardware, but also avoids the hardware from frequently performing real-time calculation of multi-dimensional to one-dimensional during execution, reducing hardware resource consumption. Taking the TPU processing of fully connected layer data as an example, the weight matrix of the fully connected layer is usually two-dimensional in structure. The memory offset flattening layer will convert the two-dimensional input neuron and output neuron memory access into one-dimensional address access through affine transformation, so that the memory controller of the TPU can directly locate the weight data according to the one-dimensional address, cooperate with the data storage of the on-chip memory in the early stage, further improve the overall efficiency of the fully connected layer calculation, and ultimately provide low-latency and high-adaptation memory access basis for subsequent vector conversion and code generation.

[0057] In an optional embodiment of step 104, further optionally, the memory allocation layer defines a SubSpanToAllocaPattern rewrite pattern, performs memory optimization on the optimized intermediate representation, and converts complex memory reference in the optimized intermediate representation into a memref::AllocaOp operation to apply on-chip private memory of the target hardware, including: initializing the SubSpanToAllocaPattern rewrite pattern; the rewrite pattern inherits from Rewrite Pattern, adapts to all operation types, and sets the optimization level. Check whether the operation to be processed in the optimized intermediate representation has a CopySource attribute and does not have a CopySourceDone attribute, and return a failure indication if not satisfied; if satisfied, check whether the User of the operation has executed a memref::CopyOp, and if yes, set the CopySourceDone attribute for the operation and remove the CopySource attribute, and return a success indication. Check whether there is a memref::CollapseShapeOp in the User of the operation to be processed, so as to determine whether the complex memory reference operation in the optimized intermediate representation needs to be replaced by a memref::AllocaOp, and update the operation pointer op synchronously. Set the memory space attribute and the alignment according to whether the operation to be processed has a vector user hasVectorUser; if there is a vector user, use the high-speed vector memory space obtained by getVectorMemoryAttr; if not, use the standard memory space. Create a new memref::AllocaOp to replace the original memory operation, and configure the memory type and the alignment of the memref::AllocaOp; insert a memref::CopyOp according to the read-only state of the memref::AllocaOp: if it is a read-only state, only insert a copy instruction from the result memory to the memref::AllocaOp; if it is a non-read-only state, in addition to the above copy instruction, additionally insert an instruction at the end of the operation to copy back from the memref::AllocaOp to the result memory. Replace all uses of the original memory operation with the updated memref::AllocaOp, and ignore the User of the memref::AssumeAlignmentOp and the memref::CopyOp; if the updated memref::AllocaOp has a sub-view User, update the sub-view operation according to the parameters of the memref::AllocaOp.

[0058] It can be understood that in the optional embodiment of step 104, when the memory allocation layer performs memory optimization through the SubSpanToAllocaPattern rewrite mode, the initialization of the rewrite mode will be completed first. Since the rewrite mode needs to adapt to various memory-related operations in the optimization intermediate representation, it will inherit from the system-based Rewrite Pattern to ensure that all operation types can be processed; at the same time, the corresponding optimization level will be set, so that the system can prioritize memory references when performing multiple rounds of optimization, avoiding the impact of other low-priority optimizations on the optimization effect of memory operations. For example, when processing optimization intermediate representations containing multiple operators such as convolution and pooling, the rewrite mode will be called first to ensure that memory-level optimization is completed before other detailed optimizations. After initialization, the rewrite mode will perform attribute checking on the operations to be processed in the optimization intermediate representation. The core is to judge whether the operation has the CopySource attribute and does not have the CopySourceDone attribute. These two attributes are key to the system marking whether the memory reference needs to be optimized. The CopySource attribute indicates that the memory reference corresponding to the operation needs further processing, and the CopySourceDone attribute indicates that the optimization has been completed. If the condition of having CopySource and not having CopySourceDone is not met, it means that the operation either does not need to be optimized or has been optimized, at which point the failure indication information will be returned to terminate the subsequent processing of the operation. If the condition is met, the User (i.e., the subsequent operation that depends on the output of the operation) of the operation will be further checked to see if it has executed memref::CopyOp. If it has, it means that the data copying process has been completed, and there is no need to adjust the current memory reference. At this point, the CopySourceDone attribute will be set for the operation and the CopySource attribute will be removed, and the success indication information will be returned to avoid repeated optimization. For example, when processing the memory reference of the weight tensor in the fully connected layer, if it is found that the subsequent User has executed memref::CopyOp, the memory reference optimization will be marked as completed and no further processing will be performed.

[0059] Subsequently, the rewriting mode determines whether the complex memory reference in the optimized intermediate representation needs to be converted into a memref::AllocaOp, specifically by checking whether there is a memref::CollapseShapeOp in the User of the operation to be processed, which is commonly used to compress high-dimensional memory structures into low-dimensional ones. The memory reference dependent on it often contains multiple layers of sub-view nesting and belongs to a complex memory reference. If the operation exists, it means that the current memory reference needs to be converted into a more concise memref::AllocaOp to reduce the access complexity, and the operation pointer op will be updated synchronously to ensure that subsequent operations can accurately locate the memory reference that needs to be replaced. For example, when processing a feature map sub-view after multiple layers of tiling, if there is a memref::CollapseShapeOp for dimension compression afterwards, it will be determined that the complex memory reference corresponding to the sub-view needs to be converted into a memref::AllocaOp.

[0060] After determining the need for conversion, the memory space attribute and alignment are configured according to whether the to-be-processed operation has a vector user (hasVectorUser). If there is a vector user, it means that the memory reference will participate in the vector operation of the vectorization layer later, and the vector operation has a higher requirement for memory access speed. Therefore, the high-speed vector memory space of the target hardware is obtained through getVectorMemoryAttr, and the alignment matching the vector instruction is set to ensure that the vector operation can efficiently read the memory data. If there is no vector user, it means that the memory reference is only used for ordinary scalar operation, and at this time, the standard memory space and basic alignment of the target hardware are used to balance the memory resource occupation and access efficiency. For example, when processing the memory reference corresponding to the vector multiplication operator on the GPU, the high-speed vector memory of the GPU is allocated for it, and the alignment is required according to the vector instruction; when processing the memory reference corresponding to the ordinary scalar addition, the standard memory space of the GPU is used. Next, a new memref::AllocaOp is created to replace the original complex memory reference operation. During the creation process, the memory type (such as matching the data type and dimension information of the tensor) and the alignment of the memref::AllocaOp are configured according to the memory requirements of the to-be-processed operation, to ensure that the newly allocated memory can accurately take over the function of the original memory reference. At the same time, the memref::CopyOp is inserted according to the read-only state of the memref::AllocaOp: if the AllocaOp is in the read-only state, it means that only data from the outside needs to be read for subsequent calculation, and the data does not need to be modified. At this time, only the copy instruction from the result memory (usually off-chip memory) to the AllocaOp (corresponding to on-chip private memory) is inserted to load the data on the chip; if it is in the non-read-only state, it means that the data in the memory will be modified in the subsequent calculation. In addition to inserting the copy instruction from the result memory to the AllocaOp, an instruction is also inserted at the end of the operation to copy the data from the AllocaOp back to the result memory, to ensure that the modified data can be synchronized to the off-chip memory to avoid data loss. For example, when processing the memory reference of the input feature map of the convolution operator, if only the feature map data needs to be read for convolution calculation, the AllocaOp is set to read-only, and only the copy from off-chip to on-chip is inserted; when processing the memory reference of the output feature map, because the convolution result needs to be stored, the AllocaOp is set to non-read-only, and an additional copy from on-chip to off-chip is inserted.

[0061] Finally, all uses of the original memory operation are replaced with the newly created memref::AllocaOp, and the User of the memref::AssumeAlignmentOp and the memref::CopyOp are ignored in the replacement process. The memref::AssumeAlignmentOp is only used to declare memory alignment information and does not depend on the specific form of the memory reference, and the memref::CopyOp has been adapted in the previous steps, so there is no need to replace the User of these two types of operations. If the new memref::AllocaOp has a subview User (i.e., subsequent operations access this memory through a subview), the access range and address calculation logic of the subview operation are updated according to the memory address, dimension, and other parameters of the AllocaOp to ensure that the subview can correctly locate the data in the newly allocated on-chip private memory. For example, when processing the subview of the convolution output feature map that depends on the pooling operator, after being replaced with the AllocaOp, the address parameters of the subview are updated to allow the pooling operator to accurately read the feature map data in the on-chip memory corresponding to the AllocaOp, ensuring the correctness of subsequent calculations.

[0062] As an optional embodiment, in step 104, the hardware optimization processing that matches the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation, including: a vectorization layer taking the optimized intermediate representation as input, calling the linalg::vectorize interface to identify parallelizable scalar operations therein, and converting the scalar operations into vector operations; synchronously rewriting the memory access mode, converting scalar memory loading into vector memory loading, and converting scalar memory storage operations into vector memory storage operations; if the target hardware does not support vector operations, calling the LinalgToLoops method to convert linear algebra operators in the optimized intermediate representation into multiple loops; after the vectorization execution is completed, a vector flattening layer splits high-dimensional vector operators into multiple one-dimensional vector operators, so that the obtained to-be-processed intermediate representation conforms to the LLVM vector type, which is a compile-time abstraction of the target hardware vector register; a software pipeline layer takes the vector flattened to-be-processed intermediate representation as input, rearranges instructions in the loop body, and obtains the target intermediate representation, so that multiple different iteration instructions are executed in parallel in each loop iteration.

[0063] In particular, in the optional embodiment of step 104, the vectorization layer, the vector flattening layer and the software pipeline layer process the optimized intermediate representation in a progressive order, optimize around the vector processing capabilities and instruction level parallelism requirements of the target hardware, and finally generate a target intermediate representation adapted for hardware execution. The vectorization layer first takes as input the optimized intermediate representation that has been tiled, memory-optimized, and the like through multiple layers, and the core task is to convert scalar operations into vector operations that can take advantage of hardware vector instructions. It calls the linalg::vectorize interface, parses the execution logic of linear algebra operators in the optimized intermediate representation line by line, and identifies parallelizable scalar operations therein. Such operations are usually independent iterations in a loop, such as the ReLU activation calculation of the output feature map of the convolution layer in an image classification model, where the activation judgment of each pixel point (retaining the original value if greater than 0, or setting to 0 if less than 0) is independent of each other and belongs to parallelizable scalar operations. The interface packs and integrates these independent scalar operations into corresponding vector operations, such as combining the scalar activation operations of multiple pixel points into a vector activation operation. At the same time, the vectorization layer simultaneously rewrites the memory access pattern. The memory load operation originally directed at a single scalar data is changed to a vector memory load operation that loads multiple scalar data at a time. The memory storage operation directed at a single scalar result is also changed to a vector memory storage operation that stores multiple vector results at a time, ensuring that the memory access and the processing rhythm of the vector operation match. If the target hardware does not support vector operations, such as some low-end CPUs that can only handle scalar operations, the vectorization layer calls the LinalgToLoops method to disassemble the linear algebra operators in the optimized intermediate representation into multiple nested loops, so that each loop iteration handles a scalar operation, adapting to the scalar computing capabilities of the hardware, such as splitting the matrix multiplication operator of the fully connected layer into multiple loops that traverse the input neurons in the outer loop and the output neurons in the inner loop, to ensure that the operator can execute normally on a low-end CPU.

[0064] After the vectorization execution is completed, the vector flattening layer will continue to process the adaptation problem of the vector dimension. Since the vector operations generated by the vectorization layer can contain high-dimensional structures, the LLVM vector type is an abstraction of the target hardware vector register at compile time, and usually only supports one-dimensional form. If a high-dimensional vector operator is used directly, it will cause subsequent LLVM dialect conversion and hardware register allocation to fail. Therefore, the vector flattening layer will traverse the vectorized intermediate representation and split the high-dimensional vector operator into multiple one-dimensional vector operators. For example, when processing a three-dimensional vector convolution operation on a TPU, the operation originally contains vector calculations in the channel, height, and width dimensions. The vector flattening layer will split it into three independent one-dimensional vector operators corresponding to the vector calculations in the channel, height, and width dimensions, so that the split intermediate representation to be processed fully complies with the specifications of the LLVM vector type, ensuring that it can be successfully mapped to the vector registers of the TPU.

[0065] Finally, the software pipeline layer takes the vector flattened intermediate representation as input, focusing on the instruction level parallel optimization of the loop code. It analyzes the loop structure in the intermediate representation to be processed, identifies the key instructions in the loop body (such as memory load, calculation, and storage instructions), and then rearranges the order of instructions to break the serial execution constraints of instructions within a single loop iteration, allowing multiple instructions from different iterations to be executed in parallel in each loop iteration. For example, when processing the weight calculation of the fully connected layer, the software pipeline layer will move the instruction to load the next weight data to the calculation of the current weight instruction during the current iteration, allowing the calculation of the current iteration to be parallel to the memory load of the next iteration. At the same time, the instruction to store the current calculation result is delayed to the calculation period of the next iteration to form a pipeline execution mode. This optimization maximizes the utilization of the target hardware resources and reduces the idle time caused by waiting for memory loading or calculation to complete. For example, after executing this optimization on a GPU, the GPU's compute cores and memory controllers can work simultaneously during each loop iteration, significantly improving the overall execution efficiency of the fully connected layer. Finally, the target intermediate representation is generated to adapt to the vector characteristics and instruction level parallel requirements of the target hardware.

[0066] As an optional embodiment, before obtaining the target intermediate representation in step 104, the tensor in the optimization intermediate representation can also be allocated to the corresponding target hardware device according to the computing power of the target hardware and the amount of tensor data through the device scheduling layer; after entering the hardware adaptive optimization phase, a stream bound to the device is created according to the target hardware device where the tensor is located; the stream bound to the GPU adapts to the CUDA backend, and the stream bound to the CPU adapts to the CPU backend; the hardware optimization processing in each stream and the generation of subsequent hardware operation units are matched with the backend characteristics of the corresponding device; the streams pass the tensor data through the data copy operation containing memref::CopyOp, ensuring the correctness of cross-device data interaction.

[0067] In an optional embodiment of step 104, the device scheduling layer plays a core role in tensor, hardware matching, and cross-device task coordination, operating between target hardware characteristic identification and hardware adaptive optimization, and defining a clear device execution range for subsequent optimization processing. The device scheduling layer first analyzes the computing capacity of each target hardware and the data volume of the tensors in the optimization intermediate representation in combination with the target hardware characteristics identified earlier: the computing capacity of the target hardware is reflected in the number of parallel tasks it supports, the type of computing core (such as Tensor Cores of GPU, vector core of CPU), etc., and the data volume of the tensor is related to data storage requirements and transmission costs. Based on these two factors, the device scheduling layer allocates different tensors in the optimization intermediate representation to the corresponding target hardware devices. For example, when processing an image segmentation model, large-size feature map tensors in the model used for feature extraction (large data volume, computationally intensive) are allocated to GPUs, as they have a large number of parallel computing cores and can efficiently handle large-scale tensor operations. Small-size parameter tensors in the model used for result post-processing (small data volume, light computation) are allocated to CPUs to avoid resource waste caused by occupying GPU resources with small data volume and reduce the frequency of cross-device data transmission.

[0068] After completing tensor allocation, in the hardware adaptive optimization phase, the device scheduling layer creates a stream directly bound to the device according to the target hardware device where each tensor is located. As a logical channel for hardware task execution, the stream forms a fixed association with the hardware backend. If the device bound to the stream is a GPU, the stream will automatically adapt to the CUDA backend, and all subsequent computing tasks in the stream will follow the task scheduling rules and memory model of CUDA; if the device bound to the stream is a CPU, the stream will adapt to the CPU backend, adapting to the thread scheduling mechanism and vector instruction set of the CPU. This binding relationship ensures that the optimization processing in the stream can accurately match the hardware bottom-layer characteristics, for example, a GPU-bound stream will prioritize the use of shared memory of GPU, thread block division, etc. in subsequent optimization. A CPU-bound stream will focus on cache utilization of CPU, vectorization instruction adaptation, and other optimization directions.

[0069] The hardware optimization processing (such as tensor shape adaptation, multi-layer tiling, vectorization, etc.) in each stream and the generation of subsequent hardware operation units strictly match the backend characteristics of the corresponding device. Taking the GPU-bound stream as an example, when performing multi-layer tiling optimization, the tiling parameters are determined according to the thread block size supported by the CUDA backend and the calculation specifications of the Tensor Cores, the large matrix operation is divided into sub-matrix calculation that adapts to the Tensor Cores, and the generated hardware operation unit is also a binary instruction that conforms to the CUDA instruction specification. When the CPU-bound stream performs vectorization optimization, it will convert scalar operations into corresponding vector operations based on the vector instruction set (such as the AVX series) supported by the CPU backend, and the generated hardware operation unit adapts to the vector registers and instruction execution logic of the CPU.

[0070] When there is a data dependency between tasks in different streams, the streams will pass tensor data through data copy operations containing memref::CopyOp to ensure the correctness of cross-device data interaction. For example, in an image segmentation model, the feature map tensor processed by the GPU stream needs to be passed to the CPU stream for final segmentation result output processing, which triggers the memref::CopyOp operation. This operation first reads the storage address and data format of the feature map tensor in the GPU stream, and then transfers the data from the GPU's off-chip memory (such as HBM) to the CPU's main memory according to the memory access rules of the CPU backend. During the transmission process, the memory layout differences between the two ends are automatically adapted to avoid calculation errors caused by incompatible data formats. Through this data transfer method, the streams on different devices can cooperate to complete the entire model's calculation task, fully leveraging the computing advantages of each hardware while ensuring the continuity of cross-device execution.

[0071] In the embodiments of the present application, the hardware operation unit is the core output product of the code generation stage, a binary instruction set that adapts to the computing characteristics of the target hardware (such as GPU, CPU, domestic TPU, etc.) and can be directly recognized and executed by the target hardware. Its essence is to map the target intermediate representation after hardware self-adaptive optimization to the specific execution carrier of the target hardware's native instruction through the dialect conversion and compilation process.

[0072] As an optional embodiment, in step 105, the target intermediate representation is converted into a hardware operation unit executable by the target hardware, and the hardware operation unit is loaded into the target hardware to build an AI accelerator. Specifically, converting the target intermediate representation into a hardware operation unit executable by the target hardware can be implemented as follows: taking the target intermediate representation after hardware optimization processing as input, and executing a dialect step-by-step conversion process; starting from the linear algebra dialect in the target intermediate representation, sequentially passing through the following processing: tensor dialect, one-time bufferization, memory reference dialect, vectorization, vector dialect, LLVM dialect conversion, and LLVM dialect; converting the LLVM dialect file into an assembly file corresponding to the target hardware through an LLVM compiler, and building the assembly file into a binary form of hardware language file and hardware operation instruction; and fusing the hardware language file and the hardware operation instruction into the hardware operation unit in combination with the configuration information and hardware attribute information of the target hardware.

[0073] It can be understood that in the optional embodiment of step 105, the process of converting the target intermediate representation into a hardware operation unit and building an AI accelerator is centered around dialect step-by-step conversion, LLVM compilation chain processing, and hardware configuration fusion, ensuring that the final output can be directly recognized and loaded for execution by the target hardware. First, the dialect step-by-step conversion process is started, which takes the target intermediate representation after hardware optimization (such as multi-layer tiling, memory optimization, and vector conversion) as input, and always starts from the linear algebra dialect. The linear algebra dialect, as a hardware-independent computing definition carrier, contains all core operation logic in the model (such as convolution, matrix multiplication, and activation operator), and subsequent conversion needs to preserve these computing semantics while gradually adapting to hardware bottom-level representation. During the conversion process, first, the tensor dialect stage is entered, which mainly normalizes the basic form description of tensor data to prepare for subsequent memory operations. Then, one-time bufferization processing is performed to convert the abstract memory representation of the tensor into a form that can be directly mapped to the hardware physical memory, solving the adaptation problem of the tensor and the hardware memory model. For example, when processing the convolution operator of an image classification model, the convolution input tensor is converted from an abstract tensor form to a buffer representation that can be associated with a hardware memory address. Subsequently, the memory reference dialect stage is entered, which explicitly describes the access mode of data in the hardware memory (such as memory address, data span, storage order, etc.), such as setting a continuous memory storage order for the weight tensor of the convolution operator to reduce the probability of cache miss failure during hardware access. Then, through vectorization processing, scalar operations are converted into vector operations, and then the vector dialect is used to standardize the representation form of the vector operations to ensure compatibility with the hardware vector register abstraction (LLVM vector type). Finally, through the LLVM dialect conversion step, the vector dialect representation is mapped to the LLVM dialect at the MLIR level, completing the transition from the high-level MLIR dialect to the LLVM compatible representation.

[0074] The LLVM dialect conversion layer plays a key role in this process, as it defines a series of patterns that serve as conversion devices to accurately convert accelerator-specific dialect operators (such as Tensor Cores operators for GPUs or systolic array operators for TPUs) into LLVM IR. To ensure that the converted LLVM IR matches the hardware instruction set, on the one hand, the IntrinsicsXXX.td file (XXX is the name of the target hardware platform, such as IntrinsicsTPU.td for TPU) is defined to define the instruction primitives of the target accelerator. These primitives are abstract descriptions of hardware-level instructions at compile time, such as defining a matrix multiplication instruction primitive for the tensor operation core of TPU or a mixed-precision calculation instruction primitive for Tensor Cores of GPU. On the other hand, existing general operations in LLVM IR are used to avoid repeated definitions, such as directly reusing standard arithmetic operations in LLVM IR for hardware-specific scalar operations like addition and multiplication. Only complex operations specific to hardware (such as systolic array data transfer for TPU or shared memory synchronization for GPU) are handled through custom patterns and instruction primitives.

[0075] After completing the dialect conversion to obtain the LLVM dialect, the mlir-translate tool is used to generate the.ll file (LLVM IR text file). Subsequently, the LLVM compiler chain is used to process it. First, the.ll file is converted into the corresponding assembly file for the target hardware by tools such as clang and llc. The assembly file will retain the execution logic of the hardware instructions, such as matrix operation instructions for Tensor Cores and shared memory access instructions for GPUs, or vector register operation instructions for CPUs. Then, the LLVM compiler further compiles the assembly file into binary hardware language files (such as.ptx files for GPUs or.o files for CPUs) and corresponding hardware operation instructions. These binary files have the format of hardware executable instructions, but still need to be combined with the configuration information and attribute information of the target hardware to form a complete hardware operation unit.

[0076] Finally, the configuration information (such as the memory specification, the number of computing cores, the supported precision type, the data transmission bandwidth, etc.) and the attribute information (such as the instruction set version of the hardware, the memory access delay threshold, etc.) of the target hardware are read, and these information is fused with the binary hardware language file and the hardware operation instruction. For example, for a TPU, the on-chip private memory capacity and the systolic array size are embedded into the hardware operation unit to ensure that the memory allocation instruction in the operation unit is adapted to the memory limit of the TPU. For a GPU, the maximum thread block size and the shared memory size are fused to ensure that the thread scheduling instruction in the operation unit conforms to the parallel execution rules of the GPU. After the fusion, the hardware operation unit obtained contains all the instruction logic required for model calculation and is adapted to the hardware characteristics and configuration parameters of the target hardware. When the hardware operation unit is loaded into the target hardware, the construction of the AI accelerator is completed. For example, after the hardware operation unit adapted to the TPU is loaded into the TPU chip, the TPU can directly call the computing core to execute the operator instruction in the hardware operation unit to achieve hardware acceleration of the deep learning model.

[0077] In step 106, in the AI accelerator, the hardware abstraction layer dialect is used to connect the IR at the compilation time and the hardware operation unit running on the target hardware at the runtime to schedule the corresponding computing resources in the target hardware and complete the hardware acceleration of the machine learning model. Specifically, in step 106, the hardware abstraction layer dialect provides two core functions: at the compilation time, a complete IR and instruction describing the program execution process are provided. At the runtime, a Runtime API (written in C language) corresponding to the compilation time IR and instruction is provided. Then, according to the target hardware type (CPU / GPU / TPU), the Runtime API is implemented through the driving API of the target hardware, and the Stream Dialect is converted into the hardware abstraction layer dialect. Further, the binary file of the hardware operation unit is combined with the converted hardware abstraction layer dialect to form a complete neural network executable program. The runtime component completes the resource management (including on-chip memory allocation management, register scheduling, computing core task allocation) of the target hardware, task scheduling synchronization, and interaction with the host code through the Runtime API, drives the hardware operation unit to execute on the target hardware, and realizes the hardware acceleration of the machine learning model.

[0078] In the embodiments of the present application, through the target-independent optimization stage processing and adaptive hardware optimization, an AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures, so that the flexible allocation of computing resources in the target hardware is realized through the AI accelerator, the model deployment efficiency and running efficiency are effectively improved, the adaptation degree of the machine learning model and the hardware resources is improved, which helps fine optimization from the machine learning model to the hardware resources, and the parallel resource utilization rate is simultaneously improved, and the accelerator resource waste is avoided.

[0079] In still another embodiment of the present application, an AI accelerator construction device is also provided. Referring to Figure 2 As described above, the device comprises the following units: The import unit is configured to import machine learning models of different AI frameworks through a multi-framework adaptation interface, parse the model representation of the machine learning model, and convert the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain; The optimization unit is configured to, in the target-independent optimization stage, retain the core computing semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; in the hardware adaptive optimization stage, the target hardware characteristics of the target hardware are identified, the target hardware characteristics include: hardware type, hardware structure, and parallelism; the optimized intermediate representation is adaptively executed with hardware optimization processing matched with the target hardware characteristics to obtain a target intermediate representation, and the hardware optimization processing matched with the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion; The construction unit is configured to convert the target intermediate representation into hardware operation units executable by the target hardware, load the hardware operation units into the target hardware, and construct an AI accelerator; The AI accelerator is configured to connect the compile-time IR and the runtime on the target hardware through the hardware abstraction layer dialect, run the hardware operation units on the target hardware, schedule the corresponding computing resources in the target hardware, and complete the hardware acceleration of the machine learning model.

[0080] Further optionally, the importing unit is configured to, if the imported AI framework is PyTorch, trigger a Dynamo component to work through a torch.compile function of PyTorch 2.0, the Dynamo component dynamically modifies Python bytecode through a hook CPython frameevaluation API, rewrites the bytecode to extract an Aten operator sequence, and finally generates an FX computation graph; traverse the FX computation graph through a SHARK Turbine tool, parse data types and operator names of operators in the FX computation graph, and map the Aten operators to Torch dialect operators defined in a Torch MLIR tool one by one; and call a conversion interface of the Torch MLIR to convert the Torch dialect operators into an MLIRModuleOp in a linear algebra dialect representation, the MLIRModuleOp containing complete computation definitions of the machine learning model.

[0081] Further optionally, the optimization unit is configured to, in the target-independent optimization stage, retain core computation semantics in the linear algebra dialect through target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; and the optimization unit is configured to: a scheduling creation layer traverses all blocks and block-in operators in an MLIR ModuleOp corresponding to the linear algebra dialect representation, identifies root operators in all blocks as optimization center points, and assigns a unique identifier kRootOpAttr to the root operators; through a fusion group attribute kFusionGroupsAttr, marks producer operators providing input for the root operators and consumer operators using output of the root operators, and fuses the producer operators and the consumer operators into the root operators; allocates linear algebra dialect operators that are not marked as the root operators and do not belong to linalg.fill operators or dequantization operators to independent scheduling sections; and synchronously performs memory management optimization, redundant computation elimination, and data access pattern optimization to generate the optimized intermediate representation.

[0082] Further optionally, the optimization unit is configured to fuse the producer operators and the consumer operators into the root operators by calling a fuseRootsWithConsumers function.

[0083] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to, after identifying the target hardware characteristics, fill the null value padding layer with the optimization intermediate representation as input, and traverse all linear algebra operator input tensors and output tensors therein: for tensors not meeting the target hardware characteristics, fill 0 using a tensor.pad operation; and for the filled tensors or tensors meeting the target hardware characteristics, extract valid data using a tensor.extract_slice operation, so that the tensor shape is aligned with the target hardware calculation mode while keeping the machine learning model output result unchanged.

[0084] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to, based on a TilingInterface interface, obtain an iteration domain of an operator in the optimization intermediate representation, calculate a tiling size and generate an external loop, generate a tiling operation implementation in an innermost loop, and allocate the cut operator to a plurality of work groups so that the amount of tensor data processed by each work group meets the memory limit of the target hardware; after the parallel allocation layer is executed, the BatchTile layer combines a private memory maximum value of the target hardware to calculate a tiling size set, so as to ensure that each dimension vector of each tensor is divided by the corresponding tiling size value in the tiling size set and the total memory occupancy after cutting is less than the private memory maximum value; the scf::tileUsingSCF interface is called to complete tiling and introduce For control flow to realize on-chip memory reuse and process thread data access of the Reduction operator; after the BatchTile layer is executed, the KernelTile layer splits complex calculations into sub-calculations matched with the hardware core according to the core calculation characteristics of the target hardware; the calculation definition and the memory transmission are segmented by using respective control flows, and the time overhead of the calculation definition and the memory transmission is balanced by the memory optimization of the BatchTile layer.

[0085] Further optionally, the optimization unit, after the memory allocation layer defines the SubSpanToAllocaPattern rewriting pattern, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and applies on-chip private memory of the target hardware, is configured to: according to whether the vector user exists, set the memory space attribute and the alignment mode, and insert the memref::CopyOp instruction to realize the data copy from the off-chip memory to the on-chip private memory; after the memory allocation layer is executed, the memory offset flattening layer calls the IREEExpandStridedMetadata Pass and the ExpandStridedMetadata Pass in turn, converts the high-dimensional memory access expression into one dimension through affine transformation, and establishes the memory copy mapping relationship from high dimension to low dimension.

[0086] Further optionally, the optimization unit, after the memory allocation layer defines the SubSpanToAllocaPattern rewriting pattern, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, and applies on-chip private memory of the target hardware, is configured to: The initialization SubSpanToAllocaPattern rewriting mode; the rewriting mode inherits from RewritePattern, adapts to all operation types, and sets the optimization level; check if the operation to be processed in the optimization intermediate representation has the CopySource attribute and does not have the CopySourceDone attribute, if not, return a failure indication; if yes, check if the User of the operation has performed memref::CopyOp, if yes, set the CopySourceDone attribute for the operation and remove the CopySource attribute, return a success indication; check if there is a memref::CollapseShapeOp in the User of the operation to be processed, so as to judge whether the complex memory reference operation in the optimization intermediate representation needs to be replaced by a memref::AllocaOp, and update the operation pointer op synchronously; set the memory space attribute and the alignment according to whether the operation to be processed has a vector user hasVectorUser; if there is a vector user, use the high-speed vector memory space obtained by getVectorMemoryAttr; if not, use the standard memory space; create a new memref::AllocaOp to replace the original memory operation, and configure the memory type and the alignment of the memref::AllocaOp; insert a memref::CopyOp according to the read-only state of the memref::AllocaOp: if it is a read-only state, only insert a copy instruction from the result memory to the memref::AllocaOp; if it is a non-read-only state, in addition to the above copy instruction, additionally insert an instruction at the end of the operation to copy back from the memref::AllocaOp to the result memory; replace all uses of the original memory operation with the updated memref::AllocaOp, and ignore the User of the memref::AssumeAlignmentOp and the memref::CopyOp; if the updated memref::AllocaOp has a sub-view User, update the sub-view operation according to the parameters of the memref::AllocaOp.

[0087] Further optionally, the optimization unit is configured to adaptively perform hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, and is configured to: take the optimization intermediate representation as input to a vectorization layer, call an linalg::vectorize interface to identify parallelizable scalar operations therein, and convert the scalar operations into vector operations; rewrite memory access patterns, convert scalar memory loading into vector memory loading, and convert scalar memory storage operations into vector memory storage operations; if the target hardware does not support vector operations, call an LinalgToLoops method to convert linear algebra operators in the optimization intermediate representation into multi-layer loops; after vectorization execution is completed, a vector flattening layer splits high-dimensional vector operators into multiple one-dimensional vector operators, so that the obtained to-be-processed intermediate representation conforms to an LLVM vector type, which is a compile-time abstraction of a target hardware vector register; and a software pipeline layer takes the vector-flattened to-be-processed intermediate representation as input, rearranges instructions in a loop body, and obtains the target intermediate representation, so that multiple different iteration instructions are executed in parallel in each loop iteration.

[0088] Further optionally, the construction unit is configured to convert the target intermediate representation into a hardware operation unit executable by the target hardware, and is configured to: take the target intermediate representation after the hardware optimization processing is completed as input, and perform a dialect step-by-step conversion process; take a linear algebra dialect in the target intermediate representation as a starting point, and sequentially pass through the following processing: a tensor dialect, one-time bufferization, a memory reference dialect, vectorization, a vector dialect, LLVM dialect conversion, and an LLVM dialect; convert the LLVM dialect file into an assembly file corresponding to the target hardware through an LLVM compiler, and construct the assembly file into a binary-form hardware language file and hardware operation instructions; and fuse the hardware language file and the hardware operation instructions into the hardware operation unit in combination with configuration information and hardware attribute information of the target hardware.

[0089] Further optionally, before adaptively performing hardware optimization processing matched with the target hardware characteristics on the optimization intermediate representation to obtain a target intermediate representation, the optimization unit is further configured to: a device scheduling layer allocates tensors in the optimization intermediate representation to corresponding target hardware devices according to computing capabilities of the target hardware and tensor data volumes; and after entering a hardware adaptive optimization stage, create a stream bound to a device according to a target hardware device where a tensor is located; a stream bound to a GPU adapts a CUDA back end, and a stream bound to a CPU adapts a CPU back end; hardware optimization processing in each stream and generation of a subsequent hardware operation unit are matched with back end characteristics of the corresponding device; and the streams pass tensor data through data copy operations containing memref::CopyOp to ensure correctness of cross-device data interaction. The system can implement various steps in the method embodiments described above, which are not described here in detail.

[0090] In the embodiments of the present application, the AI accelerator is constructed, and the AI accelerator containing personalized hardware operation units is adaptively constructed for machine learning models of different AI architectures through target-independent optimization stage processing and adaptive hardware optimization, the flexible allocation of computing resources is realized, the model deployment efficiency and running efficiency are effectively improved, the adaptation of machine learning models and hardware resources is improved, the fine optimization from machine learning models to hardware resources is facilitated, the parallel resource utilization rate is simultaneously improved, and the waste of accelerator resources is avoided.

[0091] Please refer to Figure 3 , Figure 3 The embodiment of the electronic device provided by the embodiments of the present application is shown. As shown in Figure 3 , the embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520, and the processor 520 implements each step in the above embodiments when executing the computer program 511. Please refer to Figure 4 , Figure 4 The embodiment of the computer-readable storage medium provided by the embodiments of the present application is shown. As shown in Figure 4 , the present embodiment provides a computer-readable storage medium 600, which stores a computer program 611, and each step in the above embodiments is implemented when the computer program 611 is executed by a processor. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0092] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they understand the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0093] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. An AI accelerator construction method, characterized by, The method comprises: Importing machine learning models of different AI frameworks through a multi-framework adaptation interface, parsing a model representation of the machine learning models, and converting the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain; In a target-independent optimization stage, by target-independent optimization processing, core computing semantics in the linear algebra dialect are retained, and the linear algebra dialect representation is reconstructed into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; In a hardware adaptive optimization stage, target hardware characteristics of a target hardware are identified, the target hardware characteristics including: hardware type, hardware structure, and parallelism; Adaptive execution of hardware optimization processing matched with the target hardware characteristics on the optimized intermediate representation to obtain a target intermediate representation, the hardware optimization processing matched with the target hardware characteristics at least including one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion; Converting the target intermediate representation into hardware operation units executable by the target hardware, loading the hardware operation units into the target hardware, and constructing an AI accelerator; In the AI accelerator, the hardware operation units are run on the target hardware by a hardware abstraction layer dialect to link the compile-time IR and the runtime, schedule corresponding computing resources in the target hardware, and complete hardware acceleration of the machine learning model. 2.The AI accelerator construction method of claim 1, wherein, The importing of machine learning models of different AI frameworks through a multi-framework adaptation interface, the parsing of a model representation of the machine learning models, and the conversion of the model representation into a form-unified linear algebra dialect representation by using an MLIR dialect conversion chain comprise: If the imported AI framework is PyTorch, triggering the Dynamo component to work by using the torch.compile function of PyTorch 2.0, dynamically modifying Python bytecode by using the hook CPython frame evaluation API, rewriting the bytecode to extract the Aten operator sequence, and generating an FX computation graph; Traversing the FX computation graph by using the SHARK Turbine tool, parsing the data types and operator names of the operators in the FX computation graph, and mapping the Aten operators to the Torch dialect operators defined in the Torch MLIR tool one by one; Calling the conversion interface of the Torch MLIR, converting the Torch dialect operators into MLIRModuleOps in the linear algebra dialect representation, and the MLIRModuleOps containing complete computing definitions of the machine learning models. 3.The AI accelerator construction method of claim 1, wherein, The target-independent optimization processing in the target-independent optimization stage, the retaining of core computing semantics in the linear algebra dialect, and the reconstruction of the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure comprise: The scheduling creation layer traverses all blocks and block operators within the linear algebra dialect representation corresponding MLIR ModuleOp, identifies root operators in all blocks as optimization centers, and assigns a unique identifier kRootOpAttr to the root operators; By fusing the group attribute kFusionGroupsAttr, the producer operators providing input to the root operator and the consumer operators using the output of the root operator are marked respectively, and the producer operators and the consumer operators are fused into the root operator; Linear algebra dialect operators that are not marked as the root operator and do not belong to the linalg.fill operator or the dequantization operator are assigned to an independent scheduling section; memory management optimization, redundant computation elimination and data access pattern optimization are performed synchronously to generate the optimized intermediate representation.

4. The AI accelerator construction method of claim 3, wherein, The fusion of the producer operators and the consumer operators into the root operator includes: The fuseRootsWithConsumers function is called to fuse the consumer operators into the root operator. 5.The AI accelerator construction method of claim 1, wherein, The adaptive execution of the hardware optimization process matched with the target hardware characteristics on the optimized intermediate representation includes: After identifying the target hardware characteristics, the null value filling layer takes the optimized intermediate representation as input and traverses all input tensors and output tensors of linear algebra operators in the optimized intermediate representation: For tensors that do not meet the target hardware characteristics, the tensor.pad operation is used to fill 0; for the filled tensors or tensors that meet the target hardware characteristics, the tensor.extract_slice operation is used to extract valid data, so that the tensor shape is aligned with the target hardware calculation method while keeping the output result of the machine learning model unchanged.

6. The AI accelerator construction method of claim 1, wherein, The adaptive execution of the hardware optimization process matched with the target hardware characteristics on the optimized intermediate representation includes: The parallel allocation layer obtains the iteration domain of the operator in the optimized intermediate representation based on the TilingInterface interface, calculates the tiling size and generates an external loop, generates a tiling operation implementation in the innermost loop, and allocates the cut operator to multiple workgroups so that the amount of tensor data processed by each workgroup meets the memory limit of the target hardware; After the parallel allocation layer is executed, the BatchTile layer combines the maximum private memory of the target hardware to calculate the tiling size set, ensuring that each dimension vector of each tensor is divided by the corresponding tiling size value in the tiling size set and the total memory occupancy after cutting is less than the maximum private memory; The scf::tileUsingSCF interface is called to complete tiling and introduce For control flow to realize on-chip memory reuse, and process the thread-in data access of the Reduction operator; After the BatchTile layer is executed, the KernelTile layer splits the complex calculation into sub-calculations matching the core of the target hardware according to the core calculation characteristics of the target hardware; the calculation definition and the memory transmission are separated by using respective control flows, and the time overheads of the calculation definition and the memory transmission are balanced by the memory optimization of the BatchTile layer.

7. The AI accelerator construction method of claim 6, wherein, After the calculation definition and the memory transmission are separated by using respective control flows and the time overheads of the calculation definition and the memory transmission are balanced by the memory optimization of the BatchTile layer, the method further includes: After the multi-layer tiling is executed, the memory allocation layer defines a SubSpanToAllocaPattern rewriting mode, performs memory optimization on the optimized intermediate representation, converts complex memory references in the optimized intermediate representation into memref::AllocaOp operations, applies for on-chip private memory of the target hardware, sets memory space attributes and alignment modes according to whether there is a vector user, and inserts a memref::CopyOp instruction to realize data copying from off-chip memory to on-chip private memory; After the memory allocation layer is executed, the memory offset flattening layer successively calls an IREEExpandStrided- Metadata Pass and an ExpandStridedMetadata Pass, converts a high-dimensional memory access expression into one dimension through affine transformation, and establishes a memory copying mapping relationship from high dimension to low dimension.

8. The AI accelerator construction method of claim 1, wherein, The method further includes that the optimized intermediate representation is adaptively executed to obtain a target intermediate representation by using a hardware optimization process matched with the characteristics of the target hardware, and the target intermediate representation is obtained. The vectorization layer takes the optimized intermediate representation as input, calls an linalg::vectorize interface to identify parallelizable scalar operations in the optimized intermediate representation, converts the scalar operations into vector operations, synchronously rewrites memory access modes, converts scalar memory loading into vector memory loading, and converts scalar memory storage operations into vector memory storage operations, and if the target hardware does not support vector operations, the LinalgToLoops method is called to convert linear algebra operators in the optimized intermediate representation into multi-layer loops. After the vectorization is executed, the vector flattening layer splits a high-dimensional vector operator into multiple one-dimensional vector operators, so that the obtained to-be-processed intermediate representation conforms to an LLVM vector type, which is a compile-time abstraction of a vector register of the target hardware. The software pipeline layer takes the to-be-processed intermediate representation after the vector flattening as input, rearranges instructions in a loop body, and obtains the target intermediate representation, so that multiple different iteration instructions are executed in parallel in each loop iteration.

9. The AI accelerator construction method of claim 1, wherein, The method further includes that the target intermediate representation is converted into a hardware operation unit executable by the target hardware by using a dialect step-by-step conversion process, and the target intermediate representation is taken as input. The target intermediate representation is taken as input, and a dialect step-by-step conversion process is executed; starting from a linear algebra dialect in the target intermediate representation, the following processes are sequentially performed: a tensor dialect, one-time buffering, a memory reference dialect, vectorization, a vector dialect, LLVM dialect conversion, and an LLVM dialect. The LLVM dialect file is converted into an assembly file corresponding to the target hardware by the LLVM compiler, the assembly file is built into a hardware language file in binary form and hardware operation instructions; and the hardware language file and the hardware operation instructions are fused into the hardware operation unit in combination with configuration information and hardware attribute information of the target hardware.

10. The AI accelerator construction method of claim 1, wherein, Before the hardware optimization processing that matches the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation, the method further includes: The device scheduling layer allocates tensors in the optimized intermediate representation to corresponding target hardware devices according to the computing capacity of the target hardware and the amount of tensor data; After entering the hardware adaptive optimization phase, a stream bound to a device is created according to the target hardware device where the tensor is located; the stream bound to a GPU is adapted to a CUDA backend, and the stream bound to a CPU is adapted to a CPU backend; the hardware optimization processing in each stream and the generation of the subsequent hardware operation unit are matched with the backend characteristics of the corresponding device; The streams pass tensor data through a data copying operation containing a memref::CopyOp, ensuring the correctness of cross-device data interaction.

11. An AI accelerator construction apparatus, characterized by comprising: The apparatus includes the following units, wherein, The importing unit is configured to import machine learning models of different AI frameworks through a multi-framework adaptation interface, parse a model representation of the machine learning models, and convert the model representation into a linear algebra dialect representation that is formally unified by using an MLIR dialect conversion chain; The optimization unit is configured to, in a target-independent optimization phase, retain core computing semantics in the linear algebra dialect by performing target-independent optimization processing, and reconstruct the linear algebra dialect representation into an optimized intermediate representation with an efficient data flow structure; the target-independent optimization processing at least includes operator fusion, memory management optimization, redundant computation reduction, and general data flow optimization; in a hardware adaptive optimization phase, target hardware characteristics of the target hardware are identified, the target hardware characteristics include: hardware type, hardware structure, and parallelism; hardware optimization processing that matches the target hardware characteristics is adaptively performed on the optimized intermediate representation to obtain a target intermediate representation; the hardware optimization processing that matches the target hardware characteristics at least includes one of the following: tensor shape adaptation, multi-layer tiling, memory optimization, and vector conversion; The building unit is configured to convert the target intermediate representation into a hardware operation unit executable by the target hardware, load the hardware operation unit into the target hardware, and build an AI accelerator. The AI accelerator is configured to connect the compiled-time IR and the runtime on the target hardware by the hardware abstraction layer dialect, schedule corresponding computing resources in the target hardware, and complete hardware acceleration of the machine learning model.

12. An electronic device, comprising: It includes: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing the AI accelerator building method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer software program, and the computer software program is executed by the processor to implement the AI accelerator building method of any one of claims 1-10.

14. A chip, characterized by The chip is loaded with a computer software program and / or a hardware unit for implementing the AI accelerator construction method according to any one of claims 1-10.

Images