Accelerator-based model deployment method and system

Abstract

The application discloses an accelerator-based model deployment method and system, and relates to the technical field of computer application. The method converts a model file into an intermediate representation, matches an operator node with a kernel interface function based on the intermediate representation, and generates instructions for compilation and deployment through the kernel interface function. Specifically, the intermediate representation is converted into an instruction file based on the kernel interface function, and the instruction file is deployed on a hardware accelerator. The application realizes instruction conversion through the kernel interface function, and the matching relationship between the kernel interface function and the operator facilitates the extension and deployment of new operators. The deployment method provided by the application is simple in logic, flexible and extensible in model deployment process, and is beneficial to improving the deployment efficiency and thus the inference capability of the model.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to an accelerator-based model deployment method and system. Background Technology

[0002] With technological advancements, the application areas of artificial intelligence (AI) models are becoming increasingly widespread, further accelerating their rapid iteration. Hardware accelerators serve as execution tools for AI models, deploying them to perform tasks such as inference and prediction. Therefore, improving the processing efficiency of AI models requires designing efficient model deployment systems.

[0003] In related technologies, deploying the software algorithm of a model on a hardware accelerator involves many and complex steps, such as generating a computation graph, generating hardware executable instructions, decoding and executing the instructions on the hardware accelerator, resulting in a long deployment cycle and thus affecting the inference performance of the model. Summary of the Invention

[0004] This application provides an accelerator-based model deployment method and system to at least solve the problem of long model deployment cycles in related technologies.

[0005] This application provides an accelerator-based model deployment method, comprising: receiving a model file and converting the model file into an intermediate representation; wherein the intermediate representation includes operator nodes; matching a kernel interface function according to the operator nodes, and converting the intermediate representation into an instruction file that can be read by the accelerator through the kernel interface function; the kernel interface function is a predefined instruction generation function based on model operators; and deploying the model to the accelerator according to the instruction file. This application matches kernel interface functions with operator nodes and generates compilation and deployment instructions through the kernel interface functions; simultaneously, it converts the model file into an intermediate representation, converts the intermediate representation into an instruction file based on the kernel interface function, and deploys the instruction file on a hardware accelerator. The deployment method provided by this application has simple logic, a flexible and scalable model deployment process, and is conducive to improving deployment efficiency, thereby improving the model's inference ability.

[0006] This application also provides an accelerator-based model deployment system, including:

[0007] The first conversion module is used to receive the model file and convert the model file into an intermediate representation; wherein the intermediate representation includes operator nodes;

[0008] The second conversion module is used to match a kernel interface function according to the operator node, and convert the intermediate representation into an instruction file that can be read by the accelerator through the kernel interface function; the kernel interface function is a predefined instruction generation function based on the model operator;

[0009] The deployment module is used to deploy the model to the accelerator according to the instruction file.

[0010] This application matches kernel interface functions with operator nodes and generates compilation and deployment instructions through these kernel interface functions. Simultaneously, it converts model files into intermediate representations, then converts these intermediate representations into instruction files based on the kernel interface functions, and finally deploys the instruction files on a hardware accelerator. The deployment method provided by this application has simple logic and a flexible, scalable model deployment process, which helps improve deployment efficiency and thus enhances the model's inference capabilities. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating an accelerator-based model deployment method provided in this application embodiment;

[0013] Figure 2 A schematic diagram of the structure of an accelerator-based model deployment system provided in this application embodiment;

[0014] Figure 3 A schematic diagram of another accelerator-based model deployment system provided in this application embodiment;

[0015] Figure 4 This application provides a schematic diagram of a structure for deploying a model to an FPGA-accelerator-based compilation system.

[0016] The above figures refer to the following reference numerals:

[0017] 11, Compiler front-end; 12, Compiler back-end; 13, Runtime library; 111, Computation graph unit; 112, Intermediate representation unit; 121, Encoding unit; 122, Memory unit; 131, Processor; 132, Accelerator. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] First, the concepts involved in this application will be explained:

[0021] Self-attention models (transformer models) are based on encoder-decoder structures and implement self-attention prediction mechanisms, making them widely applicable in natural language processing. Through parallel processing, they significantly accelerate training; simultaneously, they allow the model to directly focus on words at any position in the input sequence, thus enhancing modeling capabilities.

[0022] A Field-Programmable Gate Array (FPGA) is a chip that can be programmed to implement digital circuits with specific functions. It includes programmable logic blocks, which use lookup tables, flip-flops, and other methods to implement basic logic and storage functions. By configuring programmable interconnect resources, programmable logic blocks are connected to form digital circuits with specific functions. Through input / output blocks, it communicates with devices / units outside the FPGA, configuring different voltage standards and protocols. An FPGA can be used to create multiple independent circuits to achieve parallel task processing.

[0023] Direct memory access (DMA) is a technology that allows external computer devices (such as hard drives, graphics cards, sound cards, network cards, etc.) to exchange data directly with main memory without the continuous involvement of the central processing unit (CPU). DMA reduces the burden on the CPU, improves data transfer speed, and enables real-time data transfer operations.

[0024] The self-attention mechanism of the Transformer model significantly optimizes the parallel resource utilization of the underlying hardware through matrix-based global correlation computation. Field-Programmable Gate Arrays (FPGAs), with their high-concurrency computing architecture (such as programmable logic block arrays and distributed DSP modules) and hardware reconfigurability (supporting dynamic bitstream switching), provide a crucial foundation for building adaptive accelerator systems. This co-design bridges the adaptation gap between the rapid iteration of AI models and fixed hardware architectures through dynamic hardware logic reconfiguration (such as local reconfiguration techniques), thus opening up a hardware-level acceleration path for Transformer inference tasks. However, the long development cycle and broad software and hardware knowledge required for FPGA-based accelerators greatly hinder the deployment and application of models.

[0025] Specifically, with the rapid iterative evolution of Transformer models and their variants, hardware accelerators face significant challenges in model deployment. On one hand, deploying the model's software algorithms on a hardware accelerator involves numerous and complex steps, including generating a computation graph, generating hardware-executable instructions, and decoding and executing these instructions on the accelerator. Only by designing a flexible and efficient compilation system can the model's inference performance be fully realized on a hardware accelerator. On the other hand, rapid model iteration may introduce new operators, which often require redesigning hardware data paths or expanding instruction sets. This necessitates an adaptation process spanning multiple technical levels: from model structure parsing at the algorithm level and computational logic decomposition at the operator level to execution sequence reconstruction at the instruction set level, each step requires complex processing. This deep, cross-level adaptation significantly prolongs the deployment cycle of new models, severely restricting deployment efficiency. Therefore, compilation systems for model deployment on hardware accelerators have become a crucial research area.

[0026] To address this issue, this application presents a design approach for a hardware accelerator-based compilation system for artificial intelligence models. Based on computation kernel interface functions, a compilation backend architecture is designed that can flexibly extend to accommodate new operators introduced by new models. This approach minimizes modifications to the instruction set architecture and hardware logic design while enabling the system to efficiently execute inference tasks. The compilation system design method presented in this application is highly compatible with the rapidly evolving use of models in AI hardware accelerators. Using this design method, efficient model deployment and high-performance inference can be achieved.

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Reference Figure 1 This application provides a model deployment method based on accelerators, including the following steps:

[0029] Step S100: Receive the model file and convert the model file into an intermediate representation; wherein the intermediate representation includes operator nodes;

[0030] Step S200: Based on the operator node, match the kernel interface function, and convert the intermediate representation into an instruction file that can be read by the accelerator through the kernel interface function; the kernel interface function is a predefined instruction generation function based on the model operator;

[0031] Step S300: Deploy the model to the accelerator according to the instruction file.

[0032] This invention addresses the shortcomings of existing technologies by proposing a compilation method for deploying artificial intelligence models on hardware accelerators. This method allows the model's software algorithm to be flexibly and efficiently deployed to hardware accelerators, achieving high-performance model inference. The compilation process of an artificial intelligence model is a process of converting high-level, framework-dependent model descriptions into low-level, hardware-dependent, efficient executable code to improve the model's performance and efficiency on specific hardware (such as CPUs, GPUs, and NPUs). In this application, the model file can be model data; similarly, the file in this application can be understood as data. In this application, the model file is the model's software algorithm. It should be noted that the model deployment system and method provided in this application can be used for resource management and scheduling of server and storage systems; it can also be used for data processing on big data platforms. This application does not limit specific application areas and scenarios, and the artificial intelligence model in this application can be a model with any function and architecture.

[0033] In some embodiments, refer to Figure 2 and Figure 3 The deployment system of this application, as shown, includes a first conversion module 11 for receiving a model file and converting the model file into an intermediate representation; a second conversion module 12, connected to the first conversion module, including an encoding unit 121, which is used to: receive the intermediate representation, match a kernel interface function according to the operator node, and convert the intermediate representation into an instruction file that can be read by the accelerator through the kernel interface function; and a deployment module 13, connected to the second conversion module, for receiving the instruction file and deploying the model on the accelerator according to the instruction file. In some embodiments, the first conversion module of this application may be a compiler front-end, the second conversion module may be a compiler back-end, and the deployment module may be a runtime library.

[0034] Understandably, the intermediate representation acts as a bridge between the compiler front-end and back-end. It represents the computation graph in a hardware-independent form, facilitating various optimizations. The compiler front-end converts the model written in a high-level language by engineers into an initial intermediate representation of the computation graph. This intermediate representation includes operator nodes, which represent computational units or operations. The compiler back-end converts the optimized intermediate representation into an instruction file that can be actually read by the hardware accelerator. Specifically, this conversion is achieved by calling kernel interface functions. The computation kernel interface functions can be compiler-generated and called by the compiler, serving as functions for specific computational tasks executed by the hardware. Specifically, by providing kernel interface functions with the same interface for different hardware, reuse (write once, run multiple times) is achieved. In this application, the kernel interface functions are related to the model's operators. If a new operator is introduced into the artificial intelligence model, a new kernel interface function corresponding to the new operator can be established to adapt the new model. This application, through the design of kernel interface functions, achieves efficient deployment of the artificial intelligence model compilation process, improves the model's deployment efficiency on hardware accelerators, and enhances the model's inference capabilities.

[0035] Optionally, the method provided in this application also includes:

[0036] The system receives and extracts the first data from the data nodes in the intermediate representation, preprocesses the first data, and obtains a data file; wherein the data file provides parameters for model deployment.

[0037] In some embodiments, refer to Figure 3 The compilation system shown includes a second conversion module, which further includes a memory unit 122. The memory unit is connected to the first conversion module and is used to receive and extract the first data of the data nodes in the intermediate representation, preprocess the first data, and obtain a data file.

[0038] It is understood that in this application, the compiler backend converts the intermediate representation into a data file, which contains parameters required for model deployment, such as weights and biases. It is also understood that the artificial intelligence model in this application can be a Transformer model, a convolutional neural network model, or a recurrent neural network model; this application does not limit the specific type and function of the artificial intelligence model. For different types of artificial intelligence models, the parameters involved in the data file differ, and those skilled in the art can set the parameters contained in the data file according to actual needs. This application converts the first data in the intermediate representation into a data file that is more easily recognized by hardware through memory units, providing parameter support for model deployment. By using more accurate parameters for model deployment, the accuracy of model deployment is improved.

[0039] Optionally, the method provided in this application also includes:

[0040] Receive the intermediate representation, traverse the operator nodes in sequence, and generate microinstructions based on the kernel interface function;

[0041] The micro-instructions are integrated sequentially to generate an instruction file.

[0042] In one embodiment, reference is made to Figure 3 The deployment system shown includes an encoding unit comprising an interface subunit and an instruction subunit connected together. The interface subunit is used to receive the intermediate representation, traverse the operator nodes in sequence, and generate microinstructions based on the core interface function. The instruction subunit is used to receive the microinstructions in sequence, integrate the microinstructions, and generate an instruction file.

[0043] This application traverses operator nodes in node order, matches kernel interface functions based on operator types, and generates microinstructions based on the matched kernel interface functions. Specifically, fields in the intermediate representation are mapped to formal parameters of the kernel interface function, the kernel interface function is executed, and microinstructions are generated. It should be noted that this application pre-matches operator types and kernel interface functions, determining the kernel interface function based on the operator type of the operator node, and thus determining the microinstructions. Of course, those skilled in the art can also pre-match other features of operator nodes with kernel interface functions, and during the instruction generation stage, call the kernel interface function based on a specified type using the pre-matching relationship, thereby generating microinstructions based on the kernel interface function. The instruction subunit integrates the microinstructions according to the above node order to generate an instruction file. The instruction file can be loaded on a hardware accelerator. This application, through matching operator types with kernel interface functions, calls kernel interface functions to generate microinstructions and an instruction file, and through file conversion by the encoding unit, generates an instruction file that can be directly read by the hardware counter. The process is simple and can be used for various types of artificial intelligence models, improving compilation efficiency.

[0044] One embodiment of this application provides a deployment method that further includes: determining an instruction generation function based on the hardware logic computation type and providing a calling interface to match the core interface function and generate microinstructions. In this application, the core interface function is an instruction generation function generated based on the hardware logic computation model. Specifically, the compiler backend pre-implements an instruction generation function for each hardware logic computation type, including matrix multiplication, convolution, Softmax, LayerNorm, GELU, etc. The internal structure of the instruction generation function corresponds to the microarchitecture of the hardware unit to achieve a correspondence between the core interface function and the hardware instruction file, facilitating the subsequent deployment of the instruction file on the hardware accelerator.

[0045] Optionally, the method provided in this application also includes:

[0046] Define a data structure for the computation kernel descriptor, generate a computation kernel descriptor based on the intermediate representation node and the data structure, and pass the computation kernel descriptor to the kernel interface function to generate an instruction file based on the kernel interface function.

[0047] This application defines a data structure for the computation kernel descriptor based on the specific type of the artificial intelligence model. The interface subunit, based on the current intermediate representation node and according to the defined data structure, generates a computation kernel descriptor and passes it to the kernel interface function to generate microinstructions. This application uses the descriptor component to determine the kernel interface function corresponding to the current node, and then generates an instruction file based on the kernel interface function. This provides preliminary setup for the subsequent deployment of instruction files on the accelerator, enabling efficient deployment of the artificial intelligence model. Simultaneously, it ensures complete alignment between the generated instructions and the hardware implementation, which helps avoid the error-proneness and inefficiency of manually writing assembly or microcode, thus improving model deployment efficiency.

[0048] In one embodiment of this application, the computation kernel descriptor includes data dimensions, memory access mode addresses, and computation parameters. The descriptor component is used to: parse the intermediate representation nodes, extract the data dimensions, memory access mode addresses, and computation parameters, and generate the computation kernel descriptor. This application defines the specific content of the kernel descriptor's data structure to establish accurate kernel interface functions, and then generates accurate instruction files through the kernel interface functions, thereby achieving accurate deployment and operation of the artificial intelligence model.

[0049] Optionally, the method provided in this application also includes:

[0050] If a new operator is introduced into the model, a new kernel interface function is constructed based on the new operator, and the new kernel interface function is stored in the function library to provide instruction adaptation for the new operator.

[0051] This application pre-establishes a function library based on kernel descriptors to facilitate the operation of the encoding unit. Furthermore, when the artificial intelligence model involves new operators, a new kernel interface function for the new operator is constructed and added to the pre-established function library. When compiling the artificial intelligence model, if the new operator is involved, instruction adaptation can be achieved through this new kernel interface function. This alleviates the problem in related technologies where new operators require redesigning hardware data paths or expanding instruction sets, necessitating adaptation processes that span multiple technical levels. Deep adaptation across levels significantly prolongs the deployment cycle of new models, severely restricting deployment efficiency. This application minimizes changes to the instruction set architecture and hardware logic design while enabling the system to efficiently execute inference tasks. Moreover, this application associates kernel interface functions with model operators, providing a standardized and scalable access method for future additions of operators. This standardized approach allows instruction adaptation to new operators to be completed simply by adding new computation kernel interface functions when model changes introduce new operators. This helps alleviate the cumbersome process of adapting by modifying the instruction set architecture in related technologies, improving model deployment efficiency.

[0052] Optionally, receiving a model file and converting the model file into an intermediate representation includes:

[0053] Receive the model file and convert it into a computational graph based on network exchange rules;

[0054] Based on the computation graph, the nodes in the computation graph are mapped to hardware logic units;

[0055] Based on the data size and data dependencies of the computation graph, the memory address of the tensor is determined, and then an intermediate representation is generated.

[0056] In one embodiment, see Figure 3 The compilation system shown includes a first conversion module comprising a connected computation graph unit 111 and an intermediate representation unit 112. The computation graph unit receives a model file and converts the model file into a computation graph based on network exchange rules. The intermediate representation unit receives the computation graph and maps the nodes in the computation graph to hardware logic units. Based on the data scale and data dependencies of the computation graph, the memory address of the tensor is determined, and then an intermediate representation is generated.

[0057] The model files in this application are obtained through different deep learning frameworks (software-level operator descriptions). Through network exchange rules, the model files are parsed into a structurally consistent and semantically complete computation graph. The network exchange rules can be based on the ONNX specification or NNEF. This application does not limit the specific implementation of the network exchange rules; those skilled in the art can design corresponding network exchange rules based on the specific type of artificial intelligence model and accelerator. The computation graph in this application includes tensor data and nodes, with nodes corresponding to specific computational operations and attribute parameters. Tensors include multidimensional arrays, which are the basic units for representing and transforming data in AI models. The intermediate representation unit receives the computation graph and performs hardware-oriented semantic transformations on it. First, the nodes in the computation graph are mapped to already implemented hardware logic units. Based on the data flow and data dependencies in the computation graph, memory addresses are allocated to the tensors, generating a computational flow that can be piped on hardware and an instruction-oriented intermediate representation.

[0058] The network exchange rules explicitly, structurally, and statically represent all the computational steps, data flows, and parameter configurations that were originally scattered throughout the front-end code. This forms a universal description file (computation graph) that is cross-platform, persistent, and version-manageable, laying a solid foundation for subsequent hardware-oriented optimization and mapping. The generated intermediate representation can be directly read by subsequent encoding units. Based on the recorded computational flow and parameter information, it automatically generates all the instruction sequences required for the AI ​​hardware accelerator to execute, enabling the accelerator to efficiently complete model inference tasks during actual runtime.

[0059] In some embodiments, the nodes of the computation graph include: the operation type and attribute parameters of the model.

[0060] In this application, computation graph nodes include at least one of the model's operation type and attribute parameters. Specifically, the operation type includes the types of operations involved in the artificial intelligence model, and the attribute parameters include related parameters such as convolution kernels and activation functions involved in the artificial intelligence model. The specific content of the operation type and attribute parameters can be determined according to the actual artificial intelligence model type and hardware accelerator type. This application stores all the information that the operation needs to know at runtime in the computation graph through the operation type and attribute parameters in the computation graph nodes, which facilitates subsequent hardware-oriented optimization and mapping to efficiently execute the compilation process.

[0061] Optionally, deploying the model to the accelerator according to the instruction file includes:

[0062] Allocate an address for the instruction file;

[0063] Based on the address, the instruction file is read, instruction filling and decoding reasoning are performed, the reasoning result is obtained, and the reasoning result is recorded.

[0064] In some embodiments, see Figure 3 The deployment system shown includes a deployment module comprising a processor 131 and an accelerator 132 connected to each other. The processor is used to receive the instruction file, allocate an address to the instruction file, and send the address to the accelerator. The accelerator is used to receive the address, read the instruction file, perform instruction filling and decoding inference to obtain the inference result, and feed the inference result back to the processor.

[0065] Specifically, in this application, the processor stores the instruction file and data file in registers, and the accelerator reads the instruction file and data file from the registers, performs decoding inference, and obtains the inference result. Register access can be configured according to actual needs. It should be noted that the accelerator in this application can be an FPGA-based hardware accelerator, a CPLD-based hardware accelerator, or a GPU-based hardware accelerator. This application does not limit the specific type of accelerator. This application achieves efficient runtime library operation and efficient model deployment and operation on the hardware accelerator through the configuration of the processor and accelerator.

[0066] Optionally, allocating an address for the instruction file includes:

[0067] An address is allocated to the instruction file, and a first register, a second register, and a third register are configured; wherein, the first register is used to enable or stop instruction fetching, the second register is used to configure the preset number of instructions to be read, and the third register is used to determine the starting address of the instruction stream.

[0068] This application can store instruction files and data files in registers, and configure the instruction file reading logic through multiple registers to store and read instruction files and data files in sequence, thereby improving the accuracy of data during data transmission.

[0069] This application also provides a method for transmitting instruction files, including receiving inference requests, moving instruction files and data files to the off-chip storage of the accelerator, writing the starting address and length to the task descriptor, and updating the doorbell register;

[0070] The accelerator receives the doorbell interrupt, parses the task descriptor, extracts information from the instruction file and data file, and writes the information into the first register, the second register, and the third register.

[0071] Optionally, based on the address, the instruction file is read, and instruction filling is performed, including:

[0072] If the first register is set to enable instruction fetching, the instruction file is read according to the starting address;

[0073] If the number of instructions read from the instruction file is greater than or equal to the preset number of instructions, the first register is set to stop fetching instructions, and a value fetching completion signal is issued.

[0074] This application determines the instruction fetching operation through a first register, the number of instructions to be fetched through a second register, and the starting address for instruction fetching through a third register. By configuring multiple registers and access logic, the instruction file and data file are read sequentially, improving data accuracy and deployment accuracy.

[0075] Optionally, decoding reasoning is performed to obtain the reasoning result, including:

[0076] The instruction file is parsed to obtain instruction parameters. Based on the mapping relationship of the instruction parameters, operation signals are generated to drive calculation and logic processing.

[0077] If the processing of the first operator is complete, a completion counter signal is issued.

[0078] If the decoding and reasoning of the instruction file is completed, a reasoning completion signal is issued;

[0079] If a pre-defined exception occurs during processing, an error log will be issued and an interrupt will be triggered.

[0080] This application pre-defines exceptions including out-of-bounds and division by zero. The host receives an inference completion signal, indicating the completion of one end-to-end inference operation. A counter signal is used for host polling. The computation and logic processing in this application includes driving the DMA engine, driving the computation unit, and driving the logic unit. This application deploys the model by decoding inference; simultaneously, it monitors the deployment process in real time by sending relevant signals and interrupt information, enabling timely detection and handling of exceptions, thus improving deployment efficiency.

[0081] The accelerator-based model deployment system and method provided in this application will be described in detail below with reference to specific embodiments:

[0082] The compilation system proposed in this invention is as follows: Figure 4 As shown, it consists of three parts: compiler front-end, compiler back-end, and runtime library.

[0083] 1. Compiler front-end.

[0084] The core task of the compiler front-end is to completely and losslessly convert the original model software algorithm into an intermediate representation (IR) that can be directly recognized and efficiently executed by AI hardware accelerators. To achieve this goal, the front-end is clearly divided into two major functional modules: a computation graph generation module and an FPGA intermediate representation generation module.

[0085] The computation graph generation module first uses the Open Neural Network Exchange (ONNX) as a unified bridge to parse model files trained from different deep learning frameworks (such as PyTorch, TensorFlow, Keras, MindSpore, etc.). ONNX's standardized format ensures that regardless of the initial framework in which the model was built, it can be converted into a structurally consistent and semantically complete computation graph. In this computation graph, each edge represents tensor data and its metadata (such as data type, dimensionality, memory layout, source node, and target node), and each node corresponds to a specific computational operation (e.g., convolution, matrix multiplication, activation, normalization, pooling, attention, etc.). The node not only records the operation type but also includes all necessary attribute parameters: kernel size, stride, padding method, number of groups, transpose of matrix multiplication, alpha / beta values ​​of the activation function, number of attention heads and scaling factor, etc. All information required for the operation to run is completely stored in the computation graph. In this way, the computation graph generation module makes all the computational steps, data flow, and parameter configurations that were originally "implicitly" scattered in the Python front-end code explicit, structured, and static, forming a universal description file that is cross-platform, persistent, and version-manageable, laying a solid foundation for subsequent hardware-oriented optimization and mapping.

[0086] Once the computation graph and all parameter information are ready, the FPGA intermediate representation generation module begins operation. This module takes the computation graph and parameter information as input and performs a hardware-semantic transformation: First, based on the operation type and parameters of each node in the computation graph, it maps them to the hardware logic units already implemented on the AI ​​hardware accelerator (such as configurable systolic arrays, lookup tables, on-chip accumulators, DMA engines, etc.). Second, based on the data scale and data dependencies, it determines the data layout strategy of tensors between on-chip SRAM and external DDR; simultaneously, it allocates a unique memory address range for each tensor to ensure conflict-free data access during runtime. Furthermore, following the data flow order of the computation graph, it generates an operational flow that can be piped to hardware. Finally, it outputs a new intermediate representation, which is no longer a "software-level" operator description like ONNX, but an instruction-oriented IR composed of hardware logic operation types (i.e., operator nodes). This IR will be directly read by the subsequent instruction generation module, which, based on the recorded operational flow and parameter information, automatically generates all the instruction sequences required for execution by the AI ​​hardware accelerator, thus ensuring that the accelerator can efficiently complete model inference tasks during actual operation.

[0087] 2. Compiler backend.

[0088] The core task of the compiler backend is to convert the instruction-oriented FPGA intermediate representation already generated by the frontend into instruction encoding files that the AI ​​hardware accelerator can actually read and execute, as well as data files such as weights and biases required for model deployment on the board. Its workflow can be summarized as follows: for each operator node in the IR, call the corresponding computation kernel interface function (i.e., the predefined instruction generation function) to output the complete instruction sequence at once.

[0089] (1) The overall process of instruction generation.

[0090] The backend first sequentially traverses the nodes in the IR (Instructions Representation). Each node carries information such as operator type, tensor dimension, data flow direction, on-chip / off-chip address offset, parallelism, and iteration count. The backend selects the corresponding computational core interface function based on the operator type, maps the fields in the IR to the formal parameters of that function, and then executes the function to obtain all the microinstructions required by the node on the hardware. The instructions from all nodes are concatenated sequentially to form a single instruction binary file that can be loaded onto the AI ​​accelerator.

[0091] (2) Design of the kernel interface function.

[0092] To shield the subtle differences between various operators, the backend pre-implements an instruction generation function for each hardware-supported logical computation type, such as matrix multiplication, convolution, Softmax, LayerNorm, and GELU. Each function fully understands the microarchitecture of the corresponding hardware unit: systolic array size, on-chip SRAM count, DMA burst length, register configuration, etc. These functions expose a unified C / C++ or Python calling interface. During the compilation phase, the backend only needs to perform a switch-case dispatch based on the op_type field in the IR to reuse all the pre-built logic.

[0093] The interface subunit is used to: determine the instruction generation function based on the hardware logic computation type, and provide a calling interface to match the core interface function and generate microinstructions. The core interface function is an instruction generation function generated based on the hardware logic computation model.

[0094] (3) Compute Kernel Descriptor (CKD).

[0095] To seamlessly transfer high-dimensional information from the IR to each computational core interface function, the backend defines a unified CKD data structure, which encapsulates three types of core information:

[0096] Data dimensions: feature map height, width, number of channels, number of rows / columns of weight matrix, bias vector length, batch dimension, and any pad / stride related dimensions.

[0097] Memory access modes and addresses, including: Read / write modes: sequential, block mode. Address information: base address, offset, and row span of the input tensor in DDR / HBM; write-back address of the output tensor; starting row number of each bank in the on-chip SRAM; DMA descriptor index, etc. Computation parameters: such as computation type (matrix multiplication, Layernorm, Softmax, GELU, etc.), block computation scale of matrix multiplication, systolic array size, and parameter values ​​required for register configuration, etc.

[0098] (4) Construction and calling of the kernel function library.

[0099] In the early stages of compiler development, a computation kernel function library based on computation kernel descriptors is maintained. In the backend instruction generation stage, the process becomes extremely concise, including the following steps:

[0100] Step S41: parse the IR nodes and populate the compute kernel descriptor instances;

[0101] Step S42: Find the corresponding kernel function based on op_type;

[0102] Step S43: Call the kernel function, pass in the kernel descriptor, and the kernel function returns the binary execution instructions required for the operation;

[0103] Step S44: Append this instruction segment to the final instruction file;

[0104] Step S45: Repeat steps S41 to S44 until all nodes have been processed.

[0105] Through the "IR-CKD-Kernel function-instruction" pipeline provided in steps S41 to S45 above, the backend can efficiently generate the model's instruction sequence while ensuring complete alignment between the generated instructions and the hardware implementation. This avoids the error-proneness and inefficiency of manually writing assembly or microcode, and provides a standardized and scalable access method for future additions of operators. This standardized approach allows for instruction adaptation of new operators introduced when the model changes, simply by adding new computation kernel interface functions, avoiding the cumbersome process of modifying the instruction set architecture. This application aligns the instruction file with the hardware, facilitating model deployment on accelerators.

[0106] (5) Derivation of weight and bias data.

[0107] The data nodes in the IR are analyzed, and the weight tensors, bias vectors, and alpha / beta of LayerNorm are extracted. The data is rearranged and padded according to the on-chip SRAM / DDR alignment requirements of the accelerator. Finally, the .bin file of the data is generated.

[0108] 3. Runtime library.

[0109] The core responsibility of the runtime library is to execute the instruction files and weight / bias data files generated by the compiler backend on the FPGA-based AI hardware accelerator, thereby truly completing the model deployment. The entire process revolves around the collaboration between the host and the FPGA-based AI accelerator, and can be divided into five major stages: offline preparation, online transmission, register programming, instruction cache filling, and decoding execution. Details are presented below in chronological order.

[0110] (1) Offline preparation.

[0111] The compiler backend translates network topology, operator parameters, memory layout, and other information into two sets of binary files, including:

[0112] .ins instruction files: Instruction streams customized according to the accelerator microarchitecture, including DMA descriptors, compute core configurations, synchronization, etc.

[0113] .data file: Reformatted weights, biases, intermediate feature map data, etc.

[0114] These two sets of files are pre-allocated on the host side and their physical addresses and lengths are registered with the host driver.

[0115] (2) Online transmission.

[0116] When a user process issues an inference request via the runtime API, the host driver first uses the PCIe DMA engine to batch move the .ins and .data files to the off-chip DDR on the accelerator board. After the transfer is complete, the driver writes the starting address and length to a task descriptor and updates the doorbell register to notify the accelerator firmware that a new task has arrived.

[0117] (3) Register configuration.

[0118] After receiving the doorbell interrupt, the accelerator firmware parses the task descriptor and then writes the key information into three memory-mapped registers via MMIO (Memory Mapped I / O): ins_control register (i.e., the first register): controls the start / stop of instruction fetching; ins_count register (i.e., the second register): configures the number of instructions to be read; ins_addr register (i.e., the third register): specifies the starting physical address of the instruction stream in off-chip memory.

[0119] The write order is ins_addr-ins_count-ins_control to ensure that the accelerator side completes atomic configuration in one go.

[0120] (4) Instruction cache filling.

[0121] When the ins_control register is set to 1, the instruction fetch unit inside the accelerator starts, reads the instruction stream, and writes it into the instruction buffer. When the number of fetched instructions reaches ins_count, the hardware automatically clears ins_control and pulls the instruction fetch complete interrupt high.

[0122] (5) Decoding execution.

[0123] The instruction decoder sequentially scans the instruction buffer, parsing the opcode, functional unit field, register number, and immediate value in the instruction word; based on the opcode mapping table, the decoder generates micro-operation signals to drive:

[0124] DMA engine (responsible for data transfer between off-chip DDR and on-chip SRAM);

[0125] Computational units (matrix multiplication units / special function units);

[0126] Synchronization logic (ensuring the dependent order of computation and data transmission).

[0127] During execution, the accelerator reports three types of events via PCIe:

[0128] After each operator is completed, write back the "completion counter" for the host to poll.

[0129] Once the entire network operation is complete, raise the "Inference Done" MSI-X interrupt.

[0130] If an out-of-bounds error or division by zero occurs, an error is written to the 64-bit error log register and an interrupt is triggered.

[0131] Once the host receives the Inference Done, it has completed one end-to-end inference operation.

[0132] Thus, the runtime library, through the complete chain of "host-side file preparation - PCIe - transmission - register configuration - instruction cache filling - decoding execution", has achieved efficient deployment and operation of the model on an FPGA-based AI hardware accelerator.

[0133] Based on the above model, the compilation system design for inference deployment on an FPGA-based AI hardware accelerator allows for the deployment of the model inference process from software algorithms to hardware accelerators using the compilation system proposed in this invention. This approach not only enables efficient deployment of model inference but also facilitates the expansion of new operators and accelerates the adaptation process of new models due to the use of instruction generation based on computational kernel interface functions.

[0134] This invention proposes a compilation system for inference deployment on an FPGA-based AI hardware accelerator for Transformer models. Specifically, it presents the design and implementation of this compilation system; a design method for instruction generation based on computation kernel interface functions; and an extension method supporting new operators. This application matches operator nodes to kernel interface functions and uses these functions to generate instructions in the compiler backend. Simultaneously, it converts model files into intermediate representations through a compiler frontend architecture and deploys the instruction files on the hardware accelerator using a runtime library. The deployment system provided by this application has a simple architecture, a flexible and scalable model deployment process, which improves deployment efficiency and thus enhances the model's inference capabilities.

[0135] It should be noted that the architecture, connection methods, and processing logic in the above examples are all exemplary and do not substantially limit the solution of this application.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0137] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the accelerator-based model deployment method.

[0138] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above embodiments of the accelerator-based model deployment method at runtime.

[0139] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0140] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the accelerator-based model deployment method.

[0141] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the accelerator-based model deployment method.

[0142] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] The above provides a detailed description of an accelerator-based model deployment system and method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model deployment method based on accelerators, characterized in that, The method includes: Receive a model file and convert the model file into an intermediate representation; wherein the intermediate representation includes operator nodes; Based on the operator node, a kernel interface function is matched, and the intermediate representation is converted into an instruction file that can be read by the accelerator through the kernel interface function; the kernel interface function is a predefined instruction generation function based on the model operator; According to the instruction file, the model is deployed to the accelerator; The method further includes: Receive the intermediate representation, traverse the operator nodes in sequence, and generate microinstructions based on the kernel interface function; The microinstructions are sequentially integrated to generate an instruction file; If a new operator is introduced into the model, a new kernel interface function is constructed based on the new operator, and the new kernel interface function is stored in the function library to provide instruction adaptation for the new operator; the function library is a pre-built function library based on kernel descriptors.

2. The accelerator-based model deployment method according to claim 1, characterized in that, The method further includes: Define a data structure for the computation kernel descriptor, generate a computation kernel descriptor based on the intermediate representation node and the data structure, and pass the computation kernel descriptor to the kernel interface function to generate an instruction file based on the kernel interface function.

3. The accelerator-based model deployment method according to claim 1, characterized in that, The receiving model file, and converting the model file into an intermediate representation, includes: Receive the model file and convert it into a computational graph based on network exchange rules; Based on the computation graph, the nodes in the computation graph are mapped to hardware logic units; Based on the data size and data dependencies of the computation graph, the memory address of the tensor is determined, and then an intermediate representation is generated.

4. The accelerator-based model deployment method according to claim 1, characterized in that, Deploying the model to the accelerator according to the instruction file includes: Allocate an address for the instruction file; Based on the address, the instruction file is read, instruction filling and decoding reasoning are performed, the reasoning result is obtained, and the reasoning result is recorded.

5. The accelerator-based model deployment method according to claim 4, characterized in that, The process of allocating an address for the instruction file includes: An address is allocated to the instruction file, and a first register, a second register, and a third register are configured; wherein, the first register is used to enable or stop instruction fetching, the second register is used to configure the preset number of instructions to be read, and the third register is used to determine the starting address of the instruction stream.

6. The accelerator-based model deployment method according to claim 5, characterized in that, Based on the address, read the instruction file and perform instruction filling, including: If the first register is set to enable instruction fetching, the instruction file is read according to the starting address; If the number of instructions read from the instruction file is greater than or equal to the preset number of instructions, the first register is set to stop fetching instructions, and a value fetching completion signal is issued.

7. The accelerator-based model deployment method according to claim 4, characterized in that, Perform decoding reasoning to obtain the reasoning results, including: The instruction file is parsed to obtain instruction parameters. Based on the mapping relationship of the instruction parameters, operation signals are generated to drive calculation and logic processing. If the processing of the first operator is complete, a completion counter signal is issued. If the decoding and reasoning of the instruction file is completed, a reasoning completion signal is issued; If a pre-defined exception occurs during processing, an error log will be issued and an interrupt will be triggered.

8. An accelerator-based model deployment system, characterized in that, The system includes: The first conversion module is used to receive the model file and convert the model file into an intermediate representation; wherein the intermediate representation includes operator nodes; The second conversion module is used to match a kernel interface function according to the operator node, and convert the intermediate representation into an instruction file that can be read by the accelerator through the kernel interface function; the kernel interface function is a predefined instruction generation function based on the model operator; The deployment module is used to deploy the model to the accelerator according to the instruction file; The second conversion module is used for: Receive the intermediate representation, traverse the operator nodes in sequence, and generate microinstructions based on the kernel interface function; The microinstructions are sequentially integrated to generate an instruction file; If a new operator is introduced into the model, a new kernel interface function is constructed based on the new operator, and the new kernel interface function is stored in the function library to provide instruction adaptation for the new operator; the function library is a pre-built function library based on kernel descriptors.

Images