Model compiling method and device, electronic equipment and storage medium

Abstract

The application discloses a model compiling method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a target format of intermediate representation data which can be compatible with multiple different target hardware platforms; analyzing a model to be compiled, so as to represent multiple network layers included in the model as structured data in a preset format respectively; performing fusion processing on the structured data corresponding to at least two first network layers in the model, so as to combine the operation operations of the at least two first network layers; converting the structured data of the multiple network layers of the model after the fusion processing into the intermediate representation data in the target format and filling in a preset two-dimensional array, so as to obtain a two-dimensional array representation of the model. Through the standardized intermediate representation, the operator level optimization and the conversion of the efficient data structure, the fast and accurate compatibility and operation of the model on different hardware platforms are realized.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method for compiling a model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Different hardware platforms in related technologies are designed with their own unique intermediate representations of the model. The implementation and optimization of the model on different hardware platforms are often limited by the platform-specific intermediate representations, resulting in the model being difficult to migrate between different platforms and having poor compatibility. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for compiling models, in order to at least solve the problem in the related art that the intermediate representations used by models on different platforms are different, resulting in poor model portability and compatibility between different platforms.

[0004] This application provides a method for compiling a model, comprising: obtaining a target format for intermediate representation data that is compatible with multiple different target hardware platforms; parsing the model to be compiled to represent multiple network layers of the model as structured data in a preset format, wherein the structured data includes parameter information of the corresponding network layers; fusing the structured data corresponding to at least two first network layers in the model to merge the operations of the at least two first network layers, wherein the at least two first network layers include associated consecutive operations; converting the structured data of the multiple network layers of the model after the fusion process into intermediate representation data in the target format and filling it into a preset two-dimensional array to obtain a two-dimensional array representation of the model, wherein the structured data of each network layer corresponds to a set of vector data in the two-dimensional array, and the model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

[0005] This application also provides a model compilation apparatus, comprising:

[0006] The acquisition module is used to acquire the target format of intermediate representation data that is compatible with multiple different target hardware platforms;

[0007] The parsing module is used to parse the model to be compiled, so as to represent the multiple network layers included in the model as structured data in a preset format, wherein the structured data includes the parameter information of the corresponding network layers;

[0008] A fusion module is used to fuse the structured data corresponding to at least two first network layers in the model to merge the computational operations of the at least two first network layers, wherein the at least two first network layers include associated consecutive computational operations;

[0009] The compilation module is used to convert the structured data of multiple network layers of the model after the fusion process into intermediate representation data in the target format and fill it into a preset two-dimensional array to obtain a two-dimensional array representation of the model. The structured data of each network layer corresponds to a set of vector data in the two-dimensional array, and the model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

[0010] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the compilation method of any of the above models when executing the computer program.

[0011] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the compilation method of any of the above models.

[0012] This application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the compilation method of any of the above models.

[0013] This application eliminates platform barriers by defining a universal intermediate representation format, allowing models to migrate freely across different hardware platforms, reducing development and deployment costs, and enhancing model portability and ecosystem uniformity. Using structured data to describe each layer of the model simplifies the model parsing process and provides standardized input for subsequent operator fusion and instruction generation, enhancing model readability and operability. Operator fusion effectively reduces memory access and computational unit overhead, improving the parallelism and throughput of model execution, thereby reducing model inference latency and improving overall performance. Two-dimensional array representation simplifies model loading and execution, reducing the time complexity of model parsing. Simultaneously, the unified format makes model deployment on different hardware platforms faster and more efficient, further improving model execution efficiency. Through standardized intermediate representation, operator-level optimization, and efficient data structure transformation, the model achieves fast and accurate compatibility and operation on different hardware platforms, improving the speed and energy efficiency of model inference in edge computing and cloud computing scenarios. Attached Figure Description

[0014] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a hardware structure block diagram of a server device for a model compilation method according to an embodiment of this application;

[0016] Figure 2 This is one of the flowcharts of a model compilation method according to an embodiment of this application;

[0017] Figure 3 This is a second flowchart of a compilation method for a model according to an embodiment of this application;

[0018] Figure 4 This is the third flowchart of a compilation method for a model according to an embodiment of this application;

[0019] Figure 5 This is the fourth flowchart of a compilation method for a model according to an embodiment of this application;

[0020] Figure 6 This is the fifth flowchart of a compilation method for a model according to an embodiment of this application;

[0021] Figure 7 This is a flowchart of a compilation method for a model according to an embodiment of this application;

[0022] Figure 8 This is flowchart seven of a model compilation method according to an embodiment of this application;

[0023] Figure 9 This is the eighth flowchart of a compilation method for a model according to an embodiment of this application;

[0024] Figure 10 This is flowchart nine of a model compilation method according to an embodiment of this application;

[0025] Figure 11 This is flowchart ten of a model compilation method according to an embodiment of this application;

[0026] Figure 12 This is flowchart eleven of a model compilation method according to an embodiment of this application;

[0027] Figure 13 This is a structural block diagram of a compilation device for a model according to an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0029] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The specific application environment architecture or specific hardware architecture on which the compilation method of the model depends is described here.

[0032] The model compilation method embodiments provided in this application can be executed on a server device or similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a model compilation method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor, or programmable logic device) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the compilation method of the model in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0035] This application provides a method for compiling a model, applied to the aforementioned server device. The method is described in detail below, along with its execution flow. Figure 2 As shown, the method includes the following steps S200-230:

[0036] Step S200: Obtain the target format of intermediate representation data that is compatible with multiple different target hardware platforms.

[0037] Specifically, an intermediate representation format is defined that is cross-platform compatible, meaning it can be effectively parsed and executed on a variety of hardware platforms without being restricted to a specific platform's proprietary format.

[0038] For example, this study examines the characteristics of intermediate representation (IR) formats across different platforms, extracting common model description elements and parameters to form a basic, open IR framework. A general data structure, such as layer format data, is designed to contain all necessary model information, ensuring accurate representation of the model's structure and parameters while supporting necessary operators and data operations. Intermediate representation is a format transformed from source code during compilation, facilitating further optimization and generation of executable code for the target platform. The target hardware platform refers to the various hardware components on which the deep learning model will run.

[0039] Step S210: The model to be compiled is parsed so that the multiple network layers included in the model are represented as structured data in a preset format.

[0040] The structured data includes parameter information for the corresponding network layers.

[0041] Specifically, deep learning models are parsed and transformed into structured data in a preset format, such as layer format data. This data structure clearly describes the parameters and properties of each network layer, facilitating subsequent fusion and optimization processing.

[0042] For example, the model to be compiled is read, such as a model file in TensorFlow, PyTorch, or ONNX format. The model architecture is analyzed, and information for each layer is extracted, such as layer type, input / output dimensions, weights, and biases, and stored as preset layer format data.

[0043] Step S220: The structured data corresponding to at least two first network layers in the model are fused to merge the computational operations of at least two first network layers.

[0044] At least two of the first network layers include associated sequential computational operations.

[0045] Specifically, by merging continuous and functionally related network layer operations, redundant computation and data transfer are reduced, thereby improving model execution efficiency.

[0046] For example, continuous operators that can be fused in the identification model, such as Convolution-Batch Normalization-ReLU (Conv-BN-ReLU) or Fully Connected-Activation (FC-Activation), are merged into a single operator based on the principle of mathematical equivalence transformation, and the weights and bias parameters are recalculated. Operator fusion combines multiple continuous, functionally related operations into one operation to reduce the number of data read / write operations during computation and improve hardware utilization.

[0047] Step S230: The structured data of multiple network layers of the model after fusion processing is converted into intermediate representation data in the target format and filled into a preset two-dimensional array to obtain the two-dimensional array representation of the model.

[0048] In this model, the structured data of each network layer corresponds to a set of vector data in a two-dimensional array, and the model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

[0049] Specifically, the optimized model structure is transformed into an intermediate representation in a two-dimensional array format, which not only facilitates reading by the hardware platform but also supports cross-platform model execution.

[0050] For example, a two-dimensional array is constructed based on layer format data, where each row represents a network layer and includes layer type, parameter information, input and output information, etc. It is ensured that the format and arrangement of the data in the array are consistent so that parsers on different hardware platforms can understand and execute it.

[0051] Among them, a two-dimensional array is a data structure that can store detailed information about the model, making it easier for hardware actuators to quickly locate and execute specific layer operations.

[0052] In this embodiment, by defining a universal intermediate representation format, platform barriers are eliminated, allowing the model to migrate freely across different hardware platforms. This reduces development and deployment costs and enhances model portability and ecosystem uniformity. Using structured data to describe each layer of the model simplifies the model parsing process and provides standardized input for subsequent operator fusion and instruction generation, enhancing model readability and operability. Operator fusion effectively reduces memory access and computational unit overhead, improving the parallelism and throughput of model execution, thereby reducing model inference latency and improving overall performance. Two-dimensional array representation simplifies model loading and execution, reducing the time complexity of model parsing. Simultaneously, the unified format makes model deployment on different hardware platforms faster and more efficient, further improving model execution efficiency. Through standardized intermediate representation, operator-level optimization, and efficient data structure transformation, the model achieves fast and accurate compatibility and operation on different hardware platforms, improving the speed and energy efficiency of model inference in edge computing and cloud computing scenarios.

[0053] In one embodiment, such as Figure 3 As shown, step S210 involves parsing the model to be compiled, representing the multiple network layers of the model as structured data in a preset format. This includes steps S300-S340:

[0054] Step S300: The generation framework based on the model to be compiled reads the original description data of the model.

[0055] The original descriptive data includes the model's structural parameters.

[0056] Specifically, the basic descriptive information of the model, i.e., the structural parameters, is read from the deep learning model generation framework (such as TensorFlow, PyTorch, or ONNX).

[0057] For example, design a model parser that can read model files and parse out the model's structural information, such as the number of network layers, the type of each layer, and its parameters.

[0058] Model generation frameworks are software libraries or environments used to build, train, and test deep learning models, such as TensorFlow and PyTorch. The raw description data is the initial description of the model, typically contained in the model file generated by the framework, such as parameters like the weights, biases, and layer types of the neural network. Taking a linear layer as an example, the model data includes the following: current layer number, layer name, layer type, number of inputs, number of outputs, whether there are weights, whether there are biases, weight dimensions, bias dimensions, all input dimensions, all output dimensions, and data type.

[0059] Step S310: Perform semantic matching on the structural parameters of the model to determine the semantic information corresponding to each network layer included in the model.

[0060] Semantic information is used to indicate the computational operations performed by the corresponding network layer.

[0061] Specifically, after reading the model structure parameters, it is necessary to further analyze the meaning of each layer, that is, the type of operation it performs, in order to perform more accurate optimization and transformation.

[0062] For example, the function of each layer can be identified based on the read structural parameters. For instance, specific parameter patterns can be matched with known operator types, such as identifying convolutional layers, linear layers, activation function layers, etc.

[0063] Semantic information describes the operational properties and functions of each layer in a deep learning model, such as convolution, fully connected layers, and activation. Operational operations are the specific mathematical operations performed by the layers in a deep learning model, such as matrix multiplication, convolution, and dot product.

[0064] Step S320: Identify the association patterns between different network layers of the model based on the semantic information corresponding to each network layer.

[0065] Among them, the correlation pattern is used to indicate the correlation or continuity of operations between different network layers.

[0066] Specifically, it involves understanding the relationships between network layers in the model and identifying which layers are continuous and can be fused, such as identifying the batch normalization and activation function layers that follow convolutional layers.

[0067] For example, graph traversal or pattern recognition techniques are used to construct the computational graph of the model based on semantic information, and to identify continuous layers that can be fused, such as the convolution-batch normalization-activation (Conv-BN-ReLU) sequence.

[0068] Among them, the correlation pattern is the connection and dependency relationship between different network layers in a deep learning model, which can reveal which layer operations are continuous and can be optimized and fused.

[0069] Step S330: Extract the parameter information corresponding to each network layer in the model.

[0070] Specifically, extract the specific parameter information of each network layer from the model, such as weights, biases, input and output dimensions, etc.

[0071] For example, based on the read raw description data, a parameter extraction algorithm is used to obtain the parameters of each layer from the computation graph, and these parameters are organized and stored for later use.

[0072] Among them, parameter information refers to the specific values ​​or attributes of network layers in a deep learning model, including weights, biases, activation function types, input and output shapes, etc.

[0073] Step S340: Convert the parameter information, semantic information, and association patterns between each network layer into structured data in a preset format for representation.

[0074] Specifically, all key information of the model is integrated and converted into a unified, structured data format, such as layer format data, so that it can be correctly interpreted and executed on different hardware platforms.

[0075] In this embodiment, the framework-based model analysis ensures information integrity, avoids errors and omissions from manual input, and provides accurate foundational data for subsequent steps. Clearly defining the operational semantics of each layer facilitates the identification and optimization of continuously related operations in subsequent steps, improving the targeting and effectiveness of model conversion. Identifying correlation patterns provides a basis for operator fusion, significantly reducing redundant computations and improving the model's execution efficiency on Field-Programmable Gate Arrays (FPGAs). Extracting parameter information allows subsequent operator fusion and instruction generation steps to directly access the required numerical attributes, accelerating the model conversion process. Employing a unified structured data representation greatly improves the model's portability and compatibility across different platforms, while simplifying the hardware platform analysis process and reducing model loading and execution latency.

[0076] In one embodiment, such as Figure 4 As shown, step S220 involves fusing the structured data corresponding to at least two first network layers in the model to merge the computational operations of at least two first network layers. This includes steps S400-S440:

[0077] Step S400: Determine at least two first network layers that can be merged based on the association pattern between the various network layers.

[0078] Specifically, we analyze the connection relationships between network layers in a deep learning model and identify layer sequences that can be fused and optimized.

[0079] For example, graph analysis algorithms are used to traverse the model structure to identify continuous and functionally integrated network layers, such as the Convolutional layer-BatchNormalization-Rectified Linear Unit (Conv-BN-ReLU) combination.

[0080] Step S410: Determine the fusion formula of at least two first network layer operations according to the principle of mathematical equivalence.

[0081] Specifically, based on mathematical principles, a formula for fusing operations of adjacent network layers is designed to ensure that the fused operations are equivalent to the output of the original model.

[0082] For example, by utilizing the principle of mathematical equivalence transformation, such as the combination of matrix multiplication and linear transformation, the operation formula of the fusion operator and the updated parameter calculation rules are derived to ensure that the operation logic after fusion is consistent with the original model.

[0083] The principle of mathematical equivalence is a transformation rule in mathematical operations that ensures the results of operations before and after a transformation are equal. It is applicable to operator fusion in deep learning models. The fusion formula is a mathematical expression used to transform the operations of two or more consecutive layers into a more efficient operation.

[0084] Step S420: Determine the operators corresponding to at least two first network layers based on the semantic information corresponding to at least two first network layers.

[0085] Specifically, specific operator types are identified from the fused network layers to prepare for subsequent parameter extraction and instruction generation.

[0086] For example, based on the determined fusion formula, the operator type of the fused network layer is identified, such as the fused convolution-batch normalization-activation function layer, which can be regarded as a composite operator.

[0087] Step S430: Extract the operator parameters of the corresponding operators based on the structured data corresponding to at least two first network layers.

[0088] The operator parameters include at least one of the following: weight, scaling factor, bias factor, and tensor dimension.

[0089] Specifically, necessary parameters, such as weights, scaling factors, and biases, are extracted from the structured data of the fused network layers to provide numerical information for the efficient execution of operators.

[0090] For example, the structured data of the fused network layers is parsed to extract parameter information related to the operators, which will be used for operator initialization and execution.

[0091] Step S440: The operators corresponding to at least two first network layers are fused using a fusion formula, and the operator parameters are substituted to obtain the objective function.

[0092] The objective function is equivalent to a combined representation of the operations of at least two first network layers.

[0093] Specifically, the fused operators and extracted parameters are combined to generate a target function representing the operation of the fused network layer, which can be executed directly on hardware such as FPGA.

[0094] For example, using a fusion formula, combined with the extracted operator parameters, a final target function is generated through programming tools or a custom function. This function encapsulates all the computational logic and parameters of the fused operator.

[0095] In this embodiment, layer fusion significantly reduces unnecessary intermediate computation steps, decreases memory read / write operations, and improves computational efficiency, especially on resource-constrained FPGA platforms. The fusion formula, designed using mathematical equivalence principles, ensures the model's functionality and accuracy while optimizing the computational flow, reducing redundant operations, and improving execution efficiency. Clearly defining operator types facilitates further parameter extraction and optimization, as well as the generation of efficient instruction sets for hardware such as FPGAs. Parameter extraction ensures that the fused operators can execute based on accurate numerical information, avoiding error accumulation during execution and maintaining model accuracy. The generation of the objective function enables efficient hardware deployment of the model, reducing the storage of intermediate results and the number of data transfers, optimizing memory usage, and lowering computational latency. Especially for FPGAs, this significantly improves the utilization of computing resources and the speed of model inference.

[0096] In one embodiment, such as Figure 5 As shown, the method further includes steps S500-S520:

[0097] Step S500: Determine the type of each network layer of the model based on the semantic information corresponding to each network layer.

[0098] Specifically, the semantic function of each layer in the model is identified to determine which type of network layer it belongs to, such as a convolutional layer, a batch normalization layer, or an activation function layer.

[0099] Semantic information describes the functions of network layers, including the operator types, input and output structures, etc.

[0100] Step S510: Determine whether there are multiple consecutive network layers that meet the target combination, based on the type corresponding to each network layer.

[0101] The target combination includes convolutional layers, batch normalization layers, and activation function layers. The target combination is a pre-defined combination of network layers that can be executed or fused together more efficiently, such as a combination of convolutional-batch normalization-activation function layers.

[0102] Specifically, based on the type of network layers in the model, it is determined whether there exists a set of consecutive layers whose combination conforms to a predefined optimization objective, such as the "convolution-batch normalization-activation function" sequence.

[0103] Step S520: If it is determined that there is a continuous set of convolutional layers, batch normalization layers, and activation function layers, the continuous set of convolutional layers, batch normalization layers, and activation function layers are defined as the first network layer.

[0104] Specifically, once a sequence of consecutive layers that matches the target combination is identified, these layers are defined as specific “first network layers” for operator fusion optimization.

[0105] It should be noted that, without operator fusion, the data flow is typically as follows:

[0106] Convolutional layer (Conv layer): The input feature map is convolved with the weights in the convolutional layer to produce a convolutional output.

[0107] Batch Normalization (BN) Layer: The convolutional output is used as input. The mean and variance of each output channel are calculated, and then normalized using a pre-trained scaling factor (gamma) and bias factor (beta). This process produces a new intermediate result, the normalized output.

[0108] Activation layer (ReLU layer): The normalized output is then passed to the activation layer, such as ReLU, for nonlinear transformation.

[0109] In this process, whenever a layer completes a calculation and outputs the result to the next layer, the intermediate results (such as the output of the BN layer and the input of the ReLU layer) need to be written back to memory (such as Double Data Rate (DDR) memory) and read again when used by the next layer, which results in at least two memory read and write operations.

[0110] The fused operator process treats these three operations as a single, continuous operation: ReLU(BN(Conv(input))) is computed directly after fusion, without writing intermediate results back to memory. This means that convolution, batch normalization, and activation are performed consecutively using the connections between computational units (e.g., through internal RAM or registers). This reduces intermediate result buffer storage: FPGA resources are typically very valuable, especially on low- to medium-scale FPGA chips. Without fusion, the output of each layer needs to be stored in an intermediate buffer until it is used by the next layer. This means that if the model contains multiple consecutive layers, each layer needs its own buffer to store intermediate results, which quickly consumes the FPGA's limited resources. After operator fusion, since there is no need to save and read intermediate results, the buffers used to store BN outputs and ReLU inputs can be eliminated, reducing the number of buffers by two. This is because the fused computation directly uses the output of the previous layer as the input of the current layer, without requiring additional intermediate storage. In a resource-constrained FPGA environment, this is equivalent to reducing the original two buffers to one, which can theoretically reduce the amount of random-access memory (RAM) used by up to 50%.

[0111] In this embodiment, by clearly defining the type of each layer, a foundation is provided for subsequent identification of layer combinations that can be fused, simplifying the model optimization process and increasing the feasibility of operator fusion. Identifying layer sequences that can be fused and optimized helps reduce redundant operations and memory accesses during model execution, improving hardware execution efficiency. Defining consecutive layers as the first network layer provides a clear starting point for operator fusion, helping to improve model execution efficiency and reduce memory access and computational latency.

[0112] In one embodiment, such as Figure 6 As shown, step S220 involves fusing the structured data corresponding to at least two first network layers in the model to merge the computational operations of at least two first network layers. This includes steps S600-S620:

[0113] Step S600: Determine at least two consecutive linear layers in the model based on the semantic information corresponding to each network layer.

[0114] Among them, at least two consecutive linear layers are used as at least two first network layers.

[0115] Specifically, at least two consecutive linear layers are identified from the deep learning model. These layers can be fully connected layers or other layers with linear transformation properties.

[0116] Linear layers are layers that perform linear transformations in deep learning models. They typically include fully connected layers, linear regression layers, etc., and their operations can be represented by matrix multiplication.

[0117] Step S610: Extract matrix multiplication operators for at least two linear layers based on the structured data of at least two linear layers.

[0118] Specifically, matrix multiplication operators for the core operations of linear transformations are extracted from the identified continuous linear layers.

[0119] Step S620: The matrix multiplication operators of at least two linear layers and the activation functions between at least two linear layers are fused so that the output of the previous matrix multiplication operator is used as the input data of the next matrix multiplication operator.

[0120] Specifically, the matrix multiplication operator and activation function operator between consecutive linear layers are integrated, and two linear layers are regarded as a single integrated operator. The output of the previous operator is directly used as the input of the next operator, reducing data transfer.

[0121] For example, by employing the principle of mathematical equivalence transformation, the weight matrix and activation function parameters of consecutive linear layers are fused to generate an equivalent synthesized operator. For instance, the output of linear layer 1 is directly fed into the activation function, and the output of the activation function is then used as the input to linear layer 2, avoiding the storage and handling of intermediate data. This fusion operation eliminates the need for a buffer to store intermediate ReLU results, as the ReLU output is directly used as the input for subsequent calculations, requiring no additional storage. During FPGA computation, data is preferentially moved from DDR memory to the faster-access Synchronous Dynamic Random-Access Memory (SDRAM) to reduce the latency of external memory access.

[0122] In this embodiment, continuous linear layers are explicitly identified, providing the necessary information foundation for subsequent matrix multiplication operator extraction and fusion optimization. This helps reduce redundant calculations and improve model execution efficiency. Extracting matrix multiplication operator parameters prepares for operator fusion. By separating the operator parameters, more precise optimization can be performed during the fusion process, reducing memory access and computational latency. Operator fusion reduces memory read / write operations and data movement, lowers the need for intermediate result storage, effectively saves memory resources in the FPGA, and improves the continuity and efficiency of the computational flow.

[0123] In one embodiment, such as Figure 7 As shown, step S230 involves converting the structured data of multiple network layers of the fused model into intermediate representation data in the target format and filling it into a preset two-dimensional array. This includes steps S700-S720:

[0124] Step S700: Create a preset two-dimensional array.

[0125] The length of a set of vector data in the two-dimensional array is determined based on the number of parameters of the structured data corresponding to the multiple network layers of the model.

[0126] Specifically, a two-dimensional array is constructed, with the length of each row of data set based on the number of parameters and information in each network layer of the deep learning model, ensuring that it can accommodate all the necessary information of each layer.

[0127] For example, to analyze the maximum number of parameters in a network layer of the model, a two-dimensional array is defined, in which the number of rows is the same as the number of network layers, and the length of each row (i.e. the number of columns) is sufficient to store information such as the type of the layer, the number of inputs and outputs, weights, and biases.

[0128] Step S710: Fill the structured data of multiple network layers into a two-dimensional array.

[0129] The structured data of different network layers are located in different rows of a two-dimensional array.

[0130] Specifically, the structured data of all network layers in the model are filled into different rows of a two-dimensional array in a certain order, and the model information is stored in units of layers.

[0131] For example, the network layers of the model are traversed, and the parameter information of each layer is inserted into the corresponding row of the two-dimensional array according to the order of recognition, so as to ensure that each row of data completely represents a network layer.

[0132] Step S720: Create corresponding row index and column index for each row and column of the two-dimensional array.

[0133] The row index of the two-dimensional array is used to indicate the layer number of the network layer, and the column index of the two-dimensional array is used to indicate the parameter information of the network layer. The parameter information includes at least one of the following: the type of the network layer, the number of input data to the network layer, the number of output data to the network layer, weight information, and bias information.

[0134] Specifically, indices are added to the rows and columns of the two-dimensional array. The row index corresponds to the network layer number, and the column index indicates the parameter type of the layer, such as type, number of inputs and outputs, weight information, etc.

[0135] For example, define a row index array to record the layer number corresponding to each row. Define column indices to identify the specific parameter information stored in each column, such as type information in column 0, input quantity information in column 1, and so on. Taking a linear layer as an example, each parameter is defined as follows: [Parameter 0: layer type, Parameter 1: number of inputs, Parameter 2: number of outputs, Parameter 3: input ID, Parameter 4: output ID, Parameter 5: whether there is a bias (yes: 1, no: 0), Parameter 6: input features; Parameter 7: output features, Parameter 8: bias dimension parameter, Parameter 9-10: weight dimension parameter, Parameter 11: input ID, Parameter 12: number of dimensions corresponding to the input ID, Parameter 13-17: dimension parameter corresponding to the input ID, Parameter 18: output ID, Parameter 19: number of dimensions corresponding to the output ID, Parameter 20-24: dimension parameter corresponding to the output ID, End: current layer number].

[0136] In this embodiment, by creating a pre-defined two-dimensional array, the parameters and information of each network layer of the model can be accurately stored, providing a unified data representation for subsequent model conversion and deployment, and simplifying the data processing and access process. Storing the structured data of different network layers in different rows of the two-dimensional array clearly displays the model's structure and parameter layout, facilitating subsequent operations and optimizations for specific layers, and also simplifying hardware reading and execution of instructions. The addition of row and column indices makes data access more intuitive and efficient; the hardware can directly and quickly locate the parameters of a specific network layer based on the layer number, reducing parsing and location time and improving the efficiency of model deployment and execution.

[0137] In one embodiment, such as Figure 8 As shown, the method further includes steps S800-S830:

[0138] Step S800: Create the initial binary file and the initial header list.

[0139] The binary file stores the weight and bias information of the network layers, while the header list stores the header information of the network layers.

[0140] Specifically, a binary file is generated to store the weights and biases of each network layer in the model, while a header list is created to store the metadata information of each layer.

[0141] For example, using a binary file format such as .bin, initialize an empty file to store the weight and bias data. Simultaneously, create a list to store the header information for each layer, including the layer number, weight size, bias size, number of bytes per weight, and weight offset.

[0142] The header list stores metadata for each layer, including the layer's identifier, weights, biases, and their position in the binary file. Each header is 20 bytes long and contains: layer_id (layer number), weight_size (weight size), bias_size (bias size), data_size (data size, i.e., the number of bytes occupied by each weight), and offset (the starting offset position of the weight).

[0143] Step S810: Extract the weight information and bias information of each network layer from the structured data of each network layer of the model.

[0144] Specifically, the weight matrix and bias vector data for each layer are extracted from the structured data of the model, and then stored in a binary file.

[0145] Step S820: Store the weight information and bias information of each network layer in a binary file according to the order of each network layer.

[0146] Specifically, the extracted weights and biases are stored in a binary file in the order of the network layers to ensure that the order of the data is consistent with the model structure.

[0147] For example, in the binary file, the weight data and bias data of each layer are written sequentially. For each layer, the weight data is written first, followed by the bias data, and the starting position of the data of each layer in the file is recorded.

[0148] Step S830: Based on the amount and storage location of the weight data and bias data of each network layer, create the corresponding header structure data for each network layer, and store the header structure data in the header list.

[0149] The header structure data includes the layer number, weight size, offset size, number of bytes occupied by each weight, and offset position of the weight.

[0150] Specifically, a header structure is created for the weight and bias data of each layer, containing the size of the data for that layer, the number of bytes for each parameter, and the offset position in the binary file. All header data is then stored in a header list.

[0151] For example, the structured data of the model is traversed, and for each layer, a header structure is created, which is filled with the size, data type and starting position information of the weights and biases in the binary file. This header structure is then added to the header list.

[0152] In this embodiment, storing weight and bias data in a binary file reduces file size and improves loading speed. The header list facilitates quick location of weight and bias data during execution, eliminating the need for byte-by-byte file parsing and improving model loading and execution efficiency. By directly extracting weight and bias data from the model, the need for complex parsing processes during execution is avoided, ensuring direct data availability and execution efficiency. The ordered storage of weight and bias data allows for rapid location of data at specific layers during model execution, reducing unnecessary file read operations and optimizing model loading time. The creation and storage of the header structure provides a fast access mechanism for weight and bias data, enabling hardware to directly jump to the data based on the header information during model execution, significantly reducing data retrieval time.

[0153] In one embodiment, such as Figure 9 As shown, the method further includes steps S900-S920:

[0154] Step S900: Obtain the hardware parameters of the target hardware platform.

[0155] The hardware parameters include the number of interfaces and the address space corresponding to each interface.

[0156] Specifically, before deploying a deep learning model, it is necessary to collect detailed configuration information about the target hardware platform, especially the number of Direct Memory Access (DMA) interfaces and their associated address space, in order to facilitate the efficient transmission and storage of model data.

[0157] For example, the number of interfaces and the address space information of each interface can be obtained by reading the FPGA development board's specification document, using hardware queries, or directly reading attributes from the hardware.

[0158] Step S910: Allocate corresponding interfaces and address spaces to each network layer represented by the two-dimensional array of the model according to the hardware parameters.

[0159] Specifically, based on the acquired hardware parameters, a strategy is designed to allocate appropriate DMA interfaces and DDR address spaces to each network layer of the model, ensuring optimal data layout on the hardware.

[0160] Step S920: Based on the interface and address space corresponding to each network layer, write the data of each network layer, represented by a two-dimensional array, into the corresponding address space.

[0161] Specifically, after determining the DMA interface and DDR address for each layer, the weights, biases, and other relevant data for each layer in the model are written to the specified memory location in the hardware.

[0162] In this embodiment, accurately obtaining hardware parameters helps to make more reasonable and efficient resource allocation decisions in subsequent steps, avoiding data transmission bottlenecks and delays caused by improper resource allocation. Reasonable allocation of interfaces and address space can reduce data transmission latency, optimize data storage and access on the hardware, and ensure efficient model operation and data consistency on the target hardware platform. Precisely writing data into the pre-allocated address space not only accelerates the model initialization process but also optimizes hardware memory management, improving model execution efficiency and performance.

[0163] In one embodiment, such as Figure 10 As shown, step S910 allocates corresponding interfaces and address spaces to each network layer represented by the two-dimensional array of the model according to the hardware parameters. This includes steps S1000-S1030:

[0164] Step S1000: Create a first relation table and a second relation table based on the hardware parameters.

[0165] The first relation table indicates the correspondence between the identifier of the memory block of the target hardware platform when it is used as an output memory block and the layer number of the network layer of the model. The second relation table indicates the correspondence between the identifier of the memory block of the target hardware platform when it is used as an input memory block and the layer number of the network layer of the model.

[0166] Specifically, two tables are created to record the correspondence between the target hardware platform's memory blocks as outputs and inputs and the model's network layers, respectively, to facilitate subsequent data transmission and processing.

[0167] Step S1010: Traverse the two-dimensional array of the model and allocate corresponding input memory blocks and output memory blocks for each network layer of the model.

[0168] Specifically, the two-dimensional array representation of the model is traversed, and input and output memory blocks are allocated for each network layer to ensure that input data can be read correctly and output data can be written and transmitted correctly.

[0169] For example, a loop structure is used to traverse the two-dimensional array of the model. For each network layer, input and output memory blocks are dynamically allocated based on its data dependencies and hardware parameters. During allocation, optimization of data size, type, and transmission path can be considered to ensure efficient data layout in hardware memory.

[0170] Step S1020: Update the first relation table and the second relation table according to the identifiers of the input memory blocks and the output memory blocks corresponding to each network layer.

[0171] Specifically, the created relational table is updated to ensure that the input and output memory block identifiers of each network layer correspond to the correct layer number, thus preparing for data transmission.

[0172] Step S1030: Allocate corresponding interfaces and address spaces for memory blocks corresponding to each network layer according to the first relation table and the second relation table.

[0173] Specifically, based on the mapping relationships stored in the relation table, DMA interfaces and DDR memory addresses are allocated to the input and output memory blocks of each network layer to ensure efficient data transmission and storage.

[0174] For example, based on the memory block identifiers recorded in the first and second relation tables, and in conjunction with hardware parameters (such as the number of interfaces and the corresponding address space), one or more DMA interfaces and their corresponding DDR memory addresses are allocated to each memory block. An algorithm can be designed to prioritize allocating layers with similar data volumes or continuous data flows to the same interface to reduce the overhead of interface switching.

[0175] In this embodiment, by establishing a clear correspondence, data flow management is simplified, enabling rapid location of input and output data when executing the model on hardware, reducing data transmission errors and latency. Proper allocation of memory blocks reduces the data transmission distance on the hardware platform, lowers bandwidth pressure, improves data access speed, and thus optimizes model execution efficiency. The updated table provides accurate correspondences between memory block identifiers and network layers, facilitating direct access to required memory blocks during computational tasks, reducing data location time, and improving the real-time performance of model inference. Proper allocation of interfaces and DDR memory addresses reduces data transmission latency, optimizes hardware resource utilization, and improves model execution efficiency and speed.

[0176] In one embodiment, such as Figure 11 As shown, step S1030 involves allocating corresponding interfaces and address spaces to the memory blocks corresponding to each network layer based on the first and second relationship tables. This includes steps S1100-S1130:

[0177] Step S1100: Read the first identifier of each output memory block from the first relation table.

[0178] Specifically, a unique identifier for each output memory block is obtained from the first relationship table that records the correspondence between output memory blocks and network layers, in order to prepare for subsequent resource allocation and management.

[0179] Step S1110: Determine the number of input data that exist in the memory block corresponding to the first identifier when it is used as an input memory block, based on the second relation table.

[0180] Specifically, by querying the second relation table, the amount of input data that the output memory block (now used as an input memory block) corresponding to the first identifier needs to carry is determined, providing a basis for reasonable address space allocation.

[0181] For example, the second relation table is accessed to find the entry for the memory block corresponding to the first identifier, and the number of all input network layers listed in the entry is counted, which will reflect the amount of input data that needs to be stored in the memory block.

[0182] Step S1120: Allocate address space for the memory block corresponding to the first identifier according to the number of input data.

[0183] Specifically, based on the amount of input data, sufficient address space is allocated in the DDR memory of the hardware platform for the corresponding memory block to store and process the input data.

[0184] And in step S1130, if the memory block corresponding to the first identifier is used as an input memory block and no interface is allocated, then an interface is allocated for the memory block corresponding to the first identifier.

[0185] Specifically, check whether the memory block with allocated address space has been allocated a DMA interface. If not, allocate one to ensure that data can be quickly transferred to the correct memory location.

[0186] In this embodiment, accurately obtaining the identifier of the output memory block helps in evaluating the data volume and interface requirements when it is used as an input memory block in subsequent steps, thereby optimizing data management and resource allocation. Clearly defining the amount of input data helps in accurately allocating memory space, avoiding insufficient or wasted space, and ensuring efficient data transmission and stable model execution. Reasonable allocation of DDR address space ensures the continuity and speed of data transmission and processing, avoids performance bottlenecks caused by insufficient memory, and improves the utilization efficiency of hardware resources. Allocating a DMA interface to the input memory block reduces data transmission latency, increases data transmission bandwidth, and ensures the real-time performance and execution efficiency of the model.

[0187] In one embodiment, such as Figure 12 As shown, the method further includes steps S1200-S1280:

[0188] Step S1200: Extract the model files stored in the address space of the target hardware platform.

[0189] Specifically, the file used to deploy the deep learning model is read from the memory of the target hardware platform. This file contains complete information about the model and its parameters. The model file contains information about the deep learning model's structure and parameters.

[0190] Step S1210: Extract the computational parameters of each network layer from the model file in the logical order of calculation.

[0191] Specifically, the computational parameters for each layer, such as weights and biases, are parsed from the model file and arranged in the logical order of model calculation.

[0192] For example, information for each layer is parsed based on the format of the model file; this can typically be achieved by custom parsing or using an existing parsing library. The parsed parameters are then sorted according to their position in the network or the order of execution for easier subsequent operations.

[0193] Step S1220: Based on the size of the computing module of the target hardware platform, divide the computational parameters of the current network layer into multiple sub-matrices.

[0194] Specifically, considering the size limitations of the computing modules in the target hardware (such as FPGA), the computational parameters of the network layer are divided into several small matrices that can be effectively processed by the hardware to adapt to the hardware's computing power and resource utilization efficiency.

[0195] For example, a parametric partitioning algorithm is designed that, based on the size of the hardware's computation module, divides a large parametric matrix into a series of smaller matrices. This may involve matrix shape analysis, data type conversion, and storage format optimization to ensure that the partitioned submatrices can be processed efficiently by the hardware.

[0196] Step S1230: Determine the addresses and data amounts of the multiple sub-matrices in the memory block.

[0197] Specifically, the addresses and required storage space in hardware memory are determined for the segmented submatrices in order to perform data transfer and combination operations.

[0198] For example, using hardware parameters (such as DDR memory layout, interface information, etc.) and the segmented submatrix information, the exact location of each submatrix in memory and the number of bytes required for storage are calculated. This typically involves using a hash table or index structure to store this information for quick location.

[0199] Step S1240: Move the multiple sub-matrices to the target address space according to their addresses and data amounts in the memory block.

[0200] Specifically, the submatrix is ​​moved from its original position to the target address space on the hardware for computation, thus preloading and preparing the data.

[0201] For example, DMA (Direct Memory Access) or other hardware acceleration mechanisms can be used to move data from DDR memory to a cache or local RAM that is directly accessible to the computing module, based on the address and data volume of the submatrix.

[0202] Step S1250: Combine multiple sub-matrices in the target address space to generate the instruction corresponding to the current network layer.

[0203] Specifically, computation instructions are executed on the submatrix that has been moved to the target address space to generate hardware execution instructions adapted to the current network layer.

[0204] For example, based on the position of the submatrix in the target address space, hardware execution instructions, such as Verilog code or specific FPGA execution instructions, are written or generated to perform matrix multiplication, addition, and other operations to generate the hardware instructions for the current network layer.

[0205] Step S1260: Determine whether all network layers of the model have generated corresponding instructions. If any network layer has not generated corresponding instructions, proceed to step S1270. If all network layers have generated corresponding instructions, proceed to step S1280.

[0206] Step S1270: Set the next network layer of the model as the current network layer. Then return to step S1220.

[0207] Specifically, repeat the above steps until all instructions for the model network layers are generated, forming a complete set of model execution instructions.

[0208] For example, using a loop structure, the current network layer is set as the next network layer, and the process of steps S1220 to S1270 is iterated until the last network layer of the model.

[0209] Step S1280: Store the instructions corresponding to all network layers of the model in the instruction address space of the target hardware platform so that the target hardware platform can call the stored instructions at runtime.

[0210] Specifically, the generated model execution instruction set is stored in the hardware's instruction address space so that the hardware can directly call these instructions to perform calculations when the model is running.

[0211] In this embodiment, the model file is read quickly and accurately, providing the necessary data input for model parameter parsing and subsequent instruction generation, reducing model loading time. Arranging the computational parameters in logical order helps ensure the coherence and efficiency of subsequent instruction generation, guarantees the correct data reading order during model execution, and reduces unnecessary data read / write operations. Dividing parameters into sub-matrices suitable for hardware processing fully utilizes computing resources, avoiding hardware resource waste or performance degradation due to excessively large parameters, optimizing hardware resource utilization efficiency and model execution parallelism. Accurately determining the address and data volume of the sub-matrices enables efficient data transfer and storage, reducing data access latency and improving model execution speed. Moving the sub-matrices' data to the target address space reduces data processing latency, optimizes the model execution flow, and improves hardware computational efficiency and data throughput. The generated hardware instructions are highly targeted, fully utilizing hardware resources, improving computational performance, and accelerating model inference speed. Generating a complete instruction set enables automated and efficient model execution on hardware, reducing the need for manual intervention and optimization, and improving the overall model execution efficiency and resource utilization on hardware. By storing instructions in the instruction address space, the instruction invocation process is optimized, the instruction parsing and waiting time are reduced, and the model can be started up quickly and executed efficiently.

[0212] In one embodiment, the method further includes:

[0213] Operator gene library initialization.

[0214] Specifically, the initialization phase of the self-evolving operator library involves building a "gene library" containing basic operators. These basic operators account for approximately 30% of the total number of operators and cover the core operations in deep learning models, such as convolution, matrix multiplication, and normalization. The remaining operators are stored in the flash memory of the hardware platform in the form of more abstract intermediate bit-fragments (IBFs). IBFs are reconfigurable low-level logical units that are not bound to specific functions but retain the key logical architecture of the operators.

[0215] The collaborative work of operator gene compilers.

[0216] Specifically, upon the hardware platform's first power-on or before model execution, the built-in "operator gene compiler" begins operation. It parses the intermediate representation (IR) of the model, identifying all necessary operators. For operators not directly pre-configured in the FPGA, the compiler performs on-the-fly "genetic recombination" based on the IBF library, i.e., rewiring and retiming according to Pareto frontier methods to generate new operator bit segments adapted to the current model within 5ms. This process considers an optimal balance of resources, power consumption, and timing, ensuring that the generated operators are both efficient and implementable within limited FPGA resources.

[0217] Cloud-based learning and gene fragment weight updates.

[0218] Specifically, during hardware platform operation, telemetry data on model execution is collected, including operator resource consumption, power consumption, and latency. This data is transmitted back to the cloud via a secure connection. The cloud-based "edge-end collaborative learning engine" utilizes this data, employing a hybrid learning strategy combining reinforcement learning (RL) and graph neural networks (GNN), to continuously update the "gene fragment weight table." This table guides the combination order and method of IBFs to adapt to a wider range of operator needs and optimization objectives. The cloud-based learning model periodically distributes the updated gene fragment weight table to the terminal hardware platform, enabling the operator library to self-evolve and self-update without manual intervention or firmware upgrades, significantly improving the adaptability and scalability of the operator library.

[0219] In this embodiment, the abstract design and efficient storage of Intermediate Bit Fragments (IBF) ensure that the operator library can be lightweight and stored in memory while retaining sufficient logical flexibility to adapt to model requirements. Through a self-evolution mechanism, the operator library can automatically adapt to the needs of new models without manual firmware updates, simplifying the model deployment process. Real-time operator generation fully utilizes the dynamic reconfigurability of the FPGA, achieving efficient resource utilization and minimizing power consumption. A fast gene recombination algorithm ensures rapid operator generation, reducing model loading and inference latency and improving overall execution efficiency.

[0220] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0221] Embodiments of this application also provide a model compilation apparatus. Figure 13 This is a structural block diagram of a compilation apparatus for a model according to an embodiment of this application. The apparatus includes:

[0222] The acquisition module 1301 is used to acquire the target format of intermediate representation data that is compatible with multiple different target hardware platforms.

[0223] The parsing module 1302 is used to parse the model to be compiled, so as to represent the multiple network layers included in the model as structured data in a preset format, wherein the structured data includes the parameter information of the corresponding network layers.

[0224] The fusion module 1303 is used to fuse the structured data corresponding to at least two first network layers in the model to merge the operation of at least two first network layers, wherein the at least two first network layers include related continuous operation.

[0225] The compilation module 1304 is used to convert the structured data of multiple network layers of the fused model into intermediate representation data in the target format and fill it into a preset two-dimensional array to obtain a two-dimensional array representation of the model. The structured data of each network layer corresponds to a set of vector data in the two-dimensional array. The model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

[0226] In an exemplary embodiment, the apparatus is further configured to: read the original description data of the model based on the generation framework of the model to be compiled, wherein the original description data includes the structural parameters of the model; perform semantic matching on the structural parameters of the model to determine the semantic information corresponding to each network layer included in the model, wherein the semantic information is used to indicate the operation performed by the corresponding network layer; identify the association pattern between the network layers of the model based on the semantic information corresponding to each network layer, wherein the association pattern is used to indicate the correlation or continuity of the operation between different network layers; extract the parameter information corresponding to each network layer in the model respectively; and convert the parameter information corresponding to each network layer, the semantic information corresponding to each network layer, and the association pattern between each network layer into structured data in a preset format for representation.

[0227] In one exemplary embodiment, the apparatus is further configured to: determine at least two first network layers that can be merged based on the association patterns between the various network layers; determine a fusion formula for the operations of the at least two first network layers according to the principle of mathematical equivalence; determine operators corresponding to the at least two first network layers respectively based on the semantic information corresponding to the at least two first network layers; extract operator parameters of the corresponding operators based on the structured data corresponding to the at least two first network layers, wherein the operator parameters include at least one of the following: weights, scaling factors, bias factors, and tensor dimensions; fuse the operators corresponding to the at least two first network layers respectively using the fusion formula, and substitute the operator parameters to obtain an objective function, wherein the objective function is equivalent to the merged representation of the operations of the at least two first network layers.

[0228] In an exemplary embodiment, the apparatus is further configured to: determine the type corresponding to each network layer of the model based on the semantic information corresponding to each network layer; determine whether there exist multiple consecutive network layers that conform to a target combination based on the type corresponding to each network layer, wherein the target combination includes convolutional layers, batch normalization layers, and activation function layers; and, if it is determined that there exists a consecutive set of convolutional layers, batch normalization layers, and activation function layers, define the consecutive set of convolutional layers, batch normalization layers, and activation function layers as first network layers.

[0229] In one exemplary embodiment, the apparatus is further configured to: determine at least two consecutive linear layers in the model based on semantic information corresponding to each network layer, wherein the at least two consecutive linear layers are designated as at least two first network layers; extract matrix multiplication operators for the at least two linear layers based on structured data of the at least two linear layers; and fuse the matrix multiplication operators of the at least two linear layers with the activation functions between the at least two linear layers, such that the output of the preceding matrix multiplication operator is used as the input data for the following matrix multiplication operator.

[0230] In an exemplary embodiment, the apparatus is further configured to: create a preset two-dimensional array, wherein the length of a set of vector data in the two-dimensional array is determined according to the number of parameters of the structured data corresponding to the multiple network layers of the model; fill the structured data of the multiple network layers into the two-dimensional array, wherein the structured data of different network layers are located in different rows of the two-dimensional array; and create corresponding row indices and column indices for each row and each column of the two-dimensional array, wherein the row index of the two-dimensional array is used to indicate the layer number of the network layer, and the column index of the two-dimensional array is used to indicate the parameter information of the network layer, the parameter information including at least one of the following: the type of the network layer, the number of input data of the network layer, the number of output data of the network layer, weight information, and bias information.

[0231] In one exemplary embodiment, the apparatus is further configured to: create an initial binary file and an initial header list, wherein the binary file stores the data content of the weight information and the data content of the bias information of the network layers, and the header list stores the header information of the network layers; extract the data content of the weight information and the data content of the bias information of each network layer from the structured data of each network layer of the model; store the data content of the weight information and the data content of the bias information of each network layer in the binary file according to the order of each network layer; create header structure data corresponding to each network layer according to the data volume and storage location of the weight data and bias data of each network layer, and store the header structure data in the header list, wherein the header structure data includes the layer number, weight size, bias size, number of bytes occupied by each weight, and offset position of the weight.

[0232] In one exemplary embodiment, the apparatus is further configured to: acquire hardware parameters of the target hardware platform, wherein the hardware parameters include the number of interfaces and the address space corresponding to the interfaces; allocate corresponding interfaces and address spaces to each network layer represented by a two-dimensional array of the model according to the hardware parameters; and write the data of each network layer, represented by the two-dimensional array, into its corresponding address space according to the interface and address space corresponding to each network layer.

[0233] In an exemplary embodiment, the apparatus is further configured to: create a first relation table and a second relation table based on hardware parameters, wherein the first relation table indicates the correspondence between the identifier of a memory block of the target hardware platform when it is an output memory block and the layer number of the network layer of the model, and the second relation table indicates the correspondence between the identifier of a memory block of the target hardware platform when it is an input memory block and the layer number of the network layer of the model. The apparatus iterates through a two-dimensional array of the model to allocate corresponding input memory blocks and output memory blocks for each network layer of the model. It updates the first and second relation tables based on the identifiers of the input and output memory blocks corresponding to each network layer. It allocates corresponding interfaces and address spaces for the memory blocks corresponding to each network layer based on the first and second relation tables.

[0234] In one exemplary embodiment, the apparatus is further configured to: read a first identifier of each output memory block from a first relation table; determine, according to a second relation table, the amount of input data present in the case of the memory block corresponding to the first identifier being used as an input memory block; allocate address space for the memory block corresponding to the first identifier based on the amount of input data; and, if no interface is allocated for the memory block corresponding to the first identifier when it is used as an input memory block, allocate an interface for the memory block corresponding to the first identifier.

[0235] In one exemplary embodiment, the apparatus is further configured to: extract a model file stored in the address space of a target hardware platform; sequentially extract the computational parameters of each network layer from the model file according to the logical order of computation; divide the computational parameters of the current network layer into multiple sub-matrices based on the size of the computing module of the target hardware platform; determine the address and data size of each sub-matrix in a memory block; move the multiple sub-matrices to the target address space based on their addresses and data sizes in the memory block; perform combined operations on the multiple sub-matrices in the target address space to generate instructions corresponding to the current network layer; and return to execute the next network layer of the model as the current network layer. The steps of dividing the computational parameters of the current network layer into multiple sub-matrices based on the size of the computing module of the target hardware platform continue until instructions corresponding to all network layers of the model are generated. The instructions corresponding to all network layers of the model are stored in the instruction address space of the target hardware platform so that the target hardware platform can call the stored instructions during runtime.

[0236] For a description of the features in the embodiment corresponding to the model compilation device, please refer to the relevant description in the embodiment corresponding to the model compilation method, which will not be repeated here.

[0237] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in the compilation method embodiments of any of the above models.

[0238] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the compilation method embodiments of any of the above models at runtime.

[0239] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0240] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the compilation method embodiments of any of the above models.

[0241] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the compilation method embodiments of any of the above models.

[0242] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] The foregoing has provided a detailed description of the compilation method, apparatus, electronic device, computer-readable storage medium, and computer program product for a model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for compiling a model, characterized in that, The method includes: Obtain the target format of intermediate representation data that is compatible with multiple different target hardware platforms; The model to be compiled is parsed so that the multiple network layers included in the model are represented as structured data in a preset format, wherein the structured data includes the parameter information of the corresponding network layers; The structured data corresponding to at least two first network layers in the model are fused to merge the computational operations of the at least two first network layers. The at least two first network layers include associated continuous computational operations. The fusion process includes: determining the at least two first network layers that can be merged based on the association patterns between the network layers; determining a fusion formula for the computational operations of the at least two first network layers according to the principle of mathematical equivalence; determining the operators corresponding to the at least two first network layers based on the semantic information corresponding to the at least two first network layers; extracting operator parameters for the corresponding operators based on the structured data corresponding to the at least two first network layers, wherein the operator parameters include at least one of the following: weight, scaling factor, bias factor, tensor dimension; fusing the operators corresponding to the at least two first network layers using the fusion formula, and substituting the operator parameters to obtain an objective function, wherein the objective function is equivalent to the merged representation of the computational operations of the at least two first network layers. The structured data of multiple network layers of the model after the fusion process is converted into intermediate representation data in the target format and filled into a preset two-dimensional array to obtain a two-dimensional array representation of the model. The structured data of each network layer corresponds to a set of vector data in the two-dimensional array. The model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

2. The compilation method of the model according to claim 1, characterized in that, The process of parsing the model to be compiled, to represent the multiple network layers of the model as structured data in a preset format, includes: The generation framework based on the model to be compiled reads the original description data of the model, wherein the original description data includes the structural parameters of the model; Semantic matching is performed on the structural parameters of the model to determine the semantic information corresponding to each network layer included in the model, wherein the semantic information is used to indicate the computational operation performed by the corresponding network layer; The association patterns between the network layers of the model are identified based on the semantic information corresponding to each network layer, wherein the association patterns are used to indicate the correlation or continuity of the operations between different network layers. Extract the parameter information corresponding to each network layer in the model; The parameter information, semantic information, and relationship patterns between each network layer are converted into structured data in a preset format for representation.

3. The compilation method of the model according to claim 2, characterized in that, The method further includes: The type of each network layer in the model is determined based on the semantic information corresponding to each network layer; Based on the type of each network layer, determine whether there are multiple consecutive network layers that meet the target combination, wherein the target combination includes convolutional layers, batch normalization layers, and activation function layers; If it is determined that there exists a continuous set of convolutional layers, batch normalization layers, and activation function layers, then the continuous set of convolutional layers, batch normalization layers, and activation function layers are defined as the first network layer.

4. The compilation method of the model according to claim 2, characterized in that, The step of fusing the structured data corresponding to at least two first network layers in the model to merge the computational operations of the at least two first network layers further includes: Based on the semantic information corresponding to each network layer, at least two consecutive linear layers in the model are determined, wherein the at least two consecutive linear layers are referred to as the at least two first network layers; Extract the matrix multiplication operators of the at least two linear layers from the structured data of the at least two linear layers; The matrix multiplication operators of the at least two linear layers and the activation functions between the at least two linear layers are fused so that the output of the previous matrix multiplication operator is used as the input data of the next matrix multiplication operator.

5. The method for compiling the model according to any one of claims 1-4, characterized in that, The step of converting the structured data of multiple network layers of the model after the fusion process into intermediate representation data in the target format and filling it into a preset two-dimensional array includes: Create a preset two-dimensional array, wherein the length of a set of vector data in the two-dimensional array is determined according to the number of parameters of the structured data corresponding to the multiple network layers of the model; The structured data of the multiple network layers are respectively filled into the two-dimensional array, wherein the structured data of different network layers are located in different rows of the two-dimensional array; For each row and each column of the two-dimensional array, a corresponding row index and column index are created respectively. The row index of the two-dimensional array is used to indicate the layer number of the network layer, and the column index of the two-dimensional array is used to indicate the parameter information of the network layer. The parameter information includes at least one of the following: the type of the network layer, the number of input data of the network layer, the number of output data of the network layer, weight information, and bias information.

6. The compilation method of the model according to claim 5, characterized in that, The method further includes: Create an initial binary file and an initial header list, wherein the binary file is used to store the data content of the weight information and the data content of the bias information of the network layer, and the header list is used to store the header information of the network layer; Extract the weight information and bias information of each network layer from the structured data of each network layer in the model; The weight information and bias information of each network layer are stored in the binary file in the order of each network layer. Based on the amount and storage location of the weight and bias data of each network layer, create a header structure data corresponding to each network layer, and store the header structure data in the header list. The header structure data includes the layer number, weight size, bias size, number of bytes occupied by each weight, and offset position of the weight.

7. The method for compiling the model according to any one of claims 1-4, characterized in that, The method further includes: Obtain the hardware parameters of the target hardware platform, wherein the hardware parameters include the number of interfaces and the address space corresponding to the interfaces; Based on the hardware parameters, allocate corresponding interfaces and address spaces to each network layer represented by the two-dimensional array of the model; Based on the interface and address space corresponding to each network layer, the data of each network layer, represented by a two-dimensional array, is written into the corresponding address space.

8. The compilation method of the model according to claim 7, characterized in that, The process of allocating corresponding interfaces and address spaces to each network layer represented by the two-dimensional array of the model based on the hardware parameters includes: A first relation table and a second relation table are created based on the hardware parameters. The first relation table is used to indicate the correspondence between the identifier of the memory block of the target hardware platform when it is used as an output memory block and the layer number of the network layer of the model. The second relation table is used to indicate the correspondence between the identifier of the memory block of the target hardware platform when it is used as an input memory block and the layer number of the network layer of the model. Traverse the two-dimensional array of the model and allocate corresponding input memory blocks and output memory blocks for each network layer of the model; Update the first relation table and the second relation table according to the identifiers of the input memory blocks and output memory blocks corresponding to each network layer; Based on the first relation table and the second relation table, allocate corresponding interfaces and address spaces for memory blocks corresponding to each network layer.

9. The compilation method of the model according to claim 8, characterized in that, The step of allocating corresponding interfaces and address spaces to memory blocks corresponding to each network layer according to the first relationship table and the second relationship table includes: Read the first identifier of each output memory block from the first relation table; The number of input data that exist in the memory block corresponding to the first identifier when it is used as an input memory block is determined according to the second relationship table; Allocate address space for the memory block corresponding to the first identifier based on the quantity of input data; Furthermore, if no interface is allocated for the memory block corresponding to the first identifier when it is used as an input memory block, then an interface is allocated for the memory block corresponding to the first identifier.

10. The method for compiling the model according to claim 8, characterized in that, The method further includes: Extract the model files stored in the address space of the target hardware platform; The computational parameters of each network layer are extracted sequentially from the model file according to the logical order of calculation. Based on the size of the computing module of the target hardware platform, the computational parameters of the current network layer are divided into multiple sub-matrices; Determine the addresses and data amounts of the plurality of sub-matrices in the memory block respectively; Based on the addresses and data amounts of the multiple sub-matrices in the memory block, the multiple sub-matrices are moved to the target address space; The multiple sub-matrices are combined and operated on in the target address space to generate the instruction corresponding to the current network layer; The next network layer of the model is taken as the current network layer and returned for execution: the step of dividing the operation parameters of the current network layer into multiple sub-matrices according to the size of the computing module of the target hardware platform, until the instructions corresponding to all network layers of the model are generated respectively; The instructions corresponding to all network layers of the model are stored in the instruction address space of the target hardware platform so that the target hardware platform can call the stored instructions at runtime.

11. A model compilation apparatus, characterized in that, include: The acquisition module is used to acquire the target format of intermediate representation data that is compatible with multiple different target hardware platforms; The parsing module is used to parse the model to be compiled, so as to represent the multiple network layers included in the model as structured data in a preset format, wherein the structured data includes the parameter information of the corresponding network layers; A fusion module is used to fuse the structured data corresponding to at least two first network layers in the model to merge the computational operations of the at least two first network layers. The at least two first network layers include associated continuous computational operations. The fusion of the structured data corresponding to the at least two first network layers in the model includes: determining the at least two first network layers that can be merged based on the association patterns between the network layers; determining a fusion formula for the computational operations of the at least two first network layers according to the principle of mathematical equivalence; determining the operators corresponding to the at least two first network layers respectively based on the semantic information corresponding to the at least two first network layers; extracting the operator parameters of the corresponding operators based on the structured data corresponding to the at least two first network layers, wherein the operator parameters include at least one of the following: weight, scaling factor, bias factor, tensor dimension; fusing the operators corresponding to the at least two first network layers respectively using the fusion formula, and substituting the operator parameters to obtain an objective function, wherein the objective function is equivalent to the merged representation of the computational operations of the at least two first network layers. The compilation module is used to convert the structured data of multiple network layers of the model after the fusion process into intermediate representation data in the target format and fill it into a preset two-dimensional array to obtain a two-dimensional array representation of the model. The structured data of each network layer corresponds to a set of vector data in the two-dimensional array, and the model represented by the two-dimensional array is compatible with multiple different target hardware platforms.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

Images