A Method and Device for Deploying NPU Neural Network Models Based on TVM

Abstract

The present application provides a TVM-based NPU neural network model deployment method and device, including: obtaining the neural network model to be deployed and the hardware attribute information of the current NPU; determining the convolution operator to be deployed to the current NPU based on the computational logic of the neural network model; determining the optimal parameter configuration of each convolution operator in the current NPU based on the hardware attribute information of the current NPU and the parameter information of each convolution operator; adjusting the data layout and convolution weight of the TIR layer and modifying the parameters of the relay layer based on the optimal parameter configuration and the hardware attribute information of the current NPU to obtain a target TVM adapted to the current NPU; generating a deployment file for the neural network model suitable for the current NPU based on the target TVM. By adjusting the parameters of the TVM's TIR layer and relay layer accordingly, the obtained target TVM can be adapted to the NPU.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for deploying an NPU neural network model based on TVM. Background Art

[0002] With the continuous development of artificial intelligence and big data, more and more dedicated AI chips have emerged. Among them, embedded neural network processors (Neural-network Processing Unit, referred to as NPU) adopt a data-driven parallel computing architecture and are good at processing massive multimedia data such as videos and images.

[0003] In existing technologies, in order to simply, quickly, and efficiently deploy deep learning models based on different frameworks to their respective NPU hardware platforms, hardware manufacturers or IP vendors need to develop inference software stacks that integrate model optimization, code generation, and compilation. TVM, as an open-source, general-purpose machine learning compilation framework, has become the preferred choice for many developers and businesses.

[0004] However, due to the differences in hardware configurations among NPUs produced by different NPU manufacturers, the current TVM cannot adapt to NPUs from different hardware manufacturers. Summary of the Invention

[0005] This application provides a TVM-based NPU neural network model deployment method and device to address the defects of the existing technology such as TVM cannot adapt to NPUs of different hardware manufacturers.

[0006] The first aspect of the present application provides a TVM-based NPU neural network model deployment method, including:

[0007] Obtain the neural network model to be deployed and the hardware attribute information of the current NPU;

[0008] Determining a convolution operator to be deployed to the current NPU according to the computational logic of the neural network model;

[0009] Determining, based on the hardware attribute information of the current NPU and the parameter information of each convolution operator, an optimal parameter configuration of each convolution operator in the current NPU;

[0010] According to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, the data layout and convolution weight of the tir layer are adjusted, and the parameters of the relay layer are modified to obtain a target TVM adapted to the current NPU;

[0011] Based on the target TVM, a deployment file of the neural network model suitable for the current NPU is generated, so as to deploy the neural network model to the current NPU using the deployment file.

[0012] Optionally, determining the optimal parameter configuration of each convolution operator in the current NPU according to the hardware attribute information of the current NPU and the parameter information of each convolution operator includes:

[0013] Based on a preset performance evaluation model, the optimal parameter configuration of each convolution operator in the current NPU is determined according to the hardware constraints of the current NPU, the hardware attribute information of the current NPU and the parameter information of each convolution operator.

[0014] Optionally, adjusting the data layout and convolution weight of the TIR layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU includes:

[0015] Modify the data layout of the tensor buffer of the TVM storage layer according to the data input requirements represented by the hardware attribute information of the current NPU to adjust the data distribution of the TIR layer input features; wherein the tensor buffer is used to cache the TIR layer input features;

[0016] With the goal of achieving the matrix multiplication calculation corresponding to the optimal parameter configuration in the current NPU, the convolution weight of the tir layer is adjusted according to the weight distribution requirements represented by the hardware attribute information of the current NPU.

[0017] Optionally, modifying the parameters of the relay layer includes:

[0018] Obtaining adjustment information of the convolution weights of the tir layer;

[0019] According to the adjustment information of the convolution weight of the TIR layer, the convolution weight of the relay layer is adjusted accordingly to make the relay layer match the TIR layer.

[0020] Optionally, generating a deployment file of the neural network model applicable to the current NPU based on the target TVM includes:

[0021] Based on the target TVM, the tir function corresponding to the convolution operator is compiled to generate a deployment file of the neural network model suitable for the current NPU.

[0022] Optionally, before compiling the tir function corresponding to the convolution operator based on the target TVM, the method further includes:

[0023] Based on the target TVM and the computational logic of the neural network model, a fusion relay graph is created;

[0024] Locating the relay fusion operator subgraph corresponding to the convolution operator from the fusion relay graph;

[0025] According to the relay fusion operator subgraph, the TIR function corresponding to the convolution operator is determined.

[0026] Optionally, the hardware attribute information of the NPU includes input channels, output features, and the number of parallel computing channels of the NPU computing unit.

[0027] The second aspect of the present application provides an NPU neural network model deployment device based on TVM, including:

[0028] The acquisition module is used to obtain the neural network model to be deployed and the hardware attribute information of the current NPU;

[0029] An analysis module, configured to determine a convolution operator to be deployed to the current NPU based on the computational logic of the neural network model;

[0030] A determination module, configured to determine an optimal parameter configuration of each convolution operator in the current NPU based on the hardware attribute information of the current NPU and the parameter information of each convolution operator;

[0031] An adjustment module is configured to adjust the data layout and convolution weight of the tir layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, and modify the parameters of the relay layer to obtain a target TVM adapted to the current NPU;

[0032] A deployment module is used to generate a deployment file of the neural network model suitable for the current NPU based on the target TVM, so as to deploy the neural network model to the current NPU using the deployment file.

[0033] Optionally, the determining module is specifically configured to:

[0034] Based on a preset performance evaluation model, the optimal parameter configuration of each convolution operator in the current NPU is determined according to the hardware constraints of the current NPU, the hardware attribute information of the current NPU and the parameter information of each convolution operator.

[0035] Optionally, the adjustment module is specifically configured to:

[0036] Modify the data layout of the tensor buffer of the TVM storage layer according to the data input requirements represented by the hardware attribute information of the current NPU to adjust the data distribution of the TIR layer input features; wherein the tensor buffer is used to cache the TIR layer input features;

[0037] With the goal of achieving the cyclic calculation corresponding to the optimal parameter configuration in the current NPU, the convolution weight of the tir layer is adjusted according to the weight distribution requirements represented by the hardware attribute information of the current NPU.

[0038] Optionally, the adjustment module is specifically configured to:

[0039] Obtaining adjustment information of the convolution weights of the tir layer;

[0040] According to the adjustment information of the convolution weight of the TIR layer, the convolution weight of the relay layer is adjusted accordingly to make the relay layer match the TIR layer.

[0041] Optionally, the deployment module is specifically configured to:

[0042] Based on the target TVM, the tir function corresponding to the convolution operator is compiled to generate a deployment file of the neural network model suitable for the current NPU.

[0043] Optionally, the hardware attribute information of the NPU includes input channels, output features, and the number of parallel computing channels of the NPU computing unit.

[0044] Optionally, the device further includes:

[0045] A data management module is used to create a fused relay graph based on the target TVM and the computational logic of the neural network model; locate the relay fusion operator subgraph corresponding to the convolution operator from the fused relay graph; and determine the TIR function corresponding to the convolution operator based on the relay fusion operator subgraph.

[0046] A third aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0047] The memory stores computer-executable instructions;

[0048] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0049] The fourth aspect of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0050] The technical solution of this application has the following advantages:

[0051] The present application provides a TVM-based NPU neural network model deployment method and device, the method comprising: obtaining a neural network model to be deployed and hardware attribute information of the current NPU; determining the convolution operator to be deployed to the current NPU based on the computational logic of the neural network model; determining the optimal parameter configuration of each convolution operator in the current NPU based on the hardware attribute information of the current NPU and the parameter information of each convolution operator; adjusting the data layout and convolution weight of the TIR layer and modifying the parameters of the relay layer based on the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU to obtain a target TVM adapted to the current NPU; based on the target TVM, generating a deployment file for the neural network model suitable for the current NPU, so as to deploy the neural network model to the current NPU using the deployment file. The method provided by the above scheme determines the optimal parameter configuration suitable for the NPU to implement convolution calculation, and further adjusts the parameters of the TIR layer and relay layer of the TVM accordingly in combination with the hardware attribute information and optimal parameter configuration of the NPU, so that the obtained target TVM can adapt to the NPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 This is a schematic diagram of the network structure based on the embodiments of the present application;

[0054] Figure 2 A flowchart of a TVM-based NPU neural network model deployment method provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of the NPU provided in an embodiment of the present application;

[0056] Figure 4 An exemplary software loop calculation flow chart provided for an embodiment of the present application;

[0057] Figure 5A schematic diagram of calculation logic of a calculation unit provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of another computing logic of a computing unit provided in an embodiment of the present application;

[0059] Figure 7 An overview of the convolution calculation logic provided in the embodiments of this application;

[0060] Figure 8 An exemplary convolution weight rearrangement flow chart provided in an embodiment of the present application;

[0061] Figure 9 Schematic diagram of data arrangement in the input feature buffer IBUF and convolution weight buffer WBUF provided in an embodiment of the present application;

[0062] Figure 10 A schematic diagram of the structure of the TVM-based NPU neural network model deployment device provided in an embodiment of the present application;

[0063] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0064] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0065] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0066] In addition, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. In the description of the following embodiments, "plurality" means more than two, unless otherwise specifically defined.

[0067] Convolution and matrix multiplication are common and computationally intensive operations in deep learning models (neural network models). Most NPUs have dedicated modules designed to accelerate these operations, generally called NUs (neural processing units). To improve convolution computational efficiency, conventional hardware designs implement parallel computations across three dimensions: input channel, output feature, and output channel. To accommodate the unique data arrangements of the hardware, the input, output, and weights of the convolution operation must be converted accordingly during model compilation. Currently, while TVM is well-adapted to CPU and GPU hardware, it is not yet able to optimize, compile, and efficiently execute computations on the hardware backends of NPUs from different vendors.

[0068] In response to the above problems, the embodiments of the present application provide a TVM-based NPU neural network model deployment method and device, which obtains the hardware attribute information of the neural network model to be deployed and the current NPU; determines the convolution operator to be deployed to the current NPU according to the calculation logic of the neural network model; determines the optimal parameter configuration of each convolution operator in the current NPU according to the hardware attribute information of the current NPU and the parameter information of each convolution operator; adjusts the data layout and convolution weight of the TIR layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, and modifies the parameters of the relay layer to obtain a target TVM adapted to the current NPU; based on the target TVM, generates a deployment file for the neural network model suitable for the current NPU, so as to use the deployment file to deploy the neural network model to the current NPU. The method provided by the above scheme determines the optimal parameter configuration suitable for the NPU to implement convolution calculation, and further adjusts the parameters of the TIR layer and relay layer of the TVM in combination with the hardware attribute information and optimal parameter configuration of the NPU, so that the obtained target TVM can adapt to the NPU.

[0069] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of the present invention in conjunction with the accompanying drawings.

[0070] First, the network structure on which this application is based is described:

[0071] The TVM-based NPU neural network model deployment method and device provided in the embodiment of the present application are suitable for deploying neural network models on NPU to achieve multi-layer cyclic convolution acceleration of NPU. Figure 1As shown, this is a schematic diagram of the network structure on which the embodiment of the present application is based, which mainly includes an NPU and an NPU neural network model deployment device based on TVM. The device includes a TVM. Specifically, the device first collects the hardware attribute information of the NPU and the neural network model to be deployed, and then adjusts the network architecture of the TVM according to its hardware attribute information and the parameter information of the convolution operator in the neural network model, and then realizes the compilation and deployment of the neural network model based on the TVM.

[0072] The present invention provides a TVM-based method for deploying a neural network model on an NPU, which is used to deploy a neural network model on an NPU to accelerate multi-layer recurrent convolutions. The embodiments of the present invention are implemented by electronic devices, such as servers, desktop computers, laptops, tablets, and other electronic devices that can be used to deploy neural network models on an NPU.

[0073] like Figure 2 FIG. 1 is a flow chart of a TVM-based NPU neural network model deployment method provided in an embodiment of the present application, the method comprising:

[0074] Step 201: Obtain the neural network model to be deployed and the hardware attribute information of the current NPU.

[0075] Among them, the hardware attribute information of the NPU includes the input channels, output characteristics and the number of parallel calculations of the output channels of the NPU computing unit. Figure 3 As shown, this is a schematic diagram of the structure of the NPU provided in an embodiment of the present application. DM represents the storage module of the NPU, PE represents the data processing module of the NPU, cube represents the computing unit in the data processing module, WBUF represents the convolution weight buffer, which is used to store the convolution weight (weight) required by the data processing module for convolution calculation, IBUF represents the input feature buffer, which is used to store the input data required by the data processing module for convolution calculation, and OBUF represents the output feature buffer, which is used to store the intermediate results (psum) and final results generated by the PE unit when performing convolution calculation. In order to improve computing power, each NPU contains multiple PEs. Assuming that there is 1 cube in each PE, α, β, and γ in the cube are fixed values of 16. α represents 16 output channels in parallel, and β is 8 or 16 input channels in parallel depending on the input data type. For unified description, the following content of the embodiment of the present application is set to 16. γ represents 16 pixels in parallel in the height and width directions of the output feature.

[0076] Step 202: Determine the convolution operator to be deployed to the current NPU based on the computational logic of the neural network model.

[0077] It should be noted that the computing logic of the neural network model includes not only convolution computing logic, but also other types of computing logic. In order to meet the hardware characteristics of the NPU, it is necessary to focus on compiling the convolution computing logic during the model compilation stage. The compilation process of other computing logic refers to the existing technology and is not limited in the embodiments of this application.

[0078] Step 203: Determine the optimal parameter configuration of each convolution operator in the current NPU based on the hardware attribute information of the current NPU and the parameter information of each convolution operator.

[0079] Among them, the parameter information of the convolution operator includes input data, convolution weight, output data and convolution step size.

[0080] It should be noted that at the NPU hardware level, it is not possible to perform convolution calculations directly. Therefore, it is necessary to convert the convolution calculation logic of the neural network model into matrix multiplication calculations that can be run based on the NPU hardware, so that the NPU can achieve the same computing power as convolution calculations by performing matrix multiplication calculations.

[0081] Specifically, the hardware computing capability represented by the hardware attribute information of the current NPU can be used as a limitation, and according to the parameter information of each convolution operator, the optimal parameter configuration that enables the current NPU to implement the corresponding matrix multiplication calculation can be determined, so that the current NPU can perform matrix multiplication calculations equivalent to the calculation results of the convolution calculation logic based on the optimal parameter configuration.

[0082] It should be further explained that due to the hardware limitations of the NPU, the basic arithmetic unit of the NPU can only perform matrix multiplication. NPU configuration parameters are divided into software and hardware layers. By configuring the optimal parameters at the software layer, the NPU hardware layer can perform the corresponding matrix multiplication.

[0083] In step 204, according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, the data layout and convolution weight of the TIR layer are adjusted, and the parameters of the relay layer are modified to obtain the target TVM adapted to the current NPU.

[0084] It should be noted that TVM consists of the tir layer and the relay layer. The tir layer contains the definition of the low-level intermediate representation and uses tirPrimFunc to represent functions that can be converted by the tir layer. tirPrimFunc is a low-level functional programming representation that includes loop nesting, multi-dimensional load and store, thread processing, and vector and tensor operation instructions. It is generally used to define the operation of an operator, corresponding to a certain layer in a model. The relay layer contains the definition of the high-level intermediate representation, which is used to express the high-level functions of the complete model. RelayFunction is a high-level functional programming representation. A relayFunction usually corresponds to an end-to-end model and is a computational graph that supports control flow, recursion, and complex data structures. During the entire TVM compilation process, a relay function may be optimized into multiple tirPrimFuncs.

[0085] Among them, the relay layer includes multiple relay operators, which are divided into convolution operators and other conventional operators. The embodiment of the present application mainly modifies the parameters of the convolution operator of the relay layer.

[0086] Specifically, because the current TVM's compilation and conversion logic for the TIR and relay layers is not suitable for the hardware characteristics of the NPU and the way convolutional computing is implemented in the NPU, in order to obtain an NPU-adapted TVM, the data layout and convolution weights of the TIR layer can be adjusted based on the optimal parameter configuration of each convolution operator in the current NPU and the current NPU hardware attribute information. At the same time, to ensure that the TVM TIR and relay layers maintain a match, the relay layer parameters are adaptively modified to obtain a target TVM that is adapted to the current NPU.

[0087] Step 205: Generate a deployment file of the neural network model suitable for the current NPU based on the target TVM, and use the deployment file to deploy the neural network model to the current NPU.

[0088] The deployment file is a machine language file recognizable by the NPU, such as a binary file, etc. The specific compilation process and simulation process of the target TVM can refer to the existing technology and are not limited in the embodiments of the present application.

[0089] Based on the above embodiment, as an implementable manner, in one embodiment, the optimal parameter configuration of each convolution operator in the current NPU is determined based on the hardware attribute information of the current NPU and the parameter information of each convolution operator, including:

[0090] Step 2031 : Based on a preset performance evaluation model, the optimal parameter configuration of each convolution operator in the current NPU is determined according to the hardware constraints of the current NPU, the hardware attribute information of the current NPU, and the parameter information of each convolution operator.

[0091] Among them, the performance evaluation model is the costmodel, which is used to randomly generate multiple possible parameter configuration schemes corresponding to the convolution operator based on the hardware limitations of the NPU and the parameter information of the convolution operator. It also refers to the hardware attribute information of the NPU to estimate the NPU performance under different parameter configurations.

[0092] Specifically, for any convolution operator, the hardware attribute information of the current NPU and the parameter information of the convolution operator are input into the performance evaluation model. The performance evaluation model is constrained by the hardware limitations of the current NPU. According to the received information, a plurality of parameter configuration schemes are determined, and the NPU performance corresponding to the various parameter configuration schemes is determined. According to the parameter configuration scheme corresponding to the best NPU performance, the optimal parameter configuration corresponding to the convolution operator is determined. Similarly, the optimal parameter configuration of each convolution operator in the current NPU is determined. The parameter configuration includes: epsilon, delta, zeta, dense, epsilon_times, delta_times, dense_times, zeta_times and group parameters. Furthermore, based on the optimal parameter configuration corresponding to these convolution operators, the convolution calculation of the convolution operator can be converted into a matrix multiplication calculation that can be run by the NPU hardware. For example, Figure 4 The figure shows an exemplary software loop calculation flow chart provided by an embodiment of the present application. The NPU uses a multi-layer loop method to implement convolution calculations. This process includes hardware layer parameter configuration and software layer parameter configuration. Hardware layer parameters include: the number of parallel calculations alpha (α) for the output channel of the NPU computing unit, the number of parallel calculations beta (β) for the input channel, and the number of parallel calculations gamma (γ) for the output feature.

[0093] The software layer parameters from the inner layer to the outer layer are epsilon, delta, zeta, dense, epsilon_times, zeta_times, dense_times, delta_times, group. Note: kernel height = kernel width = 3, k0 represents the first value in the kernel size (kh*kw) in the weight, d0 represents the first pixel in the height and width directions of each channel dimension in the input feature value. The software layer parameters are described as follows:

[0094] Epsilon: The basic calculation loop of the software, iteratively performs calculations in the ci (input channel) direction, runs an epsilon loop for each beat, runs the calculations in the alpha (16) co (output channel) direction in parallel, the beta (16) input channel direction in parallel, and the gamma (16) pixel results, and obtains the psum results of 16 co and 16 pixels, such as Figure 4 As shown on the right, the epsilon loop parameter must be divisible by the kernel size (kh*kw).

[0095] Delta: Iterate the calculation in the co direction. In each delta loop, epsilon basic loops are run and the same input feature data is used. After the loop, the psum results of delta*alpha co and gamma pixels are obtained.

[0096] zeta: Iteratively calculates pixels in the height and width directions of the convolution. In each zeta loop, delta*epsilon basic loops are run and the same weight is used. After the loop ends, the psum results of delta*alpha co and zeta*gamma pixels are obtained.

[0097] dense: Iteratively calculates pixels in the convolution dense (depth) direction, runs zeta*delta*epsilon basic loops in each dense loop, uses the same weight for each dense loop, and obtains the psum results of delta*alpha co and dense*zeta*gamma pixels after the loop ends.

[0098] epsilon_times: Iterates the calculation in the ci direction, obtaining delta*alpha co and dense*zeta*gamma pixels at the end of the loop. The last epsilon_times epsilon cycle may be less than the previous epsilon cycle, which is set by another parameter (last_epsilon).

[0099] zeta_times: Iterates the convolution for pixels in both the height and width directions, using the same weights for each zeta_times loop. After each loop, the final result is delta*alpha co and dense*ho*wo pixels. The zeta value within the last zeta_times loop may be less than the zeta value of the previous loop, as controlled by another parameter (last_zeta).

[0100] dense_times: Iterates the convolution depthwise pixel calculations, using the same weights for each dense_times loop. At the end of the loop, delta*alpha co and do*ho*wo pixels are obtained. The last dense_times iteration may have fewer dense pixels than the previous iteration, as determined by another parameter (last_dense).

[0101] delta_times: Iterates the calculation in the co direction, using the same input feature data for each delta_times loop. After the loop completes, the final result is output channels co and do*ho*wo pixels. The last delta_times loop may have fewer deltas than the previous loop, as set by the last_delta parameter.

[0102] Group: Iterates the convolution group-wise. Each group loop is considered a complete convolution operation. After the loop completes, the final result is the number of groups, the number of output channels, and the number of pixels. Each group loop can be considered an independent convolution calculation. The number of co and pixel data in each group must be the same.

[0103] Based on the above embodiment, as an implementable manner, in one embodiment, the data layout and convolution weight of the TIR layer are adjusted according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, including:

[0104] Step 2041: Modify the data layout of the tensor buffer of the TVM storage layer based on the data input requirements represented by the current NPU hardware attribute information to adjust the data distribution of the TIR layer input features; wherein the tensor buffer is used to cache the TIR layer input features;

[0105] Step 2042 , with the goal of achieving the matrix multiplication calculation corresponding to the optimal parameter configuration in the current NPU, adjust the convolution weight of the tir layer according to the weight distribution requirements represented by the hardware attribute information of the current NPU.

[0106] In order to facilitate those skilled in the art to clearly understand the hardware characteristics of NPU and the implementation of convolution calculation in NPU, Figure 5As shown in FIG. 1 , a schematic diagram of a computing unit calculation logic provided by an embodiment of the present application is provided. A cube operation (cube-op) is a basic operation unit, that is, a hardware loop parameter is executed in parallel once to obtain the psum data of one cube-op, as shown in FIG. Figure 5 Before executing the cube operation, the convolution weights and input features need to be rearranged. Input data rearrangement is completed during the data transfer between the NPU storage module DM and the feature buffer IBUF. Since the convolution weights are mostly static data, they can be rearranged offline to reduce hardware design complexity and energy consumption.

[0107] Furthermore, if Figure 6 As shown, it is another schematic diagram of the calculation logic of the calculation unit provided by the embodiment of the present application, where delta*epsilon*zeta cube-ops form a window-op. Figure 6 This is a window-op diagram. The calculation direction of each window-op is indicated by the curved arrows. Matrix multiplication is performed on w_0_0 (the cube-op in the first row and first column of matrix A) and w_0_1 with i_0_0 (the cube-op in the first row and first column of matrix B) and i_1_0 to obtain the psum result of the cube-op. Element-wise addition is performed on the psum result to obtain the psum result of the window-op, namely p_0_0 (the cube-op in the first row and first column of matrix C). This continues in this manner, completing delta*epsilon*zeta cycles to complete a window-op. In each window-op, delta represents the reuse of input data in the input channel, while zeta represents the reuse of the convolution weights. Therefore, the delta, epsilon, and zeta values affect the bandwidth between the NPU's storage module (DM) and the data processing module (PE).

[0108] For example, Figure 7 As shown in the figure, it is an overview of the convolution calculation logic provided by the embodiment of the present application, wherein the convolution parameters are num_ci=64, num_co=64, kw(kernelwidth)=3, kh(kernel height)=3, hi(inputheight)=32, wi(input width)=32, ho(output height)=32, wo(output width)=32, padding=1, strides=1. The summary diagram of conv2d implemented in NPU is shown in the figure. Figure 7As shown in the figure, the convolution weights of Conv2d and the input data are rearranged and converted into matrix multiplication. The window-op and cube-op are executed respectively through the software layer parameters and the hardware layer parameters to obtain the final result.

[0109] Specifically, the above step 2041 includes:

[0110] Step 20411: Based on the buffer split primitive, split the target axis of the TVM tensor buffer according to the data input requirements represented by the hardware attribute information; the tensor buffer is located in the storage layer of TVM and is used to cache the input features of the TIR layer;

[0111] Step 20412, based on the buffer reorder primitive, transpose the tensor buffer according to the order of the target axes in the input features.

[0112] Specifically, the hardware attribute information of the NPU includes the input channels, output features, and the number of parallel computing of the output channels of the NPU computing unit. If the input channels, output features, and the number of parallel computing of the output channels of the NPU computing unit are all 16, then the data input requirement representing the NUP is that the dimension of the input features is 16.

[0113] It should be noted that when scheduling the tir function of the convolution calculation, if the data layout of the input feature is to be converted, the data layout of the tensor buffer needs to be changed on the storage layer memory. To this end, the embodiment of the present application adds the TVM scheduling primitives split_buffer (buffer split) and reorder_buffer (buffer rearrangement). The buffer split primitive splits the target axis of the tensor buffer tensor buffer and aligns it according to the given factor parameter, where the given factor can be set according to the data input requirements. If the data input requirement is that the dimension of the input feature is 16, the given factor is set to 16. The buffer rearrangement primitive reorder_buffer will transpose the tensor buffer according to the order of the axes in the input feature list parameters. Among them, the specific steps for modifying the data layout of the tensor buffer are as follows:

[0114] Step 1: Rewrite the input tensor buffer. Specifically, you can use the buffer split primitive to split the C (channel) dimension of the input tensor buffer so that its data layout becomes NCcHW. Assuming a factor c = 16, use the buffer reorder primitive to convert NCcHW to NCHWc; where N (batch) represents batch, C (channel) represents channel, H (height) represents input matrix height, and W (width) represents input matrix width.

[0115] Step 2: Modify the loop value of the block in the tir function. For subsequent tensor quantization, the loop value in the block corresponding to the tensor buffer in the first step can be transformed accordingly. First, use the buffer split primitive to split the loop value corresponding to the C axis by a given factor c = 16, splitting it into NCcHW format. Then use the buffer reorder primitive to transform the loop order from NCcHW to NCHWc.

[0116] Step 3: Align the last dimension of the input tensor buffer in the NPU storage module by a given factor c = 16. Specifically, the storage alignment primitive storage_align can be used to align the last dimension of the tensor buffer after the first and second steps, i.e., the c dimension, by 16.

[0117] Furthermore, the above step 2042 includes:

[0118] Step 20421, with the goal of achieving the matrix multiplication calculation corresponding to the optimal parameter configuration in the current NPU, fill the output channel and input channel of the tir layer convolution weight according to the weight distribution requirements represented by the hardware attribute information of the current NPU;

[0119] Step 20422: Perform corresponding deformation and transposition operations on the padded convolution weights.

[0120] like Figure 8 As shown, it is an exemplary convolution weight rearrangement flowchart provided in an embodiment of the present application. Figure 8 The output channel para unit co_para_unit is 1 / 2 / 3, and the embodiment of the present application is set to 1, beta_unit is 8 / 16, if the input channel, output feature and parallel calculation number of the output channel of the NPU computing unit of the current NPU are all 16, then the embodiment of the present application adapts the hardware attribute information to set beta_unit to 16, num_group represents the group parameter in the convolution, num_co_group represents the number of output channels of the convolution weight, and num_ci_group represents the number of input channels of the convolution weight. Figure 8 For further explanation:

[0121] Step 1: According to the optimal parameter configuration of the current NPU determined in the above embodiment, the output channel co and input channel ci of the tir layer convolution weight are filled, where add_co = co_para_unit*alpha-(num_co_group%(co_para_unit*alpha)). If the number of parallel calculations of the input channels of the NPU computing unit of the current NPU is 16, it is determined that the current weight distribution requirement is that the tir layer input channel ci satisfies 16 alignment. If not, filling is required, that is, add_ci = 16-(num_ci_group%16);

[0122] Step 2: Press Figure 8 The parameters shown reshape the padded convolution weights.

[0123] Wherein, new_num_co_group = new_num_co*co_para_unit*alpha, new_num_ci_group = new_num_ci*beta_unit. new_num_co represents the number of output channels after output channel filling, and new_num_ci represents the number of input channels after input channel filling.

[0124] Step 3: Transpose the convolution weights transformed in step 2 to obtain the final data layout of the convolution weights. Figure 8 The relationship shown by the arrow is transformed. In actual programming, this transformation operation can be combined with step 2.

[0125] It is worth noting that the delta, epsilon, co_para_alpha and beta_unit in the final result of the convolution weight may have bubbles (filled with 0). The bubbles in co_para_alpha and beta_unit come from add_co and add_ci in step 1 respectively. When the last delta, last_delta, and the last epsilon, last_epsilon, are not equal to delta and epsilon respectively, the last delta and epsilon need to be filled.

[0126] Furthermore, in one embodiment, modifying the parameters of the relay layer includes:

[0127] Step 2043, obtaining the adjustment information of the convolution weight of the tir layer;

[0128] Step 2044: According to the adjustment information of the convolution weight of the TIR layer, the convolution weight of the relay layer is adjusted accordingly to make the relay layer match the TIR layer.

[0129] Among them, the adjustment information of the convolution weight of the TIR layer is the convolution weight adjustment record performed on the TIR layer in the above embodiment.

[0130] Specifically, a relay optimization mechanism can be added to TVM and named PostScheduleArgumentRewrite pass. A corresponding Python-side function class, PostScheduleArgumentRewriteManager, is also added. During the TIR function scheduling phase, PostScheduleArgumentRewriteManager is responsible for tracking changes in the convolution weight data layout of the input parameters, namely, the adjustment information of the convolution weights of the TIR layer, and recording the equivalent relay transformations. After the TIR function is scheduled, it is responsible for recording the relay transformations in the TIR function attribute attr. The PostScheduleArgumentRewrite pass reads the attribute attr from the TIR function scheduling result and applies it to the parameter rewriting of the relay layer. This allows for the subsequent return of a valid relay graph. The relay graph that matches the TIR layer after weight reordering is the valid relay graph described above.

[0131] To prevent duplicate scheduling errors of the TIR function, you can add a global cache for TIR function scheduling in the TVM compilation engine. The PostScheduleArgumentRewrite pass and TVM compilation will reuse this global cache.

[0132] Furthermore, after the above steps are performed to transform the input feature data layout and rearrange the weights offline, the data arrangement after being transferred to the input feature buffer IBUF and the convolution weight buffer WBUF through the storage module DM is as follows: Figure 9 As shown on the left and right, Figure 9 The data arrangement diagrams in the input feature buffer IBUF and the convolution weight buffer WBUF provided in the embodiments of the present application are both data arrangement diagrams in addr0, ci0_d0 represents the first pixel in the first input channel of the input feature, co0_ci0_k0 represents the first data in the kernel size in the first input channel of the first output channel in the convolution weight, and if k_size = kh*kw = 9, the index of k is 0-8.

[0133] It should be noted that the two requirements of the TVM software design are completed in the TVM schedule phase. In the TVM relay compilation phase, the subgraph of each relay layer's fused operator is converted into a TIR function and supplied to the schedule logic. The schedule logic is responsible for returning the optimized TIR function to the relay compilation.

[0134] Specifically, in one embodiment, the tir function corresponding to the convolution operator can be compiled based on the target TVM to generate a deployment file of the neural network model suitable for the current NPU.

[0135] Furthermore, the deployment file of the neural network model applicable to the current NPU can be stored in the current NPU, so that the neural network model can be deployed to the current NPU using the deployment file.

[0136] Based on the above embodiment, as an implementable manner, in one embodiment, before compiling the TIR function corresponding to the convolution operator based on the target TVM, the method includes:

[0137] Step 301: Based on the target TVM and the computational logic of the neural network model, a fusion relay graph is created.

[0138] Step 302: Locate the relay fusion operator subgraph corresponding to the convolution operator from the fusion relay graph;

[0139] Step 303: Determine the TIR function corresponding to the convolution operator according to the relay fusion operator subgraph.

[0140] The fused relay graph represents the computational logic of the neural network model within the TVM architecture. The specific creation process of the fused relay graph can be referenced in the prior art and is not limited in this embodiment. The current fused relay graph includes a relay fusion operator subgraph for the convolution operator and relay fusion operator subgraphs for other operators.

[0141] Specifically, the relay fusion operator subgraphs of each convolution operator can be extracted from the fusion relay graph, and then these relay fusion operator subgraphs are converted into the form of TIR functions (tirPrimFunc). These TIR functions are then scheduled based on the TVM architecture to enter the TIR function scheduling stage of TVM.

[0142] Specifically, the scheduled TIR function can be recorded in the global cache of TVM. During relay compilation, the scheduled TIR function can be directly obtained from the global cache to obtain the TIR function after the input data layout transformation and convolution weight rewriting, so as to achieve the purpose of matching TIR and relay graphs.

[0143] Specifically, through the above transformation and adaptation of the TIR function, that is, the TIR function in the global cache, compiler compilation and simulator operation can be performed, and a deployment file suitable for the current NPU of the neural network model can be generated, thereby achieving the purpose of development and application.

[0144] The TVM-based NPU neural network model deployment method and device provided in the embodiment of the present application obtains the hardware attribute information of the neural network model to be deployed and the current NPU; determines the convolution operator to be deployed to the current NPU according to the calculation logic of the neural network model; determines the optimal parameter configuration of each convolution operator in the current NPU according to the hardware attribute information of the current NPU and the parameter information of each convolution operator; adjusts the data layout and convolution weight of the TIR layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, and modifies the parameters of the relay layer to obtain a target TVM adapted to the current NPU; based on the target TVM, generates a deployment file for the neural network model suitable for the current NPU, so as to use the deployment file to deploy the neural network model to the current NPU. The method provided by the above scheme determines the optimal parameter configuration suitable for the NPU to implement convolution calculation, and further adjusts the parameters of the convolution operators of the TIR layer and relay layer of the TVM in combination with the hardware attribute information and optimal parameter configuration of the NPU, so that the obtained target TVM can adapt to the NPU.

[0145] An embodiment of the present application provides a TVM-based NPU neural network model deployment device, which is used to execute the TVM-based NPU neural network model deployment method provided in the above embodiment.

[0146] like Figure 10 FIG2 is a schematic diagram of the structure of the TVM-based NPU neural network model deployment device provided in an embodiment of the present application. The TVM-based NPU neural network model deployment device 100 includes: an acquisition module 1001, an analysis module 1002, a determination module 1003, an adjustment module 1004, and a deployment module 1005.

[0147] Among them, the acquisition module is used to obtain the neural network model to be deployed and the hardware attribute information of the current NPU; the analysis module is used to determine the convolution operator to be deployed to the current NPU according to the calculation logic of the neural network model; the determination module is used to determine the optimal parameter configuration of each convolution operator in the current NPU according to the hardware attribute information of the current NPU and the parameter information of each convolution operator; the adjustment module is used to adjust the data layout and convolution weight of the tir layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, and modify the parameters of the relay layer to obtain the target TVM adapted to the current NPU; the deployment module is used to generate a deployment file of the neural network model suitable for the current NPU based on the target TVM, so as to use the deployment file to deploy the neural network model to the current NPU.

[0148] Specifically, in one embodiment, the determination module is specifically configured to:

[0149] Based on the preset performance evaluation model, the optimal parameter configuration of each convolution operator in the current NPU is determined according to the hardware constraints of the current NPU, the hardware attribute information of the current NPU, and the parameter information of each convolution operator.

[0150] Specifically, in one embodiment, the adjustment module is specifically configured to:

[0151] Based on the data input requirements represented by the current NPU hardware attribute information, the data layout of the tensor buffer of the TVM storage layer is modified to adjust the data distribution of the TIR layer input features. The tensor buffer is used to cache the TIR layer input features.

[0152] With the goal of achieving the matrix multiplication calculation corresponding to the optimal parameter configuration in the current NPU, the convolution weights of the tir layer are adjusted according to the weight distribution requirements represented by the hardware attribute information of the current NPU.

[0153] Specifically, in one embodiment, the adjustment module is specifically configured to:

[0154] Get the adjustment information of the convolution weight of the tir layer;

[0155] According to the adjustment information of the convolution weight of the TIR layer, the convolution weight of the relay layer is adjusted accordingly to make the relay layer match the TIR layer.

[0156] Specifically, in one embodiment, the deployment module is specifically configured to:

[0157] Based on the target TVM, compile the tir function corresponding to the convolution operator to generate a deployment file for the neural network model suitable for the current NPU.

[0158] Specifically, in one embodiment, the device further includes:

[0159] The data management module is used to create a fused relay graph based on the target TVM and the computational logic of the neural network model; locate the relay fusion operator subgraph corresponding to the convolution operator from the fused relay graph; and determine the TIR function corresponding to the convolution operator based on the relay fusion operator subgraph.

[0160] Specifically, in one embodiment, the hardware attribute information of the NPU includes input channels, output features, and the number of parallel computing of the output channels of the NPU computing unit.

[0161] Regarding the TVM-based NPU neural network model deployment device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiments of the method and will not be elaborated here.

[0162] The TVM-based NPU neural network model deployment device provided in the embodiment of the present application is used to execute the TVM-based NPU neural network model deployment method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0163] An embodiment of the present application provides an electronic device for executing the TVM-based NPU neural network model deployment method provided in the above embodiment.

[0164] like Figure 11 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 110 includes at least one processor 1101 and a memory 1102 .

[0165] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the TVM-based NPU neural network model deployment method provided in the above embodiment.

[0166] An electronic device provided in an embodiment of the present application is used to execute the TVM-based NPU neural network model deployment method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0167] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the TVM-based NPU neural network model deployment method provided in any of the above embodiments is implemented.

[0168] The storage medium containing computer-executable instructions in the embodiment of the present application can be used to store computer-executable instructions of the TVM-based NPU neural network model deployment method provided in the aforementioned embodiment. Its implementation method and principle are the same and will not be repeated here.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0172] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0173] Those skilled in the art will clearly understand that for the sake of convenience and brevity, the division of the above-mentioned functional modules is only used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A TVM-based NPU neural network model deployment method, characterized in that: include: Obtain the neural network model to be deployed and the hardware attribute information of the current NPU; Determining a convolution operator to be deployed to the current NPU according to the computational logic of the neural network model; Determining, based on the hardware attribute information of the current NPU and the parameter information of each convolution operator, an optimal parameter configuration of each convolution operator in the current NPU; According to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, the data layout and convolution weight of the tir layer are adjusted, and the parameters of the relay layer are modified to obtain a target TVM adapted to the current NPU; Based on the target TVM, a deployment file of the neural network model suitable for the current NPU is generated, so as to deploy the neural network model to the current NPU using the deployment file.

2. The method according to claim 1, characterized in that The determining, based on the hardware attribute information of the current NPU and the parameter information of each convolution operator, the optimal parameter configuration of each convolution operator in the current NPU includes: Based on a preset performance evaluation model, the optimal parameter configuration of each convolution operator in the current NPU is determined according to the hardware constraints of the current NPU, the hardware attribute information of the current NPU and the parameter information of each convolution operator.

3. The method according to claim 1, characterized in that The adjusting the data layout and convolution weight of the TIR layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU includes: Modify the data layout of the tensor buffer of the TVM storage layer according to the data input requirements represented by the hardware attribute information of the current NPU to adjust the data distribution of the TIR layer input features; wherein the tensor buffer is used to cache the TIR layer input features; With the goal of achieving the matrix multiplication calculation corresponding to the optimal parameter configuration in the current NPU, the convolution weight of the tir layer is adjusted according to the weight distribution requirements represented by the hardware attribute information of the current NPU.

4. The method according to claim 1, wherein Modifying the parameters of the relay layer includes: Obtaining adjustment information of the convolution weights of the tir layer; According to the adjustment information of the convolution weight of the TIR layer, the convolution weight of the relay layer is adjusted accordingly to make the relay layer match the TIR layer.

5. The method according to claim 1, characterized in that The generating, based on the target TVM, a deployment file of the neural network model applicable to the current NPU includes: Based on the target TVM, the tir function corresponding to the convolution operator is compiled to generate a deployment file of the neural network model suitable for the current NPU.

6. The method according to claim 5, characterized in that Before compiling the tir function corresponding to the convolution operator based on the target TVM, the method further includes: Based on the target TVM and the computational logic of the neural network model, a fusion relay graph is created; Locating the relay fusion operator subgraph corresponding to the convolution operator from the fusion relay graph; According to the relay fusion operator subgraph, the TIR function corresponding to the convolution operator is determined.

7. The method according to claim 1, characterized in that The hardware attribute information of the NPU includes the input channels, output characteristics, and the number of parallel computing channels of the NPU computing unit.

8. A TVM-based NPU neural network model deployment device, characterized in that: include: The acquisition module is used to obtain the neural network model to be deployed and the hardware attribute information of the current NPU; An analysis module, configured to determine a convolution operator to be deployed to the current NPU based on the computational logic of the neural network model; A determination module, configured to determine an optimal parameter configuration of each convolution operator in the current NPU based on the hardware attribute information of the current NPU and the parameter information of each convolution operator; An adjustment module is configured to adjust the data layout and convolution weight of the tir layer according to the optimal parameter configuration of each convolution operator in the current NPU and the hardware attribute information of the current NPU, and modify the parameters of the relay layer to obtain a target TVM adapted to the current NPU; A deployment module is used to generate a deployment file of the neural network model suitable for the current NPU based on the target TVM, so as to deploy the neural network model to the current NPU using the deployment file.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

Images