A neural network inference optimization method for adapting hardware tensor instructions and memory

Abstract

The application discloses a neural network inference optimization method suitable for adapting hardware tensor instructions and memory, and belongs to the technical field of neural network inference optimization, and comprises the following steps: uniformly mapping convolution operations of different types of convolution layers into equivalent matrix multiplication to obtain an input feature map matrix, a convolution kernel matrix and an output feature map matrix; dividing the input feature map matrix into an input feature map sub-matrix and the convolution kernel matrix into a convolution kernel sub-matrix by using hardware instruction parameters; obtaining an output feature map sub-matrix based on the input feature map sub-matrix and the convolution kernel sub-matrix; comparing the total read amount of registers from a global buffer under different multiplexing modes to obtain an optimal multiplexing mode; and under the optimal multiplexing mode, using a software pipeline to slice DRAM to hide the scheduling time of data from DRAM to the global buffer. The application can not only improve the utilization rate of hardware computing resources, but also significantly improve the inference performance of a neural network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network inference optimization, and more particularly to a neural network inference optimization method suitable for adapting hardware tensor instructions and memory. BACKGROUND

[0002] At present, convolutional neural networks (CNN) have achieved great success in many practical applications of computer vision, such as image classification, object detection and video analysis, etc. With the improvement of algorithm accuracy, the most advanced CNN currently has millions of parameters and high-dimensional feature maps. In order to deploy such applications to embedded devices, many hardware manufacturers have designed special accelerators to efficiently process tensor calculations in CNN.

[0003] Current neural network inference accelerators usually use 2D processing element (PE) arrays and multi-level memory hierarchies for computation and storage, respectively, and complete convolution operations through tensor operation (such as matrix multiplication) instructions, such as CEVA AIDSP, NVIDIA's Tensor Core, etc. The data flow arrangement of the model determines whether the tensor is efficiently mapped to the PE array and scheduled between the multi-level memory hierarchy through the tensor instruction, which directly affects the resource utilization and data reuse rate of the accelerator. In order to maximize the performance of the accelerator, an effective way is to design a neural network data flow optimizer to adjust the model data flow to match the hardware tensor instruction and multi-level memory structure.

[0004] Data flow optimization includes accelerator mapping and tensor computation scheduling. Existing data flow mapping schemes such as CoSA obtain fine-grained PE-level data flow mapping by searching the mapping space, and the work is based on known hardware implementation details such as PE connection mode. Therefore, for accelerators that only disclose instructions, it is difficult to meet the data flow mapping relationship.

[0005] For the multi-level memory hierarchy of the accelerator, the purpose of data flow scheduling is to maximize the reuse of data (input, output and weight) in the buffer, thereby reducing the memory access time, and selecting the size and execution order of the slice in different memory levels through loop transformation techniques (such as slicing, rearrangement). SmartShuttle calculates different data reuse methods according to different convolution layer sizes; Ansor uses a cost model to search for the optimal scheduling scheme from the scheduling space, and since the scheduling scheme does not match the tensor instruction, it often leads to suboptimal performance or too long search time.

[0006] Therefore, how to provide a neural network model inference optimization method which can not only improve the utilization of hardware computing resources, but also significantly improve the inference performance of the neural network model is a problem that those skilled in the art need to solve. SUMMARY

[0007] In view of this, the purpose of this invention is to provide a neural network inference optimization method that adapts to hardware tensor instructions and memory.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A neural network inference optimization method adapted to hardware tensor instructions and memory includes the following steps:

[0010] The convolution operations of different types of convolutional layers are uniformly mapped to equivalent matrix multiplication to obtain the input feature map matrix, convolution kernel matrix, and output feature map matrix;

[0011] The input feature map matrix is ​​divided into an input feature map submatrix and the convolution kernel matrix is ​​divided into a convolution kernel submatrix using hardware instruction parameters; the output feature map submatrix is ​​obtained based on the input feature map submatrix and the convolution kernel submatrix.

[0012] By comparing the total amount of registers read from the global buffer under different multiplexing methods, the optimal multiplexing method can be obtained.

[0013] In the optimal multiplexing method, DRAM is sliced ​​using a software pipeline to hide the scheduling time of data from DRAM to the global buffer.

[0014] Preferably, the input feature map is expanded to obtain the input feature map matrix;

[0015] The input feature map matrix has a total of H I *W I Columns, each column has a size of C I *K*K;

[0016] The convolution kernel is unfolded to obtain the convolution kernel matrix;

[0017] The convolution kernel matrix has a total of C O Rows, each row has a size of C I *K*K;

[0018] The output feature map is expanded to obtain the output feature map matrix;

[0019] The output feature map matrix has a total of C O Rows, each row has a size of H O *W O ;

[0020] Among them, C I For the number of channels in the input feature map, C O H is the number of channels in the output feature map. I For the input feature map height, W I H is the width of the input feature map. OHout is the height of the output feature map, Wout is the width of the output feature map, and K is the size of the convolution kernel. O Hout is the height of the output feature map, Wout is the width of the output feature map, and K is the size of the convolution kernel.

[0021] Preferably, the size of the input feature map sub-matrix is xy.

[0022] The size of the convolution kernel sub-matrix is yz.

[0023] The size of the output feature map sub-matrix is xz.

[0024] wherein x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at one time; y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at one time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at one time; and z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at one time.

[0025] Preferably, the memory arrangement of the output tensor in the output feature map matrix is initialized as wherein N represents the batch of processing images.

[0026] Preferably, the optimal multiplexing mode is obtained based on the following formula:

[0027] V min = minimize(V InR ,V OutR ,V WtR );

[0028] wherein V InR represents the total read amount of the register from the global buffer under the input multiplexing mode; V OutR represents the total read amount of the register from the global buffer under the output multiplexing mode; V WtR represents the total read amount of the register from the global buffer under the convolution kernel multiplexing mode; and V min represents the minimum value of V InR , V OutR , and V WtR , and the corresponding multiplexing mode is the optimal multiplexing mode. min

[0029] Preferably,

[0030]

[0031] wherein R in , R wt , and R out represent the slice size of the input feature map, the output feature map, and the convolution kernel read by the register from the global buffer, respectively. represents the read frequency.

[0032] ​Preferably,

[0033] wherein C I is the input feature map channel number, C O is the output feature map channel number, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, and K is the convolution kernel size; x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at a time; y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at a time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at a time; and z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at a time.

[0034] Preferably, R in = xy; R wt = yz; and R out = xz.

[0035] Preferably, the size of the DRAM slice satisfies the following formula:

[0036]

[0037] wherein transfer cost(S) is the DRAM-to-global buffer transmission time consumption; compute cost(S) is the register calculation time consumption under the optimal multiplexing mode; S in is the input slice size in the DRAM; S out is the output slice size in the DRAM; S wt is the convolution kernel slice size in the DRAM; S is the total slice size in the DRAM; and GBuf is the global buffer size.

[0038] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the hardware resource-aware neural network model inference optimization method described above when executing the computer program.

[0039] Via the technical solution described above, compared with the prior art, the present disclosure provides a neural network inference optimization method that adapts to hardware tensor instructions and memory, which not only improves the utilization rate of hardware computing resources, but also significantly improves the inference performance of the neural network model. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only aim to explain the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on the provided drawings.

[0041] Figure 1 A neural network inference optimization method flowchart for adapting hardware tensor instructions and memory is provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the mapping and sub-matrix partitioning method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the inner loop scheduling method corresponding to different multiplexing modes provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.

[0046] As shown in Figure 1 , the present application discloses a neural network inference optimization method for adapting hardware tensor instructions and memory, comprising the following steps:

[0047] 1) mapping the convolution operations of different types of convolution layers into equivalent matrix multiplication to obtain an input feature map matrix, a convolution kernel matrix and an output feature map matrix;

[0048] In an embodiment, as shown in Figure 2 , the input feature map is unfolded to obtain the input feature map matrix;

[0049] The input feature map matrix has H I *W I columns, and each column has a size of C I *K*K;

[0050] The convolution kernel is unfolded to obtain the convolution kernel matrix;

[0051] The convolution kernel matrix has C Oeach row has a size of C I *K*K

[0052] unfolding the output feature map to obtain the output feature map matrix;

[0053] the output feature map matrix has a size of C O each row has a size of H O *W O ;

[0054] wherein C I is the input feature map channel number, C O is the output feature map channel number, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, and K is the convolution kernel size.

[0055] It can be understood that the neural network comprises a plurality of layers, each layer being composed of an input feature map, an output feature map and a convolution kernel, and the output feature map of the current layer being the input feature map of the next layer; the application firstly constructs a unified mapping for different convolution layers, and maps the convolution operation into equivalent matrix multiplication. The convolution kernel, the input feature map and the output feature map are actual data storage forms. Correspondingly, the convolution kernel matrix, the input feature map matrix and the output feature map matrix are data forms after mapping.

[0056] 2) dividing the input feature map matrix into an input feature map sub-matrix and the convolution kernel matrix into a convolution kernel sub-matrix by using a hardware instruction parameter;

[0057] obtaining an output feature map sub-matrix based on the input feature map sub-matrix and the convolution kernel sub-matrix (the input feature map sub-matrix and the convolution kernel sub-matrix are multiplied to obtain the output feature map sub-matrix);

[0058] In a certain embodiment, as shown in Figure 2 :

[0059] the size of the input feature map sub-matrix is xy;

[0060] the size of the convolution kernel sub-matrix is yz;

[0061] the size of the output feature map sub-matrix is xz;

[0062] wherein x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at a time; y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at a time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at a time; and z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at a time.

[0063] It can be understood that: the application divides the input feature map matrix and the convolution kernel matrix required by matrix multiplication into matrices required to be called once by a tensor instruction (corresponding to Figure 2 the light gray shaded part: A sub-mapping and B sub-mapping), and finally obtains an output feature map sub-matrix (corresponding to Figure 2 the light gray shaded part: C sub-mapping). Specifically, the matrix multiplication of xy and yz is performed, and only a fixed size of input tensor xy and yz can be loaded once, and a fixed size of output tensor xz is obtained after calculation. The hardware instruction parameters (xy, yz) are taken as the basic mapping unit, that is, the sub-matrix is mapped in the input feature map matrix and the convolution kernel matrix, and the sub-matrix is mapped to the accelerator and iterative calculation is performed.

[0064] In an embodiment, the memory arrangement of the output tensor in the output feature map matrix is initialized as wherein N represents a batch of processing images, and N is 1 in this embodiment.

[0065] It can be understood that, as shown in Figure 2 , C in the output feature map matrix is a mapping in the output feature map, and therefore the memory arrangement of the output tensor needs to be adjusted in the output feature map, such as the memory arrangement It should be noted that the memory arrangement of the output tensor needs to refer to the tensor instruction parameters of the subsequent operator, so as to reduce the data loading time when the next operator is calculated, so that the accelerator can continuously read data from the buffer without additional conversion, and the calculation resources of the accelerator are maximized.

[0066] 3) Compare the total read amount of the register from the global buffer under different multiplexing modes to obtain the optimal multiplexing mode;

[0067] In an embodiment, the optimal multiplexing mode is obtained based on the following formula:

[0068] V min = minimize(V InR ,V OutR ,V WtR );

[0069]

[0070]

[0071] wherein V InR represents the total read amount of the register from the global buffer under the input multiplexing mode; V OutR represents the total read amount of the register from the global buffer under the output multiplexing mode; V WtR represents the total read amount of the register from the global buffer under the convolution kernel multiplexing mode; and V min represents taking VInR ,V OutR ,V WtR the minimum value in V min The corresponding multiplexing mode is the optimal multiplexing mode; R in , R wt , R out in turn represent the slice size of the input feature map, the output feature map and the convolution kernel read by the register from the global buffer; represents the number of reading times; C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the height of the input feature map, W I is the width of the input feature map, H O is the height of the output feature map, W O is the width of the output feature map, and K is the size of the convolution kernel; x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at a time; y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at a time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at a time; and z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at a time.

[0072] In an embodiment, R in = xy; R wt = yz; and R out = xz.

[0073] Since the tensor instruction can only load xy-sized input, yz-sized weight and xz-sized partial sum at a time, the extra data transmission is obviously unnecessary, so the slice size is taken as the basic unit of the inner loop (global buffer to register) scheduling, that is, R in = xy, R wt = yz, and R out = xz. At this time, the global buffer reading amount is only related to the multiplexing mode.

[0074] It can be understood that there are three data multiplexing modes in convolution operation, including input multiplexing (InR, one input is used by multiple convolution kernels), output reuse (OutR, partial sum is reused) and convolution kernel multiplexing (WtR, one convolution kernel is used by multiple inputs); if the multiplexing mode corresponds to each slice of the data type which is read only once by the register, it is the minimum global buffer reading amount in the multiplexing mode, and the slice of the multiplexed data type needs to be saved in the register for multiplexing until all the calculations in which it participates are completed, and other types of data are scheduled to be calculated with it. For example, Figure 3As shown, for input multiplexing, the convolution kernel slices of each output channel need to be loaded in sequence, for output multiplexing, the feature map and convolution kernel slices of each input channel need to be loaded in sequence, and for convolution kernel multiplexing, the feature map slices of the feature map length and width need to be loaded in sequence. Figure 3 Taking output multiplexing in the above formula as an example, the feature map slices and the convolution kernel slices are loaded from the DRAM to the global buffer in sequence according to the input channels, and the accelerator loads the feature map slices of xy size and the convolution kernel slices of yz size from the global buffer to the register to perform tensor operation each time, and the partial sum and xz are always saved in the register for multiplexing, without additional global buffer access.

[0075] Further, after confirming the optimal multiplexing mode, the memory arrangement of the output tensors in the output feature map matrix can be automatically updated based on the optimal multiplexing mode, and the scheduling order of the inner loop (global buffer to register) is obtained.

[0076] 4) Under the optimal multiplexing mode, the DRAM slices are used to hide the scheduling time of data from the DRAM to the global buffer by using software pipelining.

[0077] In an embodiment, the size of the DRAM slice satisfies the following formula:

[0078]

[0079] Where, transfer cost(S) is the transmission time from the DRAM to the global buffer; compute cost(S) is the calculation time of the register under the optimal multiplexing mode; S in is the input slice size in the DRAM; S out is the output slice size in the DRAM; S wt is the convolution kernel slice size in the DRAM; S is the total slice size in the DRAM; and GBuf is the global buffer size.

[0080] It can be understood that the software pipeline of the present application uses double buffering to achieve loading, calculation and storage of slices, and overlaps calculation and DRAM access. The scheduling mode of the outer loop (DRAM to global buffer) of the present application does not affect the memory access time of the neural network model, i.e. the total memory access time is only the access time of the global buffer to the register.

[0081] In order to verify the effectiveness of the reasoning optimization method of the present application, the mapping method of the present application and the method of direct mapping without optimization were compared in performance on three models of MobileNet, ResNet and GoogleNet, as shown in Table 1.

[0082] Table 1 Comparison of reasoning delay time under different mapping methods (unit: ms)

[0083] Base Ours MobileNet 34.3 17.6 ResNet18 52.2 22 GoogleNet 153.4 61.2

[0084] As shown in Table 1, the mapping method of the present invention achieves a geometric mean speedup of 2.3 times compared with the unoptimized direct mapping method.

[0085] This invention was also compared with different scheduling methods, as shown in Table 2:

[0086] Table 2 - Comparison of inference latency under different scheduling methods (unit: ms)

[0087] Ours CEVA IfR WtR MobileNet 10.8 11 15.3 15.6 ResNet18 15.1 24 17.4 38.8 GoogleNet 41.5 67 47.6 93.1

[0088] As shown in Table 2, the scheduling method of the present invention achieves a speedup of 1.37 times the geometric mean compared with other scheduling methods.

[0089] This invention also discloses a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions stored in the memory 403 to execute the aforementioned hardware resource-aware neural network model inference optimization method.

[0090] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] In another aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being capable of performing the hardware resource-aware neural network model inference optimization method described above when executed by a processor.

[0092] The various embodiments described in the specification are presented for the purpose of illustration and description. Each of the embodiments highlights a certain aspect of the application. The embodiments are not mutually exclusive, and can be combined with each other. The same or similar parts of the embodiments are mutually referred to each other.

[0093] The above description of disclosed embodiments provides enabling disclosure necessary to make or use the application. Numerous modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A neural network inference optimization method for adapting hardware tensor instructions and memory, the method comprising: The method comprises the following steps: The convolution operations of different types of convolution layers are uniformly mapped to equivalent matrix multiplications to obtain an input feature map matrix, a convolution kernel matrix and an output feature map matrix; The input feature map is unfolded to obtain the input feature map matrix; input feature map matrix, H I *W I columns, each column having a size of C I *K*K; The convolution kernel is unfolded to obtain the convolution kernel matrix; C kernels O C rows, each of size C I *K*K; The output feature map is unfolded to obtain the output feature map matrix; The output feature map matrix has C O rows, and each row has a size of H O *W O ; wherein C I is the input feature map channel number, C O is the output feature map channel number, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, and K is the convolution kernel size. The input feature map matrix is divided into input feature map sub-matrices by using a hardware instruction parameter, the convolution kernel matrix is divided into convolution kernel sub-matrices, and output feature map sub-matrices are obtained based on the input feature map sub-matrices and the convolution kernel sub-matrices; The optimal multiplexing mode is obtained by comparing the total read amount of the register from the global buffer under different multiplexing modes; The optimal multiplexing mode is obtained based on the following formula: V min = minimize (V InR , V OutR , V WtR ); wherein, V InR represents the total read amount of the register from the global buffer under the input multiplexing mode; V OutR represents the total read amount of the register from the global buffer under the output multiplexing mode; V WtR represents the total read amount of the register from the global buffer under the convolution kernel multiplexing mode; V min represents taking the minimum value of V InR ,V OutR ,V WtR in the minimum value, V min The corresponding multiplexing mode is the optimal multiplexing mode. wherein R in , R wt , R out represent the slice size of the input feature map, the output feature map and the convolution kernel read by the register from the global buffer in sequence; represents the reading times; wherein C I is the input feature map channel number, C O is the output feature map channel number, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, K is the convolution kernel size; x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at a time; y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at a time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at a time; z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at a time. Under the optimal multiplexing mode, the DRAM slice is cut by using a software pipeline to hide the scheduling time of data from the DRAM to the global buffer.

2. The neural network inference optimization method for adapting hardware tensor instructions and memory according to claim 1, characterized in that: The size of the input feature map sub-matrix is xy; The size of the convolution kernel sub-matrix is yz; The size of the output feature map sub-matrix is xz; wherein x represents the minimum matrix width obtained by the tensor instruction from the input feature map matrix at a time, y represents the minimum matrix height obtained by the tensor instruction from the input feature map matrix at a time and the minimum matrix width obtained by the tensor instruction from the convolution kernel matrix at a time, and z represents the minimum matrix height obtained by the tensor instruction from the convolution kernel matrix at a time.

3. The neural network inference optimization method of claim 2, wherein, initializing a memory arrangement of output tensors in the output feature map matrix as where N represents a batch of processing images.

4. The method of claim 1, wherein the method further comprises: determining a hardware tensor instruction for each of the plurality of operations; and determining a memory layout for each of the plurality of operations. R in = xy; R wt = yz; R out = xz.

5. The method of claim 1, wherein the method further comprises: determining a hardware tensor instruction for the tensor operation; and determining a memory access pattern for the tensor operation. The size of the DRAM slice satisfies the following formula: where transfer cost(S) is the time cost of DRAM to global buffer transfer; compute cost(S) is the time cost of register computation under optimal multiplexing; S in is the input slice size in DRAM; S out is the output slice size in DRAM; S wt is the convolution kernel slice size in DRAM; S is the total slice size in DRAM; GBuf is the global buffer size.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the hardware resource-aware neural network model inference optimization method according to any one of claims 1 to 5 when executing the computer program.

Images