Neural network reasoning optimization method adaptive to hardware tensor instruction and memory

Abstract

The invention discloses a neural network reasoning optimization method adaptive to a hardware tensor instruction and a memory, and belongs to the technical field of neural network reasoning optimization, and the method comprises the following steps: uniformly mapping convolution operations of different types of convolution layers into equivalent matrix multiplication, and obtaining an input feature map matrix, a convolution kernel matrix and an output feature map matrix; dividing the input feature map matrix into input feature map sub-matrixes by using hardware instruction parameters, and dividing the convolution kernel matrix into convolution kernel sub-matrixes; obtaining an output feature map sub-matrix based on the input feature map sub-matrix and the convolution kernel sub-matrix; comparing the total reading quantity of the register from the global buffer area under different multiplexing modes to obtain an optimal multiplexing mode; and in the optimal multiplexing mode, slicing the DRAM by using a software assembly line so as to hide the scheduling time of data from the DRAM to a global buffer area. According to the method, the hardware computing resource utilization rate can be improved, and the reasoning performance of the neural network model can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network reasoning optimization, and more specifically to a neural network reasoning optimization method adapted to hardware tensor instructions and memory. Background Art

[0002] At present, convolutional neural networks (CNNs) have achieved great success in many practical applications of computer vision, such as image classification, object detection, and video analysis. With the improvement of algorithm accuracy, the most advanced CNNs currently have millions of parameters and high-dimensional feature maps. In order to deploy such applications to embedded devices, many hardware manufacturers have designed special accelerators to efficiently process tensor calculations in CNNs.

[0003] Current neural network inference accelerators usually use 2D processing element (PE) arrays and multi-level memory hierarchies for computing and storage, respectively, and complete convolution operations through tensor operations (such as matrix multiplication) instructions, such as CEVA AIDSP and NVIDIA's Tensor Core. The data flow arrangement of the model determines whether the tensor is mapped to the PE array through tensor instructions and whether the scheduling between multi-level memory hierarchies is efficient, which directly affects the resource utilization and data reuse rate of the accelerator. In order to maximize the performance of the accelerator, an effective way is to design a neural network data flow optimizer to adjust the model data flow to match the hardware tensor instructions and multi-layer memory structure.

[0004] Data flow optimization includes accelerator mapping and tensor computation scheduling. Existing data flow mapping solutions such as CoSA obtain fine-grained PE-level data flow mapping by searching the mapping space. The work is based on known hardware implementation details such as PE connection methods. Therefore, for accelerators that only disclose instructions, it is difficult to meet the data flow mapping relationship.

[0005] For the multi-level memory hierarchy of the accelerator, the purpose of data flow scheduling is to maximize the reuse of data (input, output, and weight) in the buffer, thereby reducing memory access time, and selecting the size and execution order of slices in different memory levels through loop transformation techniques (such as segmentation and rearrangement). SmartShuttle calculates different data reuse methods based on different convolutional layer sizes; Ansor uses a cost model to search for the optimal scheduling solution from the scheduling space. Since the scheduling solution does not match the tensor instructions, it often leads to suboptimal performance or long search time.

[0006] Therefore, how to provide a neural network model reasoning optimization method, which can not only improve the utilization rate of hardware computing resources, but also significantly improve the reasoning performance of the neural network model is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a neural network reasoning optimization method that is adapted to hardware tensor instructions and memory.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A neural network reasoning optimization method adapted to hardware tensor instructions and memory includes the following steps:

[0010] The convolution operations of different types of convolutional layers are uniformly mapped into equivalent matrix multiplications to obtain the input feature map matrix, convolution kernel matrix, and output feature map matrix;

[0011] The input feature map matrix is ​​divided into input feature map sub-matrices and the convolution kernel matrix is ​​divided into convolution kernel sub-matrices by using hardware instruction parameters; based on the input feature map sub-matrix and the convolution kernel sub-matrix, an output feature map sub-matrix is ​​obtained;

[0012] Compare the total amount of registers read from the global buffer under different multiplexing modes to obtain the optimal multiplexing mode;

[0013] In the optimal multiplexing mode, the DRAM is sliced ​​using software pipelines to hide the scheduling time of data from the DRAM to the global buffer.

[0014] Preferably, the input feature map is expanded to obtain the input feature map matrix;

[0015] The input feature map matrix is ​​H I *W I Columns, each column has a size of C I *K*K;

[0016] Expand the convolution kernel to obtain the convolution kernel matrix;

[0017] The convolution kernel matrix is ​​C O rows, each row has a size of C I *K*K;

[0018] Expand the output feature map to obtain the output feature map matrix;

[0019] The output feature map matrix is ​​C O rows, each row has a size of H O *W O ;

[0020] Among them, C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H Ois the output feature map height, W O is the output feature map width, and K is the convolution kernel size.

[0021] Preferably, the size of the input feature map submatrix is ​​xy;

[0022] The size of the convolution kernel matrix is ​​yz;

[0023] The size of the output feature map submatrix is ​​xz;

[0024] Among them, x represents the minimum matrix width that the tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that the tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that the tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that the tensor instruction obtains from the convolution kernel matrix at one time.

[0025] Preferably, the memory arrangement of the output tensor in the output feature map matrix is ​​initialized as Here, N represents the batch of processed images.

[0026] Preferably, the optimal multiplexing mode is obtained based on the following formula:

[0027] V min =minimize(V InR ,V OutR ,V WtR );

[0028] Among them, V InR Indicates the total amount of registers read from the global buffer in input multiplexing mode; V OutR Indicates the total amount of registers read from the global buffer in output multiplexing mode; V WtR Indicates the total amount of registers read from the global buffer in the convolution kernel multiplexing mode; V min Indicates taking V InR ,V OutR ,V WtR The minimum value in V min The corresponding multiplexing mode is the optimal multiplexing mode.

[0029] Preferably,

[0030]

[0031] Among them, R in , R wt , R out It represents the input feature map, output feature map, and slice size of the convolution kernel read by the register from the global buffer respectively; Indicates the number of reads.

[0032] Preferably,

[0033] Among them, C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, K is the convolution kernel size; x represents the minimum matrix width that a tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that a tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that a tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that a tensor instruction obtains from the convolution kernel matrix at one time.

[0034] Preferably, R in =xy; R wt =yz; R out =xz.

[0035] Preferably, the size of the DRAM slice satisfies the following formula:

[0036]

[0037] Among them, transfer cost(S) The time taken for the transfer from DRAM to the global buffer; compute cost(S) is the calculation time of the register under the optimal multiplexing mode; S in S is the slice size input in DRAM; out S is the output slice size in DRAM; wt is the convolution kernel slice size in DRAM; S is the total slice size in DRAM; GBuf is the global buffer size.

[0038] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned hardware resource-aware neural network model inference optimization method is implemented.

[0039] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a neural network reasoning optimization method adapted to hardware tensor instructions and memory, which can not only improve the utilization rate of hardware computing resources, but also significantly improve the reasoning performance of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flowchart of a neural network reasoning optimization method for adapting hardware tensor instructions and memory is provided in a certain embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a mapping and sub-matrix partitioning method provided in an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of an inner loop scheduling method corresponding to different multiplexing modes provided in a certain embodiment of the present invention;

[0044] Figure 4 The figure is a schematic diagram of an electronic device provided in a certain embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, the embodiment of the present invention discloses a neural network reasoning optimization method adapted to hardware tensor instructions and memory, comprising the following steps:

[0047] 1) The convolution operations of different types of convolutional layers are uniformly mapped into equivalent matrix multiplications to obtain the input feature map matrix, convolution kernel matrix, and output feature map matrix;

[0048] In one embodiment, if Figure 2 As shown, the input feature map is expanded to obtain the input feature map matrix;

[0049] The input feature map matrix is ​​H I *W I Columns, each column has a size of C I *K*K;

[0050] Expand the convolution kernel to obtain the convolution kernel matrix;

[0051] The convolution kernel matrix is ​​C Orows, each row has a size of C I *K*K;

[0052] Expand the output feature map to obtain the output feature map matrix;

[0053] The output feature map matrix is ​​C O rows, each row has a size of H O *W O ;

[0054] Among them, C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, and K is the convolution kernel size.

[0055] It can be understood that: the neural network contains many layers, each layer is composed of an input feature map, an output feature map, and a convolution kernel, and the output feature map of the current layer is the input feature map of the next layer; the present invention first constructs a unified mapping for different convolution layers, and maps the convolution operation to an equivalent matrix multiplication. The convolution kernel, the input feature map, and the output feature map are the actual storage forms of the data. Correspondingly, the convolution kernel matrix, the input feature map matrix, and the output feature map matrix are the data forms after mapping.

[0056] 2) using hardware instruction parameters to divide the input feature map matrix into input feature map sub-matrices and the convolution kernel matrix into convolution kernel sub-matrices;

[0057] Based on the input feature map submatrix and the convolution kernel submatrix, the output feature map submatrix is ​​obtained (the input feature map submatrix and the convolution kernel submatrix are multiplied to obtain the output feature map submatrix);

[0058] In one embodiment, if Figure 2 As shown:

[0059] The size of the input feature map submatrix is ​​xy;

[0060] The size of the convolution kernel matrix is ​​yz;

[0061] The size of the output feature map submatrix is ​​xz;

[0062] Among them, x represents the minimum matrix width that the tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that the tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that the tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that the tensor instruction obtains from the convolution kernel matrix at one time.

[0063] It can be understood that the present invention divides the input feature map matrix and convolution kernel matrix required for matrix multiplication into matrices required for one call by tensor instructions (corresponding to Figure 2 Middle gray shaded area: A sub-mapping and B sub-mapping), and finally the output feature map sub-matrix (corresponding to Figure 2 Middle gray shaded area: C submapping). Specifically, when performing the matrix multiplication of xy⊙yz=xz, only fixed-size input tensors xy and yz can be loaded at a time, and a fixed-size output tensor xz is obtained after calculation. The present invention uses the hardware instruction parameters (xy, yz) as the basic mapping unit, that is, mapping a submatrix in the input feature map matrix and the convolution kernel matrix, mapping the submatrix to the accelerator and performing iterative calculations.

[0064] In one embodiment, the memory arrangement of the output tensor in the output feature map matrix is ​​initialized as Wherein, N represents a batch of processed images, and in this embodiment, N is 1.

[0065] It is understandable that: Figure 2 As shown, C in the output feature map matrix is ​​a mapping in the output feature map, so it is necessary to adjust the memory layout of the output tensor in the output feature map, such as the memory layout is It should be noted here that the memory layout of the output tensor needs to refer to the tensor instruction parameters of the subsequent operator to reduce the data loading time when the next operator is calculated, so that the accelerator can continuously read data from the buffer without additional conversion, and maximize the use of the accelerator's computing resources.

[0066] 3) Compare the total amount of registers read from the global buffer under different multiplexing modes to obtain the optimal multiplexing mode;

[0067] In one embodiment, the optimal multiplexing mode is obtained based on the following formula:

[0068] V min =minimize(V InR ,V OutR ,V WtR );

[0069]

[0070]

[0071] Among them, V InR Indicates the total amount of registers read from the global buffer in input multiplexing mode; V OutR Indicates the total amount of registers read from the global buffer in output multiplexing mode; V WtR Indicates the total amount of registers read from the global buffer in the convolution kernel multiplexing mode; V min Indicates taking VInR ,V OutR ,V WtR The minimum value in V min The corresponding multiplexing mode is the optimal multiplexing mode; R in , R wt , R out It represents the input feature map, output feature map, and slice size of the convolution kernel read by the register from the global buffer respectively; Indicates the number of reads; C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, K is the convolution kernel size; x represents the minimum matrix width that a tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that a tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that a tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that a tensor instruction obtains from the convolution kernel matrix at one time.

[0072] In one embodiment, R in =xy; R wt =yz; R out =xz.

[0073] Since the tensor instruction can only load the input of size xy from the register at a time, and the weight of yz is calculated to obtain the partial sum of xz, the redundant data transmission is obviously unnecessary, so the slice size is used as the basic unit of the inner loop (global buffer to register) scheduling, that is, R in =xy,R wt =yz,R out = xz. At this time, the global buffer read amount is only related to the multiplexing method.

[0074] It can be understood that there are three data reuse modes in convolution operations, including input reuse (InR, one input is used by multiple convolution kernels), output reuse (OutR, part of the sum is reused) and convolution kernel reuse (WtR, one convolution kernel is used by multiple inputs); if each slice of the data type corresponding to the reuse mode is read only once by the register, it is the minimum global buffer read amount under this reuse mode, and the slice of the reused data type needs to be kept in the register for reuse until all the calculations in which it participates are completed, and other types of data are scheduled for calculation with it. Figure 3As shown in the figure, for input multiplexing, the convolution kernel slices of each output channel need to be loaded in sequence. For output multiplexing, the feature map and convolution kernel slices of each input channel need to be loaded in sequence. For convolution kernel multiplexing, the feature map slices of the length and width of the feature map need to be loaded in sequence. Figure 3 Taking the output reuse in as an example, the feature map slices and convolution kernel slices are loaded from DRAM to the global buffer in sequence according to the input channels. Each time the accelerator loads feature map slices of size xy and convolution kernel slices of size yz from the global cache to the register to perform tensor operations, the xy and yz are kept in the register for reuse, without the need for additional global buffer access.

[0075] Furthermore, after confirming the optimal multiplexing method, the memory layout of the output tensors in the output feature map matrix can be automatically updated based on the optimal multiplexing method to obtain the scheduling order of the inner loop (global buffer to register).

[0076] 4) Under the optimal multiplexing mode, the DRAM is sliced ​​using software pipeline to hide the scheduling time of data from the DRAM to the global buffer.

[0077] In one embodiment, the size of the DRAM slice satisfies the following formula:

[0078]

[0079] Among them, transfer cost(S) The time taken for the transfer from DRAM to the global buffer; compute cost(S) is the calculation time of the register under the optimal multiplexing mode; S in S is the slice size input in DRAM; out S is the output slice size in DRAM; wt is the convolution kernel slice size in DRAM; S is the total slice size in DRAM; GBuf is the global buffer size.

[0080] It is understandable that the software pipeline of the present invention uses double buffering to load, calculate and store slices, overlapping the calculation with DRAM access. The scheduling method of the outer loop of the present invention (DRAM to global buffer) does not affect the memory access time of the neural network model, that is, the total memory access time is only the access time from the global buffer to the register.

[0081] In order to verify the effectiveness of the inference optimization method of the present invention, the mapping method of the present invention and the unoptimized direct mapping method are compared in performance on three models: MobileNet, ResNet, and GoogleNet, as shown in Table 1.

[0082] Table 1 Comparison of inference delay time under different mapping methods (unit: ms)

[0083] Base Ours MobileNet 34.3 17.6 ResNet18 52.2 22 GoogleNet 153.4 61.2

[0084] It can be seen from Table 1 that the mapping method of the present invention achieves a geometric mean speedup of 2.3 times compared with the unoptimized direct mapping method.

[0085] The present invention is also compared with different scheduling methods, as shown in Table 2:

[0086] Table 2-Comparison of inference delay time under different scheduling methods (unit: ms)

[0087] Ours CEVA Ir Wlq MobileNet 10.8 11 15.3 15.6 ResNet18 15.1 24 17.4 38.8 GoogleNet 41.5 67 47.6 93.1

[0088] It can be seen from Table 2 that the scheduling method of the present invention achieves a 1.37 times geometric mean acceleration compared with other scheduling methods.

[0089] The embodiment of the present invention also discloses a physical structure diagram of an electronic device, such as Figure 4 As shown, the electronic device may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The processor 401 can call the logic instructions in the memory 403 to execute the above-mentioned hardware resource-aware neural network model reasoning optimization method.

[0090] In addition, the logic instructions in the above-mentioned memory 403 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0091] On the other hand, an embodiment of the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above-mentioned hardware resource-aware neural network model inference optimization method.

[0092] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0093] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A neural network reasoning optimization method adapted to hardware tensor instructions and memory, characterized in that: The following steps are involved: The convolution operations of different types of convolutional layers are uniformly mapped into equivalent matrix multiplications to obtain the input feature map matrix, convolution kernel matrix, and output feature map matrix; The input feature map matrix is ​​divided into input feature map sub-matrices and the convolution kernel matrix is ​​divided into convolution kernel sub-matrices by using hardware instruction parameters; based on the input feature map sub-matrix and the convolution kernel sub-matrix, an output feature map sub-matrix is ​​obtained; Compare the total amount of registers read from the global buffer under different multiplexing modes to obtain the optimal multiplexing mode; In the optimal multiplexing mode, the DRAM is sliced ​​using software pipelines to hide the scheduling time of data from the DRAM to the global buffer.

2. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 1, characterized in that: Expand the input feature map to obtain the input feature map matrix; The input feature map matrix is ​​H I *W I Columns, each column has a size of C I *K*K; Expand the convolution kernel to obtain the convolution kernel matrix; The convolution kernel matrix is ​​C O rows, each row has a size of C I *K*K; Expand the output feature map to obtain the output feature map matrix; The output feature map matrix is ​​C O rows, each row has a size of H O *W O ; Among them, C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, and K is the convolution kernel size.

3. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 2, characterized in that: The size of the input feature map submatrix is ​​xy; The size of the convolution kernel matrix is ​​yz; The size of the output feature map submatrix is ​​xz; Among them, x represents the minimum matrix width that the tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that the tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that the tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that the tensor instruction obtains from the convolution kernel matrix at one time.

4. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 3, characterized in that: Initialize the memory layout of the output tensor in the output feature map matrix as Here, N represents the batch of processed images.

5. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 3, characterized in that: The optimal multiplexing method is obtained based on the following formula: V min =minimize(V InR ,V OutR ,V WtR ); Among them, V InR Indicates the total amount of registers read from the global buffer in input multiplexing mode; V OutR Indicates the total amount of registers read from the global buffer in output multiplexing mode; V WtR Indicates the total amount of registers read from the global buffer in the convolution kernel multiplexing mode; V min Indicates taking V InR ,V OutR ,V WtR The minimum value in V min The corresponding multiplexing mode is the optimal multiplexing mode.

6. The neural network reasoning optimization method adapted to hardware tensor instructions and memory according to claim 5, characterized in that: Among them, R in , R wt , R out It represents the input feature map, output feature map, and slice size of the convolution kernel read by the register from the global buffer respectively; Indicates the number of reads.

7. The neural network reasoning optimization method adapted to hardware tensor instructions and memory according to claim 6, characterized in that: Among them, C I is the number of input feature map channels, C O is the number of output feature map channels, H I is the input feature map height, W I is the input feature map width, H O is the output feature map height, W O is the output feature map width, K is the convolution kernel size; x represents the minimum matrix width that a tensor instruction obtains from the input feature map matrix at one time; y represents the minimum matrix height that a tensor instruction obtains from the input feature map matrix at one time and the minimum matrix width that a tensor instruction obtains from the convolution kernel matrix at one time; z represents the minimum matrix height that a tensor instruction obtains from the convolution kernel matrix at one time.

8. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 6, characterized in that: R in =xy;R wt =yz;R out =xz。 9. The neural network reasoning optimization method for adapting hardware tensor instructions and memory according to claim 1, characterized in that: The size of the DRAM slice satisfies the following formula: Among them, transfer cost(S) The time taken for the transfer from DRAM to the global buffer; compute cost(S) is the calculation time of the register under the optimal multiplexing mode; S in S is the slice size input in DRAM; out S is the output slice size in DRAM; wt is the convolution kernel slice size in DRAM; S is the total slice size in DRAM; GBuf is the global buffer size.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hardware resource-aware neural network model reasoning optimization method as described in any one of claims 1 to 9 is implemented.

Images