DEEP LEARNING INFERENCE SYSTEM AND INFERENCE SERVING METHOD
By dynamically configuring FPGA accelerators with optimal ALU and GEMM circuits based on client requests, the system addresses inefficiencies in power consumption and processing time, achieving efficient inference serving.
Patent Information
- Application Number
- JP2023555945
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-10-27
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-10-27
Smart Images

Figure 0007582505000001 
Figure 0007582505000002 
Figure 0007582505000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a deep learning inference system and an inference serving method that perform inference serving using a multi-layer neural network. [Background technology]
[0002] In recent years, there are many services that use multi-layer neural networks to process information and utilize the results. The process of giving a computing unit the operation of a neural network calculation, the parameters of the neural network, and the data to be processed to obtain processed data is called inference. Inference requires a large number of calculations and memory. For this reason, inference is sometimes performed on a server.
[0003] A client transmits a request and data to be processed to a server, and receives the result of the processing as a response. The provision of such a service is called inference serving. Various methods have been proposed for inference serving (see Non-Patent Document 1).
[0004] Here, we will describe a case where an FPGA (field-programmable gate array) accelerator is used as the computing unit. Figure 4 is a block diagram showing the configuration of the server. A CPU (Central Processing Unit) 100 controls serving. A NIC (Network Interface Card) 101 receives requests from clients via a network and returns inference results from a neural network to the client via the network.
[0005] The memory 102 stores operation codes (machine language instructions) for all neural network operations, neural network parameters, and data to be processed. The FPGA accelerator 103 is composed of a PCIe (PCI Express) circuit, a DRAM (Dynamic Random Access Memory), and an FPGA circuit area. The CPU 100, the NIC 101, the memory 102, and the FPGA accelerator 103 are connected by a PCIe bus 104.
[0006] When using the FPGA accelerator 103 as a computing unit for inference serving, a method of constructing a von Neumann type computer on the FPGA accelerator 103 is common (see Non-Patent Document 2). A generalized configuration of a von Neumann type computer constructed on the FPGA accelerator 103 is shown in FIG.
[0007] An operation code 200 for the neural network calculation, neural network parameters 201, and data to be processed are input to the FPGA accelerator 103 via the PCIe bus and stored in the DRAM 105. In the example of Fig. 5, the data to be processed and data in the middle of the calculation are set as input data 202.
[0008] An instruction fetch module 106 reads out an operation code 200 from the DRAM 105 and transfers it to a load module 107 , a compute module 108 , and a store module 109 .
[0009] The load module 107 reads the input data 202 from the DRAM 105 , batches a plurality of pieces of input data 202 , and transfers them to the compute module 108 . The compute module 108 performs neural network calculations using input data 202 and parameters 201 according to the operation code 200 transferred from the instruction fetch module 106. The compute module 108 is equipped with an ALU (Arithmetic Logic Unit) 1080 and a GEMM (General matrix multiply) circuit 1081. After performing calculations according to the operation code 200, the compute module 108 transfers the calculation results to the store module 109.
[0010] The store module 109 stores the results of the calculations performed by the compute module 108 in the DRAM 105. At this time, not only the processed data is stored in the DRAM 105 as the output data 203, but also data in the middle of the calculation may be temporarily stored as the output data 203. The data in the middle of the calculation becomes the input data 202 to the load module 107.
[0011] The operation of the inference serving by the server is shown in Fig. 6. The CPU 100 of the server operates as an application function unit 1000 and a loader 1001 according to the inference serving program.
[0012] When started, the application function unit 1000 constructs the aforementioned von Neumann type computer in the FPGA accelerator 103 (step S100 in FIG. 6). The NIC 101 transfers an inference request received from a client via the network and data to be processed to the application function unit 1000 (Step S101 in FIG. 6). The inference request specifies which model is to be used for inference.
[0013] The application function unit 1000 passes the data to be processed to the loader 1001, and specifies the operation code to be executed according to the inference request (step S102 in FIG. 6). The loader 1001 reads the specified operation code and the neural network parameters from the memory 102 (steps S103 and S104 in FIG. 6).
[0014] The loader 1001 passes the operation code, the parameters, and the data to be processed to the FPGA accelerator 103 (step S105 in FIG. 6). The FPGA accelerator 103 performs neural network calculations on the data to be processed in accordance with the operation code transferred from the loader 1001, and stores the processed data obtained as a result of the calculations in the DRAM 105 inside the FPGA accelerator 103 (step S106 in FIG. 6).
[0015] The loader 1001 reads the processed data from the DRAM 105 in the FPGA accelerator 103 (step S107 in FIG. 6). The application function unit 1000 receives the processed data from the loader 1001, and returns this data to the client as a response to the inference request (step S108 in FIG. 6).
[0016] The operation code of the neural network is capable of parallel execution and is executed by the ALU 1080 and the GEMM circuit 1081 . The power efficiency is most improved by using an ALU 1080 with a degree of parallelism suited to the operation code and a GEMM circuit 1081 with a size suited to the operation code.
[0017] However, in conventional systems, a computing unit that is too large or too small for a neural network may be used. When a computing unit that is too large for a neural network is used, there is a problem that the power efficiency decreases because the computing unit is not used. Also, when a computing unit that is too small for a neural network is used, the processing time increases, so the total amount of power consumption increases, and the power efficiency decreases. [Prior art documents] [Non-patent literature]
[0018] [Non-Patent Document 1] Christopher Olston, et al., “Tensorflow-serving: Flexible, high-performance ml serving”, Cornell University Library, arXiv preprint arXiv:1712.06139, 2017 [Non-Patent Document 2] Thierry Moreau, Tianqi Chen, Luis Ceze, “Leveraging the vta-tvm hardware-software stack for fpga acceleration of 8-bit resnet-18 inference”, Proceedings of the 1st on Reproducible Quality-Efficient Systems Tournament on Co-designing Pareto-efficient Deep Learning, 2018 Summary of the Invention [Problem to be solved by the invention]
[0019] The present invention has been made to solve the above-mentioned problems, and aims to provide a deep learning inference system and an inference serving method that can perform inference serving with optimal power efficiency. [Means for solving the problem]
[0020] The deep learning inference system of the present invention includes an FPGA accelerator configured to perform at least a part of the calculations of a neural network using an ALU and a GEMM circuit in response to a request from a client; a loader configured to transfer an operation code of the calculations of the neural network and data to be processed to the FPGA accelerator; and an application function unit configured to pass the data to be processed received from the client to the loader, specify the operation code to be executed in accordance with the request to the loader, and return processed data obtained as a result of the calculations of the neural network to the client, wherein the application function unit determines the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on the operation code transferred by the loader to the FPGA accelerator, and changes the configuration of the FPGA accelerator so that an ALU of the determined parallelism and a GEMM circuit of the determined size are arranged.
[0021] Furthermore, in one configuration example of the deep learning inference system of the present invention, the application function unit is characterized in that it determines the parallelism of the ALU and the size of the GEMM circuit based on a result of estimating the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on the operation code transferred by the loader to the FPGA accelerator, and based on the number of computing elements that can be implemented in the FPGA accelerator. In addition, in one configuration example of the deep learning inference system of the present invention, the application function unit is characterized in that it changes the configuration of the FPGA accelerator by a partial reconfiguration function.
[0022] Furthermore, the inference serving method of the present invention is characterized by including a first step of receiving an inference request and data to be processed from a client; a second step of transferring the data to be processed received from the client to a loader and specifying to the loader an operation code of a neural network operation to be executed in accordance with the request; a third step of determining the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on the operation code transferred to an FPGA accelerator that performs at least a part of the operation of the neural network using an ALU and a GEMM circuit; a fourth step of changing a configuration of the FPGA accelerator so that an ALU with the determined parallelism and a GEMM circuit with the determined size are arranged; a fifth step of transferring the operation code and the data to be processed from the loader to the FPGA accelerator; and a sixth step of returning processed data obtained as a result of the operation of the neural network to the client.
[0023] Furthermore, in one configuration example of the inference serving method of the present invention, the third step is characterized by including a step of determining the parallelism of the ALU and the size of the GEMM circuit based on a result of estimating the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on an operation code transferred to the FPGA accelerator and on the number of computing elements that can be implemented in the FPGA accelerator. In one configuration example of the inference serving method of the present invention, the fourth step is characterized by including a step of changing the configuration of the FPGA accelerator by a partial reconfiguration function. Effect of the Invention
[0024] According to the present invention, it is possible to allocate the optimal ALU and GEMM circuit to the FPGA accelerator in response to a request from a client, and to execute inference serving with optimal power efficiency. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a block diagram showing a configuration of a server of a deep learning inference system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a von Neumann type computer built on an FPGA accelerator according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a sequence diagram illustrating an inference serving operation by a server according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram showing the configuration of the server. [Diagram 5] FIG. 5 is a block diagram showing the configuration of a von Neumann type computer built on an FPGA accelerator. [Figure 6] FIG. 6 is a sequence diagram illustrating an inference serving operation by the server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] [Principle of the invention] The present invention performs inference serving by adjusting a von Neumann type computer after receiving a request from a client, using an ALU and a GEMM circuit that are optimal for the request.
[0027] [Example] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a server of a deep learning inference system according to an embodiment of the present invention. The server includes a CPU 100a, a NIC 101, a memory 102, and an FPGA accelerator 103. The FPGA accelerator 103 performs at least a part of the calculations of the multilayer neural network using an ALU and a GEMM circuit.
[0028] A program for implementing the inference serving method of this embodiment is stored in the memory 102. The CPU 100a executes processing according to the program stored in the memory 102, and functions as an application function unit 1000a and a loader 1001.
[0029] 2 is a block diagram showing the configuration of a von Neumann-type computer constructed on the FPGA accelerator 103 of this embodiment. In this embodiment, an ALU 1080 and a GEMM circuit 1081 are arranged in a partial reconfiguration region 1082 of the FPGA accelerator 103.
[0030] The operation of inference serving by the server of this embodiment is shown in Fig. 3. When started up, the application function unit 1000a constructs a von Neumann type computer in the FPGA accelerator 103 (step S200 in Fig. 3).
[0031] The NIC 101 transfers an inference request received from a client via the network and data to be processed to the application function unit 1000a (Step S201 in FIG. 3). The inference request specifies which model is to be used for inference.
[0032] The application function unit 1000a passes the data to be processed to the loader 1001, and specifies to the loader 1001 an operation code to be executed in accordance with the inference request and parameters of the neural network required for inference (FIG. 3, step S202). The loader 1001 reads the specified operation code and the neural network parameters from the memory 102 (steps S203 and S204 in FIG. 3).
[0033] Next, the application function unit 1000a determines the parallelism of the ALUs 1080 (the number of ALUs 1080) and the size of the GEMM circuit 1081 corresponding to the inference requested by the inference request from the client based on the operation code read by the loader 1001 (FIG. 3, step S205).
[0034] In neural networks, the data to be processed, such as images, and parameters such as the weights of the neural network are expressed as tensors, as disclosed in the literature "Thierry Moreau1,et al.,“A Hardware-Software Blueprint for Flexible Deep Learning Specialization”,Cornell University Library,USA,arXiv:1807.04188v3,2019". The ALU 1080 performs element-wise tensor operations such as addition. The GEMM circuit 1081 performs operations such as matrix multiplication of the inputs and parameters.
[0035] The operation code for the GEMM circuit 1081 specifies the tensor to be input to the GEMM circuit 1081. The size of the tensor is known at the stage where the tensor is stored in a buffer (not shown) provided in the stage preceding the GEMM circuit 1081. Therefore, the application function unit 1000a can read information on what size of tensors will be multiplied by a matrix from a plurality of operation codes for the GEMM circuit 1081, and can estimate the size of the GEMM circuit 1081 that is optimal for inference based on this information.
[0036] Here, the optimal size of GEMM circuit 1081 will be described. Matrix multiplication can be performed in parallel. The size of GEMM circuit 1081 referred to in the present invention means the number of parallel operations. Considering an example of matrix multiplication of a 2×2 matrix, eight multiplications are performed in this example, and each multiplication can be performed in parallel. In other words, the optimal number of parallel operations is eight.
[0037] However, since the number of calculation elements in the FPGA accelerator 103 is limited, it is not always possible to prepare a GEMM circuit 1081 with a parallel operation number of 8. When it is not possible to prepare a GEMM circuit 1081 with an optimal number of parallel operations, it is more efficient to implement a GEMM circuit 1081 that can execute parallel operations of a divisor of the optimal number of parallel operations.
[0038] For example, when a GEMM circuit 1081 capable of executing 4 parallel operations is implemented with respect to the optimal number of parallel operations being 8, a 2×2 matrix multiplication operation can be executed in 2 cycles. On the other hand, when a GEMM circuit 1081 capable of executing 6 parallel operations is implemented, a 2×2 matrix multiplication operation is executed in 2 cycles, but since more computational elements are operating than in the GEMM circuit 1081 capable of executing 4 parallel operations, it can be said that this is not optimal. In other words, the application function unit 1000a may estimate the optimal number of parallel operations for the matrix multiplication operation to be executed from the tensor input to the GEMM circuit 1081, and determine the size of the GEMM circuit 1081 based on the optimal number of parallel operations and the number of computational elements that can be implemented in the FPGA accelerator 103.
[0039] Similarly, the application function unit 1000a may estimate the optimal parallelism (the number of ALUs 1080) of the ALU 1080. Specifically, the application function unit 1000a may read information on the degree of parallelism of the tensor operation to be executed from a plurality of operation codes for the ALU 1080, estimate the optimal parallelism of the ALU 1080 for inference based on this information, and determine the number of ALUs 1080 based on the optimal parallelism and the number of computing elements that can be implemented in the FPGA accelerator 103.
[0040] The application function unit 1000a uses a partial reconfiguration function to change the configuration of the FPGA accelerator 103 so that the ALU 1080 with the parallelism determined in step S205 and the GEMM circuit 1081 with the size determined in step S205 are arranged in the FPGA accelerator 103 (step S206 in FIG. 3). The application function unit 1000a can rewrite the partially reconfigurable area 1082 of the FPGA accelerator 103 by sending bit stream data for partial reconfiguration to the FPGA accelerator 103. As is well known, the partial reconfiguration function makes it possible to rewrite the FPGA accelerator 103 while it is operating.
[0041] The loader 1001 passes the operation code and parameters read from the memory 102 and the data to be processed received from the application function unit 1000a to the FPGA accelerator 103 (step S207 in FIG. 3). Note that the process of step S207 is executed in parallel with the partial reconfiguration of step S206.
[0042] The FPGA accelerator 103 performs neural network calculations on the data to be processed in accordance with the operation code transferred from the loader 1001, and stores the processed data obtained as a result of the calculations in the DRAM 105 inside the FPGA accelerator 103 (step S208 in FIG. 3).
[0043] The loader 1001 reads the processed data from the DRAM 105 in the FPGA accelerator 103 (step S209 in FIG. 3). The application function unit 1000a receives the processed data from the loader 1001, and returns this data to the client as a response to the inference request (step S210 in FIG. 3).
[0044] As described above, in this embodiment, it is possible to allocate the optimal ALU and GEMM circuit to an inference request from a client, and to execute inference serving with optimal power efficiency. In this embodiment, by partially reconfiguring only the ALU and GEMM circuit, almost no overhead occurs. [Industrial Applicability]
[0045] The present invention can be applied to a technology for providing services using neural networks. [Explanation of symbols]
[0046] 100a...CPU, 101...NIC, 102...memory, 103...FPGA accelerator, 104...PCIe bus, 105...DRAM, 106...instruction fetch module, 107...load module, 108...compute module, 109...store module, 1000a...application function unit, 1001...loader, 1080...ALU, 1081...GEMM circuit, 1082...partially reconfigurable area.
Claims
1. An FPGA accelerator configured to perform at least a part of the calculation of a neural network using an ALU and a GEMM circuit in response to a request from a client; a loader configured to transfer an operation code of the neural network operation and data to be processed to the FPGA accelerator; an application function unit configured to pass the data to be processed received from the client to the loader, specify the operation code to be executed in accordance with the request to the loader, and return processed data obtained as a result of the operation of the neural network to the client; The application function unit determines the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on the operation code transferred by the loader to the FPGA accelerator, and changes the configuration of the FPGA accelerator so that an ALU of the determined parallelism and a GEMM circuit of the determined size are arranged.
2. The deep learning inference system according to claim 1, The application function unit determines the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on the operation code transferred by the loader to the FPGA accelerator, and determines the parallelism of the ALU and the size of the GEMM circuit based on the result of estimating the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client, and based on the number of computing elements that can be implemented in the FPGA accelerator.
3. The deep learning inference system according to claim 1 or 2, The deep learning inference system is characterized in that the application function unit changes the configuration of the FPGA accelerator by a partial reconfiguration function.
4. a first step of receiving a request for inference and data to be processed from a client; a second step of passing the data to be processed received from the client to a loader and specifying to the loader an operation code of a neural network operation to be executed in accordance with the request; a third step of determining a parallel degree of the ALU and a size of the GEMM circuit corresponding to an inference requested by a request from the client based on the operation code transferred to an FPGA accelerator that performs at least a part of the calculations of the neural network using an ALU and a GEMM circuit; a fourth step of changing a configuration of the FPGA accelerator so that an ALU having the determined parallelism and a GEMM circuit having the determined size are arranged; a fifth step of transferring the operation code and the data to be processed from the loader to the FPGA accelerator; and a sixth step of returning processed data resulting from the operation of the neural network to the client.
5. 5. The inference serving method of claim 4, further comprising: The third step is an inference serving method characterized in that it includes a step of determining the parallelism of the ALU and the size of the GEMM circuit based on a result of estimating the parallelism of the ALU and the size of the GEMM circuit corresponding to the inference requested by the request from the client based on an operation code transferred to the FPGA accelerator, and based on the number of computing elements that can be implemented in the FPGA accelerator.
6. 6. The inference serving method according to claim 4, further comprising: The inference serving method, wherein the fourth step includes a step of changing a configuration of the FPGA accelerator by a partial reconfiguration function.
Citation Information
Patent Citations
Convolutional layer acceleration unit, embedded system having the same, and method for operating the embedded system
US20200302291A1
