A deep learning training and inference system based on FPGA

Abstract

The present invention relates to a deep learning training reasoning system based on FPGA, including cloud training equipment, sensor equipment, result comprehensive display equipment and edge training reasoning equipment; cloud training equipment, sensor equipment and result comprehensive display equipment are respectively connected to edge training reasoning equipment; cloud training equipment is used to obtain initial training parameters, and transmit the initial training parameters to edge training reasoning equipment; sensor equipment is used to collect data, and transmit the collected data to edge training reasoning equipment; edge training reasoning equipment will receive the initial training parameters and data for reasoning and online training; and the updated training parameters and reasoning results are output to the result comprehensive display equipment. Its advantages are: it can realize high-speed and high-efficiency reasoning, meet the training requirements of edge online learning, have the ability of training reasoning, and have the advantages of complete self-controllability, customizable development, low power consumption, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning algorithm training, and more specifically, to a deep learning training inference system based on FPGA. Background Art

[0002] Deep learning algorithm training generally includes four steps: forward reasoning, error back propagation, weight gradient calculation, and weight update. These four steps are iterated multiple times to obtain the best parameter training results that meet the requirements of the algorithm model. Deep learning algorithm reasoning reads in the parameters obtained from training and receives real-time data sources to achieve forward reasoning. The training process focuses on calculation accuracy and smaller or more certain error input. The reasoning process accepts a small amount of accuracy loss and cares about speed, cost, and power consumption.

[0003] Due to the requirements of equipment power consumption, autonomous control, customizable development, data security and confidentiality in application scenarios, and high transmission bandwidth pressure after data collection, the inference equipment needs to have certain online learning capabilities after being deployed to the edge. The existing training platforms mainly include central processing units (CPU), graphics processing units (GPU), application specific integrated circuits (ASIC), and field programmable gate arrays (FPGA). Among them, the CPU adopts a process control mode with low computing efficiency; the GPU has a large number of computing cores, but its power consumption is too high; ASIC is a fully customized hardware with good performance, but its design and manufacturing cycle is too long and the development cost is too high.

[0004] As a high-speed, customizable hardware, FPGA has the advantages of high energy efficiency, flexible design, and strong parallel acceleration capabilities. Compared with the above-mentioned training platforms, it is an ideal platform for implementing training reasoning at the edge with limited resources. However, since the training process requires operations such as error back propagation, weight gradient calculation, and weight update that are not available in reasoning, the reasoning device cannot be directly used for the training of deep learning algorithms. It is necessary to combine the characteristics of deep learning algorithms and do corresponding customized design work based on the reasoning device.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] The purpose of the present invention is to provide a deep learning training and inference system based on FPGA, which can not only realize high-speed and high-efficiency reasoning, but also meet the training requirements of edge online learning, and at the same time have the ability of training and inference, and also have the advantages of full autonomous control, customizable development, and low power consumption.

[0007] The present invention provides a deep learning training and reasoning system based on FPGA, comprising a cloud training device, a sensor device, a result comprehensive display device and an edge training and reasoning device; the cloud training device, the sensor device and the result comprehensive display device are respectively connected to the edge training and reasoning device; the cloud training device is used to obtain initial training parameters and transmit the initial training parameters to the edge training and reasoning device; the sensor device is used to collect data and transmit the collected data to the edge training and reasoning device; the edge training and reasoning device will receive the initial training parameters and data for reasoning and online training; and output the updated training parameters and reasoning results to the result comprehensive display device.

[0008] Furthermore, the edge training and inference device includes an edge online training parameter calculation module, a parameter storage module, a cache module, a convolutional neural network and a management module; the parameter storage module is respectively connected to the cloud training device, the edge online training parameter calculation module, the cache module and the management module; the cache module is also connected to the sensor device, the edge online training parameter calculation module, the convolutional neural network and the result comprehensive display device; the management module is also connected to the convolutional neural network and the edge online training parameter calculation module.

[0009] Furthermore, the parameter storage module includes a cloud initial training parameter storage module and an edge online training parameter storage module, the management module includes a training reasoning switching management module and an inter-layer relationship configuration management module, and the cache module includes an input data row and column cache module, an inter-layer data cache module, an input parameter cache module, a calculation parameter cache module and an output data cache module; the cloud initial training parameter storage module is respectively connected to the cloud training device, the training reasoning switching management module and the input parameter cache module; the edge online training parameter storage module is respectively connected to the edge online training parameter calculation module, the training reasoning switching management module, the input parameter cache module The training and inference switching management module is also connected to the edge-end online training parameter calculation module, and the edge-end online training parameter calculation module is also connected to the inter-layer data cache module and the calculation parameter cache module; the convolutional neural network is respectively connected to the input data row and column cache module, the inter-layer data cache module, the input parameter cache module, the calculation parameter cache module, the output data cache module and the inter-layer relationship configuration management module; the input data row and column cache module is also connected to the sensor device, and the output data cache module is also connected to the result comprehensive display device.

[0010] Furthermore, the convolutional neural network includes a convolutional layer module, which is used for forward reasoning, error back propagation, weight gradient calculation and weight update; the forward reasoning is to convolve the feature map of the layer input with the layer weight to obtain the current layer feature map output and use it as the input of the next layer, and its calculation formula (1) is:

[0011]

[0012] The error reverse propagation uses the loss function to calculate the error, and then performs a convolution operation on the error Error and the transpose of the weight to obtain the error of the current layer and use it for the propagation of the previous layer. Its calculation formula (2) is:

[0013]

[0014] The weight gradient calculation is to perform convolution calculation on the layer input feature map and the layer input error to obtain the weight gradient, and its calculation formula (3) is:

[0015]

[0016] The weight update is performed according to the learning rate η and the weight gradient, and the calculation formula (4) is:

[0017] Weight′(Chi,Chj)=Weight(Chi,Chj)-η·δWeight(Chi,Chj)

[0018] Wherein, ChI and ChO in formula (1), formula (2), formula (3) and formula (4) represent the number of input and output channels, Conv represents the convolution operation, and Weight T Represents the transpose of Weight (layer weight).

[0019] Furthermore, the convolutional neural network also includes a batch normalization layer module, the forward propagation of the batch normalization layer module is to normalize the feature map of the layer input, and then linearly transform the standardized data through the scale change factor γ and the offset β; the back propagation of the batch normalization layer module requires the derivation of the feature map, γ and β, and its calculation can be achieved by directly operating the current layer data by the processing system part in the FPGA.

[0020] Furthermore, the feature map, γ and β are derived, and their calculation can be achieved by outputting the current layer data to an FPGA external computing device.

[0021] Furthermore, the convolutional neural network also includes a nonlinear layer module, and the forward propagation of the nonlinear layer module selects a linear correction unit function to limit the current layer feature map within a reasonable range; the reverse propagation of the nonlinear layer module requires the derivation of the feature map, and its calculation can be achieved by outputting the current layer data to an FPGA external computing device.

[0022] Furthermore, the derivation of the feature map is achieved by directly operating the current layer data through the processing system in the FPGA.

[0023] The FPGA-based deep learning training and inference system provided by the present invention respectively receives the initial training parameters provided by the cloud training device and the data collected by the sensor device, independently completes the inference and online training, and outputs the updated training parameters and inference results to the result comprehensive display device; it can not only achieve high-speed and high-efficiency inference, but also meet the training requirements of edge online learning; the device also has the ability of training and inference, and also has the advantages of complete autonomous control, customizable development, low power consumption, etc., which is of great significance in applications such as large amounts of data collected by terminal sensors, large demands on terminal devices, and complex terminal application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the modules of an FPGA-based deep learning training inference system provided in an embodiment of the present invention.

[0025] Figure 2 for Figure 1Schematic diagram of another module of the FPGA-based deep learning training and inference system.

[0026] Figure 3 for Figure 1 Another module diagram of the FPGA-based deep learning training and inference system.

[0027] The reference numerals and components involved in the drawings are as follows:

[0028] 1. Cloud training equipment 2. Sensor equipment

[0029] 3. Result comprehensive display device 4. Edge training and inference device

[0030] 41. Edge online training parameter calculation module 42. Parameter storage module

[0031] 421. Cloud initial training parameter storage module 422. Edge online training parameter storage module

[0032] 43. Cache module 431. Input data row and column cache module

[0033] 432. Inter-layer data cache module 433. Input parameter cache module

[0034] 434. Calculation parameter cache module 435. Output data cache module

[0035] 44. Convolutional Neural Network 441. Convolutional Layer Module

[0036] 442. Batch Normalization Layer Module 443. Nonlinear Layer Module

[0037] 45. Management module 451. Training and reasoning switching management module

[0038] 452. Inter-layer relationship configuration management module DETAILED DESCRIPTION

[0039] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0040] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence.

[0041] Example 1

[0042] Figure 1 A schematic diagram of a module of a deep learning training and reasoning system based on FPGA provided by an embodiment of the present invention. Figure 1The FPGA-based deep learning training and reasoning system provided in an embodiment of the present invention includes a cloud training device 1, a sensor device 2, a result comprehensive display device 3 and an edge training and reasoning device 4; the cloud training device 1, the sensor device 2 and the result comprehensive display device 3 are respectively connected to the edge training and reasoning device 4; the cloud training device 1 is used to obtain initial training parameters and transmit the initial training parameters to the edge training and reasoning device 4; the sensor device 2 is used to collect data and transmit the collected data to the edge training and reasoning device 4; the edge training and reasoning device 4 performs reasoning and online training on the received initial training parameters and data; and outputs the updated training parameters and reasoning results to the result comprehensive display device 3.

[0043] It should be noted that in the FPGA-based deep learning training and inference system of the present invention, the edge training and inference device 4 receives the initial training parameters provided by the cloud training device 1 and the data collected by the sensor device 2 respectively, and after independently completing the inference and online training, the updated training parameters and inference results are output to the result comprehensive display device 3; it realizes both high-speed and high-efficiency inference and meets the training requirements of edge online learning; the device has the ability of training and inference at the same time, and also has the advantages of complete autonomous control, customizable development, low power consumption, etc., which is of great significance in applications such as large amount of data collected by terminal sensors, large demand for terminal equipment, and complex terminal application scenarios.

[0044] Figure 2 for Figure 1 Another module diagram of the FPGA-based deep learning training and inference system. Figure 3 for Figure 1 Another module diagram of the FPGA-based deep learning training and inference system. Please refer to Figure 2 , Figure 3 The edge training and inference device 4 of the present invention includes an edge online training parameter calculation module 41, a parameter storage module 42, a cache module 43, a convolutional neural network 44 and a management module 45; the parameter storage module 42 is respectively connected to the cloud training device 1, the edge online training parameter calculation module 41, the cache module 43 and the management module 45; the cache module 43 is also connected to the sensor device 2, the edge online training parameter calculation module 41, the convolutional neural network 44 and the result comprehensive display device 3; the management module 45 is also connected to the convolutional neural network 44 and the edge online training parameter calculation module 41.

[0045] Specifically, the parameter storage module 42 includes a cloud initial training parameter storage module 421 and an edge online training parameter storage module 422, the management module 45 includes a training reasoning switching management module 451 and an inter-layer relationship configuration management module 452, and the cache module 43 includes an input data row and column cache module 431, an inter-layer data cache module 432, an input parameter cache module 433, a calculation parameter cache module 434 and an output data cache module 435;

[0046] like Figure 3 As shown, the cloud initial training parameter storage module 421 is respectively connected to the cloud training device 1, the training and reasoning switching management module 451 and the input parameter cache module 433; the FPGA-based deep learning training and reasoning system of the present invention reads the initial training parameters from the cloud training device 1, and stores them in the storage space reserved by the device of the present invention (such as Flash, DDR and other storage chips outside the FPGA chip), and the cloud initial training parameter storage module 421 is used to implement the writing and reading operations of the initial training parameters; when the training is started, the device of the present invention waits for the control signal from the training and reasoning switching management module 451, and reads the stored parameters into the internal cache of the FPGA.

[0047] like Figure 3 As shown, the convolutional neural network 44 is respectively connected to the input data row and column cache module 431, the inter-layer data cache module 432, the input parameter cache module 433, the calculation parameter cache module 434, the output data cache module 435 and the inter-layer relationship configuration management module 452; the inter-layer relationship configuration management module 452 of the present invention can set the corresponding inter-layer connection relationship and layer calculation parameters according to different application scenarios and algorithm model requirements, and can adjust the network structure, layer sequence, and layer connection method to meet the customized design of the device according to the use requirements; for a determined network model, the inter-layer relationship configuration management module 452 can set the corresponding operation sequence, and the input data and input parameters flow into and out of the selected module in the form of data stream according to the corresponding requirements;

[0048] like Figure 3 As shown, the input data of the first layer, i.e., the FeatureMap input, flows into the selected layer module from the input data row and column cache module 431, and the layer result flows out to the inter-layer data cache module 432; the input data of the last layer flows into the selected layer module from the inter-layer data cache module 432, and the layer result, i.e., the FeatureMap output, flows out to the output data cache module 435; the input data of the middle layer flows into the selected layer module from the inter-layer data cache module 432, and the layer result flows out to the inter-layer data cache module 432.

[0049] like Figure 3As shown, the edge-end online training parameter storage module 422 is respectively connected to the edge-end online training parameter calculation module 41, the training and reasoning switching management module 451, the input parameter cache module 433 and the calculation parameter cache module 434; the training and reasoning switching management module 451 is also connected to the edge-end online training parameter calculation module 41, and the edge-end online training parameter calculation module 41 is also connected to the inter-layer data cache module 432 and the calculation parameter cache module 434;

[0050] It should be noted that in each layer, the steps of forward reasoning, error back propagation, weight gradient calculation, etc. are completed independently. Among them, the result of forward reasoning can be sent directly; the error back propagation and weight gradient calculation can be configured by the inter-layer relationship configuration management module 452 to configure additional convolutional layers to complete the convolution operations required for these two steps, and the calculated weight gradient δWeight is sent to the edge online training parameter calculation module 41; the weight update can be calculated by the edge online training parameter calculation module 41, and the new weight parameter Weight is written into the calculation parameter cache module 434 and the edge online training parameter storage module 422 in sequence. In order to accelerate the training process, part of the intermediate data FeatureMap can be written from the layer module to the inter-layer data cache module 432 and the edge online training parameter calculation module 41 in sequence, and after the calculation acceleration is completed, it is returned to the layer module.

[0051] like Figure 3 As shown, the input data row and column cache module 431 is also connected to the sensor device 2, and the output data cache module 435 is also connected to the result comprehensive display device 3. The FPGA-based deep learning training inference system of the present invention receives data collected by the sensor device 2, and the input data row and column cache module 431 implements data caching by row and column respectively according to the data caching requirements of the current layer (for example, the current layer is a convolution layer, and the convolution kernel is 3×3, at this time, it is necessary to cache 2 rows and 2 columns of the input data).

[0052] Further references Figure 3 The FPGA-based deep learning training and reasoning system of the present invention adopts the most common convolutional neural network (CNN) in deep learning algorithms and is widely used in target detection and recognition, semantic segmentation and other fields. The training process includes four steps: forward reasoning, error back propagation, weight gradient calculation and weight update. By performing multiple iterations on these four steps, the optimal parameter training results that meet the requirements of the algorithm model are obtained; the entire network is composed of multiple layers, and typical layers include convolutional layers, batch normalization layers and nonlinear layers. Forward reasoning is based on the forward connection relationship of the network, and error back propagation is based on the reverse connection relationship of the network.

[0053] Specifically, the convolutional neural network 44 includes a convolutional layer module 441, which is used for forward reasoning, error back propagation, weight gradient calculation, and weight update;

[0054] Forward reasoning is to convolve the feature map of the layer input with the layer weight to obtain the feature map output of the current layer and use it as the input of the next layer. Its calculation formula (1) is:

[0055]

[0056] Error back propagation uses the loss function to calculate the error, and then performs a convolution operation on the error Error and the transpose of the weight to obtain the error of the current layer and use it for the propagation of the previous layer. Its calculation formula (2) is:

[0057]

[0058] The weight gradient calculation is to perform convolution calculation on the layer input feature map and the layer input error to obtain the weight gradient. The calculation formula (3) is:

[0059]

[0060] Weight update updates the weight according to the learning rate η and the weight gradient. The calculation formula (4) is:

[0061] Weight′(Chi,Chj)=Weight(Chi,Chj)-η·δWeight(Chi,Chj)

[0062] Among them, ChI and ChO in Formula 1, Formula 2, Formula 3 and Formula 4 represent the number of input and output channels, Conv represents the convolution operation, and Weight T Represents the transpose of the Weight layer weights.

[0063] Furthermore, the convolutional neural network 44 also includes a batch normalization layer module 442. The forward propagation of the batch normalization layer module 442 is to normalize the feature map of the layer input, and then linearly transform the normalized data through the scale change factor γ and the offset β; the reverse propagation of the batch normalization layer module 442 requires the derivation of the feature map, γ and β, and its calculation can be achieved by directly operating the current layer data by the processing system part in the FPGA, or the calculation can be achieved by outputting the current layer data to the external computing device of the FPGA.

[0064] Furthermore, the convolutional neural network 44 also includes a nonlinear layer module 443. The forward propagation of the nonlinear layer module 443 selects a linear correction unit function to limit the current layer feature map within a reasonable range; the reverse propagation of the nonlinear layer module 443 requires the differentiation of the feature map, and its calculation can be achieved by outputting the current layer data to an external computing device of the FPGA or by directly operating the current layer data by the processing system within the FPGA.

[0065] It should be noted that, considering different application scenarios, the FPGA of the device of the present invention can select the Zynq series chip with its own system on chip (SoC), and implement the edge-end online training parameter calculation module 41 in the PS (processing system) part of the FPGA, and implement other modules in the PL (Processing Logic) part of the FPGA to meet the needs of miniaturization and integration. The device can also select an FPGA chip without SoC, and combine it with an ARM or a small-scale HiSilicon chip, implement the edge-end online training parameter calculation module 41 on an ARM or HiSilicon chip outside the FPGA, and implement other modules on the FPGA chip to meet the needs of miniaturization, high-speed reasoning, nationalization, and complex scenarios.

[0066] Based on the above description, it can be seen that the advantages of the present invention are:

[0067] 1. In the FPGA-based deep learning training and reasoning system of the present invention, the edge training and reasoning device 4 receives the initial training parameters provided by the cloud training device 1 and the data collected by the sensor device 2 respectively, completes the reasoning and online training autonomously, and outputs the updated training parameters and reasoning results to the result comprehensive display device 3; it realizes both high-speed and high-efficiency reasoning and meets the training requirements of edge online learning; the device has the ability of training and reasoning at the same time, and also has the advantages of complete autonomous control, customizable development, low power consumption, etc., which is of great significance in applications such as large amount of data collected by terminal sensors, large demand for terminal devices, and complex terminal application scenarios.

[0068] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A deep learning training and reasoning system based on FPGA, characterized in that: It comprises a cloud training device (1), a sensor device (2), a result comprehensive display device (3) and an edge training and reasoning device (4); the cloud training device (1), the sensor device (2) and the result comprehensive display device (3) are respectively connected to the edge training and reasoning device (4); The cloud training device (1) is used to obtain initial training parameters and transmit the initial training parameters to the edge training and reasoning device (4); the sensor device (2) is used to collect data and transmit the collected data to the edge training and reasoning device (4); The edge training and inference device (4) performs inference and online training on the received initial training parameters and data; and outputs the updated training parameters and inference results to the result comprehensive display device (3); The edge training and inference device (4) comprises an edge online training parameter calculation module (41), a parameter storage module (42), a cache module (43), a convolutional neural network (44) and a management module (45); The parameter storage module (42) is respectively connected to the cloud training device (1), the edge-end online training parameter calculation module (41), the cache module (43) and the management module (45); the cache module (43) is also connected to the sensor device (2), the edge-end online training parameter calculation module (41), the convolutional neural network (44) and the result comprehensive display device (3); the management module (45) is also connected to the convolutional neural network (44) and the edge-end online training parameter calculation module (41); The parameter storage module (42) includes a cloud initial training parameter storage module (421) and an edge online training parameter storage module (422); the management module (45) includes a training reasoning switching management module (451) and an inter-layer relationship configuration management module (452); the cache module (43) includes an input data row and column cache module (431), an inter-layer data cache module (432), an input parameter cache module (433), a calculation parameter cache module (434) and an output data cache module (435); The cloud initial training parameter storage module (421) is respectively connected to the cloud training device (1), the training reasoning switching management module (451) and the input parameter cache module (433); The edge-end online training parameter storage module (422) is respectively connected to the edge-end online training parameter calculation module (41), the training reasoning switching management module (451), the input parameter cache module (433) and the calculation parameter cache module (434); The training and reasoning switching management module (451) is also connected to the edge-end online training parameter calculation module (41), and the edge-end online training parameter calculation module (41) is also connected to the inter-layer data cache module (432) and the calculation parameter cache module (434); The convolutional neural network (44) is respectively connected to the input data row and column cache module (431), the inter-layer data cache module (432), the input parameter cache module (433), the calculation parameter cache module (434), the output data cache module (435) and the inter-layer relationship configuration management module (452); The input data row and column cache module (431) is also connected to the sensor device (2), and the output data cache module (435) is also connected to the result comprehensive display device (3); The convolutional neural network (44) comprises a convolutional layer module (441), and the convolutional layer module (441) is used for forward reasoning, error back propagation, weight gradient calculation and weight update; The forward reasoning is to convolve the feature map of the layer input with the layer weight to obtain the feature map output of the current layer and use it as the input of the next layer. Its calculation formula (1) is: The error reverse propagation uses the loss function to calculate the error, and then performs a convolution operation on the error Error and the transpose of the weight to obtain the error of the current layer and use it for the propagation of the previous layer. The calculation formula (2) is: The weight gradient calculation is to perform convolution calculation on the layer input feature map and the layer input error to obtain the weight gradient, and its calculation formula (3) is: The weight update is performed according to the learning rate η and the weight gradient, and the calculation formula (4) is: Weight′(Chi,Chj)=Weight(Chi,Chj)-η·δWeight(Chi,Chj) Wherein, in formula (1), formula (2), formula (3) and formula (4), ChI and ChO represent the number of input and output channels; Conv represents the convolution operation, and Weight T Indicates that Weight is the transpose of the layer weight.

2. The FPGA-based deep learning training inference system according to claim 1, characterized in that: The convolutional neural network (44) also includes a batch normalization layer module (442). The forward propagation of the batch normalization layer module (442) is to normalize the feature map of the layer input, and then linearly transform the normalized data through the scale change factor γ and the offset β; the reverse propagation of the batch normalization layer module (442) requires the derivation of the feature map, γ and β, and its calculation can be achieved by directly operating the current layer data by the processing system part in the FPGA.

3. The FPGA-based deep learning training inference system according to claim 2, characterized in that: The feature map, γ and β are derived, and their calculation can be achieved by outputting the current layer data to the FPGA external computing device.

4. The FPGA-based deep learning training inference system according to claim 1, characterized in that: The convolutional neural network (44) also includes a nonlinear layer module (443), and the forward propagation of the nonlinear layer module (443) selects a linear correction unit function to restrict the current layer feature map; the reverse propagation of the nonlinear layer module (443) requires the derivation of the feature map, and its calculation can be achieved by outputting the current layer data to an FPGA external computing device.

5. The FPGA-based deep learning training inference system according to claim 4, characterized in that: The derivation of the feature map is achieved by directly operating the current layer data through the processing system in the FPGA.

Images