A method and system for implementing BNN forward inference suitable for resource-constrained FPGA

Abstract

The application relates to a method and system for implementing BNN forward inference on an FPGA with limited resources, comprising the following steps: S1, reading feature map information, and storing weight information into an on-chip RAM of the FPGA; S2, performing normalization and mean removal on a first-layer input image, and then performing 8-bit quantization to enter an intermediate network, wherein the intermediate network is divided into multiple macro convolution layers, binary weight and an activation function are adopted, and each convolution layer is processed in the order of convolution, batch regularization and activation; and S3, multiplying the last fully connected layer by using 8-bit quantization precision, and then judging and outputting. The application can greatly reduce the overhead of multipliers and memories, and greatly reduce running power consumption.

Description

Technical Field

[0001] This invention relates to the field of parallel computing technology, and in particular to a method and system for implementing BNN forward inference on resource-constrained FPGAs. Background Technology

[0002] Binary Neural Networks (BNNs) are neural networks that use only two values, +1 and -1, to represent weights and activations. The BNN method not only greatly reduces the demand for memory capacity and computing resources, but also changes the traditional calculation method of convolution, reducing the complexity and power consumption of the operation.

[0003] FPGAs offer great flexibility in hardware architecture design, supporting efficient bitwise computation with low power consumption. Their ultimate counterpart, the Application-Specific Integrated Circuit (ASIC), is even more efficient and energy-saving than FPGAs. Currently, AI accelerators designed using FPGAs primarily utilize Xilinx devices and development tools. While they have designed a dedicated architecture, FINN, for binary neural networks, implementing BNN forward inference suffers from significant overhead in multipliers and memory, resulting in high power consumption. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a method for implementing BNN forward inference on resource-constrained FPGAs, which can significantly reduce the overhead of multipliers and memory, while also greatly reducing operating power consumption.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for implementing BNN forward inference on resource-constrained FPGAs, including

[0007] Step S1: Read the feature map information and store the weight information in the FPGA's on-chip RAM;

[0008] Step S2: The first layer input image is first normalized and the mean is removed, and then 8-bit quantization is performed before entering the intermediate network. The intermediate network is divided into multiple macro-convolutional layers, using binarized weights and activation functions. Each convolutional layer is processed in the order of convolution, batch regularization, and activation.

[0009] Step S3: The final fully connected layer is multiplied using 8-bit quantization precision and then output as a decision.

[0010] Furthermore, in step S1, the feature map and weight information are directly read from an external memory or an external interface, and the weight information is binarized.

[0011] Furthermore, step S2 normalizes the first-layer input image (Xr, Xg, Xb), removes the mean, and then performs 8-bit quantization, as follows:

[0012] Xr1=(Xr-min(Xr)) / (max(Xr)-min(Xr));

[0013] Xr2 = Xr1 - mean(Xr1);

[0014] Xg1=(Xg-min(Xg)) / (max(Xg)-min(Xg));

[0015] Xg2 = Xg1 - mean(Xg1);

[0016] Xb1=(Xb-min(Xb)) / (max(Xb)-min(Xb));

[0017] Xb 2 = Xb 1 - mean(Xb 1);

[0018] Then, Xr2, Xg2, and Xb2 are subjected to signed 8-bit conventional linear quantization.

[0019] Furthermore, the convolution operation in the macro-convolutional layer is as follows:

[0020] Map -1 in the feature map and weights to 0;

[0021] Then, the bitwise XOR operation is used to replace the multiplication operation. Finally, the sum of the XOR results is multiplied by 2 and subtracted from the total length to obtain the convolution result.

[0022] Furthermore, the batch regularization and activation in the macro-convolutional layer are combined to replace the multiplication and division operations in BN processing, as follows:

[0023] The BN layer is processed in forward inference as follows:

[0024]

[0025] Where μ represents the mean of the input vector, σ represents the standard deviation of the input vector, x is the input, y is the BN output, and γ and β are parameters learned during training, used to perform affine transformations on the activation values ​​to ensure the restoration of the original input.

[0026] Activation function for forward inference:

[0027]

[0028] Combining BN layers and activation, we obtain

[0029]

[0030] Where σ>0, γ and β are known parameters, and γ≠0, and remains unchanged during forward inference, therefore we obtain:

[0031]

[0032] That is, different calculation formulas are obtained when γ > 0 and γ < 0.

[0033] Furthermore, the fully connected layer performs 8-bit precision feature map and weight multiplication processing, and the multiplication operation is completed by the DSP in the FPGA. By combining the input data, one DSP can complete the multiplication operation of two data simultaneously. For the same weight coefficient b, one DSP module can simultaneously complete the multiplication of two feature values, i.e., a1xb and a2xb are completed by one DSP module.

[0034] A BNN forward inference system adapted to resource-constrained FPGAs includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the method for implementing BNN forward inference adapted to resource-constrained FPGAs as described above.

[0035] A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0036] The present invention has the following beneficial effects:

[0037] This invention addresses the challenges of implementing BNN forward inference on FPGAs with limited computing and storage resources. It presents a method that meets the requirements for implementing BNN forward inference on FPGAs with such constraints, significantly reducing the overhead of multipliers and memory, while also substantially lowering power consumption. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a block diagram of the main architecture of the BNN neural network in one embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] This invention provides a method for implementing BNN forward inference on resource-constrained FPGAs, comprising the following steps:

[0042] Step S1: Read the feature map information and store the weight information in the FPGA's on-chip RAM;

[0043] Step S2: The first layer input image is first normalized and the mean is removed, and then 8-bit quantization is performed before entering the intermediate network. The intermediate network is divided into multiple macro-convolutional layers, using binarized weights and activation functions. Each convolutional layer is processed in the order of convolution, batch regularization, and activation.

[0044] Step S3: The final fully connected layer is multiplied using 8-bit quantization precision and then output as a decision.

[0045] Preferably, in this embodiment, in step S1, the feature map and weight information are directly read from an external memory or an external interface, and the weight information is binarized.

[0046] Preferably, in this embodiment, step S2 normalizes the first-layer input image (Xr, Xg, Xb), removes the mean, and then performs 8-bit quantization, as follows:

[0047] Xr1=(Xr-min(Xr)) / (max(Xr)-min(Xr));

[0048] Xr2 = Xr1 - mean(Xr1);

[0049] Xg1=(Xg-min(Xg)) / (max(Xg)-min(Xg));

[0050] Xg2 = Xg1 - mean(Xg1);

[0051] Xb1=(Xb-min(Xb)) / (max(Xb)-min(Xb));

[0052] Xb 2 = Xb 1 - mean(Xb 1);

[0053] Then, Xr2, Xg2, and Xb2 are subjected to signed 8-bit conventional linear quantization.

[0054] In this embodiment, the intermediate macro-convolutional layers are combined according to different functions and uses in the forward inference operation of BNN, but the macro-convolutional layers of the basic unit are the same. Therefore, this method focuses on the macro-convolutional layer of the basic unit. In addition, depending on the purpose of the network, pooling, softmax, etc. may be involved and implemented according to the general method, which will not be elaborated here.

[0055] Preferably, the convolution operation in the macro-convolutional layer is as follows:

[0056] Map -1 in the feature map and weights to 0; then replace multiplication with bitwise XOR (xnor) operation; finally, sum the XOR results, multiply by 2, and subtract the total length to obtain the convolution result. If the convolution kernel is 3x3 and the binarization coefficients are (1, -1, 1; -1, 1, -1; 1, 1, -1), and the region values ​​for convolution corresponding to the feature values ​​are (-1, -1, 1; 1, 1, 1; -1, 1, -1), then according to normal convolution multiplication = 1*(-1) + (-1)*(-1) + 1*1 + (-1)*1 + 1*1 + (-1)*1 + 1*(-1) + 1*1 + (-1)*(-1) = 1; however, this implementation does not require multiplication. First,... -1 is mapped to 0, meaning the binarization coefficients are (1,0,1; 0,1,0; 1,1,0), and the eigenvalue convolution region values ​​are (0,0,1; 1,1,1; 0,1,0). Then, an XOR operation is performed to obtain (0,1,1,0,1,0,0,1,1). The sum of the XOR results is multiplied by 2 = sum(0,1,1,0,1,0,0,1,1) x 2 = 10. Finally, the result is subtracted from the total length to obtain 10 - 9 = 1, which is the same result as normal convolution multiplication and addition.

[0057] Preferably, batch regularization and activation in the macro-convolutional layer are combined to replace the multiplication and division operations in BN processing, as follows:

[0058] The BN layer is processed in forward inference as follows:

[0059]

[0060] Where μ represents the mean of the input vector, σ represents the standard deviation of the input vector, x is the input, y is the BN output, and γ and β are parameters learned during training, used to perform affine transformations on the activation values ​​to ensure the restoration of the original input.

[0061] Activation function for forward inference:

[0062]

[0063] Combining BN layers and activation, we obtain

[0064]

[0065] Where σ>0, γ and β are known parameters, and γ≠0, and remains unchanged during forward inference, therefore we obtain:

[0066]

[0067] That is, different calculation formulas are obtained when γ > 0 and γ < 0. Since γ and β are known parameters, in the specific implementation process... This is input as a fixed value before performing forward inference; therefore, in actual forward inference, only the input value needs to be changed to the input value before forward inference. The operation of multiplying this fixed value by σ simultaneously performs the subtraction operation with μ, and finally compares it with x. Therefore, compared to traditional Batch Normalization (BN), it eliminates the need for division, requiring only one multiplication operation, significantly reducing computational overhead.

[0068] The fully connected layer performs 8-bit precision feature map and weight multiplication. The multiplication operation is completed using a DSP in the FPGA. By combining input data, one DSP can simultaneously perform multiplication of two data points. For the same weight coefficient b, one DSP module can simultaneously perform multiplication of two feature values, i.e., a1xb and a2xb, both using the same DSP module. For example, in a Xilinx FPGA, the bit width of the DSP module is 25 bits for input A (A[24:0]) and 18 bits for input B (B[17:0]). The method to combine the two multiplications is a1 = A[24:17], a2 = A[7:0], b = B[7:0], that is, the 25-bit combination of A is A = [a1, A[16:8], a2]. In this way, the DSP module completes two multiplications with the same coefficient in one multiplication operation, reducing operating overhead and increasing operating speed.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for implementing BNN forward inference on resource-constrained FPGAs, characterized in that, include Step S1: Read the feature map information and store the weight information in the FPGA's on-chip RAM; Step S2: The first layer input image is first normalized and the mean is removed, and then 8-bit quantization is performed before entering the intermediate network. The intermediate network is divided into multiple macro-convolutional layers, using binarized weights and activation functions. Each convolutional layer is processed in the order of convolution, batch regularization, and activation. Step S3: The final fully connected layer is multiplied using 8-bit quantization precision before the decision is made and output. The convolution operation in the macro-convolutional layer is as follows: Map -1 in the feature map and weights to 0; Then, the bitwise XOR operation is used to replace the multiplication operation. Finally, the sum of the XOR results is multiplied by 2 and subtracted from the total length to obtain the convolution result. The batch regularization and activation in the macroconvolutional layer are combined to replace the multiplication and division operations in BN processing, as follows: The BN layer is processed in forward inference as follows: ； in This represents the mean of the input vector. Let x represent the standard deviation of the input vector, and y represent the Batch Normalization (BN) output. and These are the parameters learned during training, used to perform affine transformations on the activation values ​​to ensure the restoration of the original input; Activation function for forward inference: ； Combining BN layers and activation, we obtain ； in >0, and Given parameters, and ≠0, and remains unchanged during forward reasoning, therefore we obtain: ， ； That is when and Different calculation formulas are obtained at different times.

2. The method for implementing BNN forward inference on resource-constrained FPGAs according to claim 1, characterized in that, In step S1, the feature map and weight information are read directly from an external memory or external interface, and the weight information is binarized.

3. The method for implementing BNN forward inference on resource-constrained FPGAs according to claim 1, characterized in that, Step S2 involves normalizing the first-layer input image (Xr, Xg, Xb), removing the mean, and then performing 8-bit quantization, as detailed below: Xr1 = (Xr – min(Xr)) / (max(Xr) – min(Xr)); Xr2 = Xr1 - mean(Xr1); Xg1 = (Xg – min(Xg )) / (max(Xg ) – min(Xg )); Xg2 = Xg1 - mean(Xg1); Xb1 = (Xb – min(Xb )) / (max(Xb ) – min(Xb )); Xb 2 = Xb 1 - mean(Xb 1); Then, Xr2, Xg2, and Xb2 are subjected to signed 8-bit conventional linear quantization.

4. The method for implementing BNN forward inference on resource-constrained FPGAs according to claim 1, characterized in that, The fully connected layer performs 8-bit precision feature map and weight multiplication. The multiplication operation is performed by a DSP in the FPGA. By combining the input data, one DSP can perform the multiplication of two data simultaneously. For the same weight coefficient b, one DSP module can simultaneously perform the multiplication of two feature values, i.e., a1xb and a2xb, which are performed by one DSP module.

5. A BNN forward inference system adapted to resource-constrained FPGA implementation, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of a method for implementing BNN forward inference on a resource-constrained FPGA as described in any one of claims 1-4.

6. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1 to 4.

Images