An end-to-end CNN accelerator and pruning model compression method
By implementing a deviation-driven pruning model compression method and end-to-end CNN accelerator on FPGA, the problem of insufficient low-power hardware solutions in the prior art is solved, and efficient continuous attention level monitoring is achieved, reaching 84.2% Classification accuracy and 0.11W power consumption, suitable for portable BMI applications.
Patent Information
- Application Number
- CN202210289492.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-23
AI Technical Summary
The prior art lacks low-power hardware solutions for long-term, real-time continuous attention level monitoring, especially in portable BMI applications, and front-end neural network accelerator design is difficult to effectively compress models to reduce power consumption.
Adopt a pruning model compression method based on deviation-driven, including group convolution, global average pooling, near-zero pruning, deviation-driven pruning and weight clustering, and combine FPGA to realize end-to-end CNN accelerator to reduce model size and computational load .
It achieves a model compression ratio of 183.11 times and an attention-level classification accuracy of 84.2%, while achieving a design power consumption of 0.11W and an energy efficiency of 8.19GOps/W, making it suitable for portable BMI applications.
Smart Images

Figure CN115034353B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of CNN accelerators, and in particular to an end-to-end CNN accelerator and a pruning model compression method for evaluating sustained attention level using EEG. Background Art
[0002] Brain-computer interface (BMI) aims to establish an effective communication channel between the human brain and the outside world. Sustained attention level assessment is one of the most critical application scenarios of BMI. It can be used in ubiquitous scenarios such as the diagnosis of attention deficit hyperactivity disorder (ADHD) and classroom interaction efficiency.
[0003] Frontal electroencephalogram (EEG) is one of the common biomarkers in BMI or brain-computer interface (BCI) applications. A large number of literatures have explored the relationship between EEG and human mental state. Studies have found that both time domain features (such as mobility) and frequency domain features (such as band energy) are related to sustained attention levels. However, previous algorithms and systems are still based on manual feature design. Given that EEG is related to attention level, end-to-end neural network (NN)-based solutions are expected to achieve better attention level classification accuracy. In addition, although there are currently solutions to apply algorithms to mobile devices, there is still a lack of low-power hardware solutions for long-term, real-time and sustained attention level monitoring.
[0004] As neural network-based models have achieved great success in fields such as biomedical signal processing, the design of neural network accelerators has attracted more and more attention from researchers. Some researchers have proposed a biological neural network for emotion recognition based on EEG signals based on field programmable gate array (FPGA), which has a low power consumption of 0.15W. However, it uses a traditional scheme of separating feature extraction and classification modules. The feature extraction program runs offline, and CNN is only used as a classifier, making it unsuitable for the development of portable BMI. Other scholars have developed end-to-end CNNs for EEG-based epilepsy detection and electrocardiogram (ECG)-based heartbeat classification, respectively. The former achieved an accuracy of nearly 100% for epilepsy detection on hardware, but power consumption of more than 1W is still a problem in clinical practice. The lightweight one-dimensional CNN in the latter achieved high throughput efficiency and classification accuracy. However, it is a challenge to use a model with only 11k weights to handle more difficult tasks in biomedical signal processing. Summary of the invention
[0005] The objectives of the present invention are achieved through the following technical solutions.
[0006] According to a first aspect of the present invention, a bias-driven pruning model compression method is provided, which evaluates sustained attention level based on EEG, comprising:
[0007] The CNN model is subjected to group convolution, global average pooling, near-zero pruning, bias-driven pruning, weight clustering and quantization steps in sequence to obtain a compressed CNN model;
[0008] The CNN model consists of four convolution blocks, a global average pooling layer and a linear layer, and each convolution block includes a one-dimensional convolution layer, a one-dimensional batch normalization layer and a rectified linear unit layer.
[0009] Furthermore, the kernel size of each convolutional layer is 16.
[0010] Furthermore, the group convolution is as shown in equation (1),
[0011]
[0012] The boundary points s and e in grouped convolution are calculated by formulas (2) and (3), where g is the group number;
[0013]
[0014]
[0015] Cout and Cin represent the channel numbers of the output mapping and input mapping respectively, and g is a constant.
[0016] Furthermore, the near-zero pruning includes: removing a preset number of weights with smaller absolute values.
[0017] Furthermore, the bias-driven pruning includes: in the CNN model, three different thresholds are set for the first three batch normalization layers, and for each channel output by the batch normalization layer, when the BN deviation is less than the threshold, the convolution weight of the channel is pruned.
[0018] Furthermore, the weight clustering includes: performing K-means clustering on the weights of each convolutional layer.
[0019] Furthermore, the expression of the batch normalization layer is as follows:
[0020]
[0021] Among them, γ and β are two learnable parameters, E[χ] BN and Var[χ] BN Represents the mean and variance of each channel in the mini-batch.
[0022] According to a second aspect of the present invention, an end-to-end CNN accelerator for implementing the method of the first aspect is provided, wherein the accelerator is implemented by an FPGA and specifically comprises:
[0023] Controllers, on-chip random access memory, and processing engine arrays are interconnected in pairs.
[0024] Furthermore, the weights of the CNN network and the outputs of the first three convolution blocks are stored in an on-chip block random access memory, and the output map of the last convolution block is directly used for the global average pooling layer output; the inference of the four convolution blocks requires 5016, 5076, 5016 and 5024 processing engine cycles respectively, and the inference of the linear layer only requires one processing engine cycle.
[0025] Furthermore, the processing engine array includes 16 processing engines, each of which occupies 128 multipliers, 64 adders and 128 registers.
[0026] This paper proposes an end-to-end CNN architecture for continuous attention level assessment, which has the following advantages:
[0027] a) This paper proposes a bias-driven pruning model compression method. The compression process includes group convolution, global average pooling (GAP), near-zero pruning, bias-driven pruning, weight clustering and quantization, achieving a model compression ratio of 183.11 times.
[0028] b) The present invention achieves a subject-independent accuracy of 84.2% on the binary attention level classification task.
[0029] c) The present invention is implemented on FPGA with a design power consumption of 0.11 W and an energy efficiency of 8.19 GOps / W. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0031] Attached Figure 1 A schematic diagram of a CNN network architecture according to an embodiment of the present invention is shown.
[0032] Attached Figure 2 A schematic diagram showing the relationship between the BN deviation and the BN output negativity rate according to an embodiment of the present invention is shown.
[0033] Attached Figure 3 A schematic diagram of the hardware architecture of a CNN accelerator according to an embodiment of the present invention is shown.
[0034] Attached Figure 4 A schematic diagram showing the comparison of the effects between the present invention and several state-of-the-art attention evaluation works is shown.
[0035] Attached Figure 5 A schematic diagram showing the comparison of the effects of group convolution and global average pooling.
[0036] Attached Figure 6 A schematic diagram of the results of near-zero pruning (NZP) and bias-driven pruning (BDP) of the present invention is shown.
[0037] Attached Figure 7 A schematic diagram showing the comparison between the CNN accelerator of the present invention and several most advanced FPGA solutions in the field of physiological signal processing. DETAILED DESCRIPTION
[0038] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0039] 1. A bias-driven pruning model compression method
[0040] This paper first proposes a bias-driven pruning model compression method based on EEG to evaluate sustained attention level. The compression process includes group convolution, global average pooling (GAP), near-zero pruning, bias-driven pruning, weight clustering and quantization. The specific description of each step is as follows:
[0041] A. Group convolution, global average pooling
[0042] like Figure 1 As shown, the CNN network architecture of the embodiment of the present invention consists of four convolution blocks, a global average pooling layer and a linear layer. Each convolution block includes a one-dimensional convolution layer, a one-dimensional batch normalization (BN) layer and a rectified linear unit (ReLU) layer. The kernel size of each convolution layer is 16. In order to compress the CNN model, the group convolution is shown in equation (1), which is applicable to the last three convolution layers with group numbers of 8, 8 and 16.
[0043]
[0044] The boundary points s and e in grouped convolution are calculated by formulas (2) and (3), where g is the group number.
[0045]
[0046]
[0047] Cout and Cin represent the channel numbers of the output map and input map, respectively. The regular convolution layer is equivalent to the condition of g = 1 in equation (1). Since group convolution treats the input feature maps as separate groups, the number of weights and operations of the convolution layer can be reduced by g times.
[0048] The linear layer in CNN usually requires huge memory and computational load. In the CNN architecture of the present invention, a global average pooling layer is applied to avoid large linear layers. It can reduce the number of weights and operations of the linear layer to 1 / p, where p is the pool size.
[0049] B. Near-zero pruning and negative pruning
[0050] ReLU performs nonlinear reflection from negative values to zero, as shown in Equation 4:
[0051] ReLU(x)=max(x,0) (4)
[0052] This significantly increases the sparsity of the data flow in the forward propagation and makes it possible to use pruning methods such as zero pruning to reduce the model size and computational load. The present invention proposes two pruning strategies, namely near-zero pruning and bias-driven pruning. Since the global average pooling layer greatly reduces the size of the linear layer, the convolutional layer controls the parameters and number of operations of the CNN network, and its contribution rate is 97.11% and 97.82% respectively. Therefore, the present invention designs a pruning method for the weights of the convolutional layer.
[0053] 1) Near-zero pruning: In classification tasks, the accuracy depends only on the comparison results, not the exact value of the final linear layer output. It is feasible to remove a few weights with small absolute values, which are usually irrelevant to the model accuracy.
[0054] 2) Bias-driven pruning: This paper proposes an efficient pruning method based on BN layer bias. Since each BN layer is always followed by a ReLU layer, the negative output of the BN layer has no meaning for the network's inference results. Figure 2 As shown, the horizontal axis is the BN deviation value and the vertical axis is the negative output rate. In most BN layers, the negative rate of BN output is closely related to the BN deviation. Therefore, the BN deviation can be used as an indicator to determine whether to perform the corresponding convolution and BN. In the CNN network of the present invention, we set three different thresholds for the first three BN layers. For each channel of the BN output, when the BN deviation is less than the threshold, the convolution weight of the channel is pruned.
[0055] C. Weight Clustering and Quantization
[0056] Low-bit quantization can reduce the model's memory requirements and operation width, which will further improve power consumption and inference latency. After network pruning, we use weight clustering to reduce the bit width of the convolutional layer weights. K-means clustering is performed on the weights of each convolutional layer to obtain a sub-8-bit convolutional model.
[0057] For the quantization stage, there is a trade-off between model size and classification accuracy. We minimize the model width while keeping the classification accuracy of attention evaluation above 80%.
[0058] 2. CNN Accelerator Hardware Implementation
[0059] 1) Processing Engine: As shown in equation (1), one-dimensional convolution is equivalent to several inner product operations. The BN layer in the CNN network of the present invention is shown in equation (5).
[0060]
[0061] Among them, γ and β are two learnable parameters. E[χ] BN and Var[χ] BN Represents the mean and variance of each channel in a mini-batch, which are calculated and stored before the inference phase.
[0062] The processing engine (PE) architecture according to formula (6) is designed for one-dimensional convolutional blocks and linear layers.
[0063]
[0064] When used in a convolutional block, w represents the weight of the convolutional layer, while b and w BN Calculated by formulas (7) and (8):
[0065] b=bias-E BN (7)
[0066]
[0067] Where bias represents the convolutional layer bias. When the processing engine is used for inference of the linear layer, w and b in equation (6) represent the weight and bias of the linear layer, and w BN and β are set to 1 and 0, respectively.
[0068] The CNN accelerator hardware of the present invention is implemented using FPGA to implement the algorithm described above.
[0069] 2) Hardware architecture: The hardware architecture on FPGA is as follows Figure 3As shown, it specifically includes: controllers connected to each other in pairs, on-chip block random access memory, and processing engine arrays. The weights of the CNN network of the present invention and the outputs of the first three convolution blocks are stored in the on-chip block random access memory (BRAM). A total of 16 processing engines are implemented, and each processing engine (PE) occupies 128 multipliers, 64 adders, and 128 registers. According to the network configuration, the inference of the four convolution blocks requires 5016, 5076, 5016, and 5024 PE cycles, respectively. Due to the existence of the global average pooling layer, the inference of the linear layer only requires one PE cycle.
[0070] Due to the large number of convolution kernel channels and the long range of input EEG signals, the output map of each convolution block in the CNN network of the present invention occupies more than 1Mb of memory. It is necessary to apply a memory reuse scheme to the output map of the convolution block. To this end, the output maps of the first three convolution blocks occupy the same space of the on-chip BRAM. According to the coordinated design of the group number and the PE number, when the data slice is loaded into the processing engine, the PE result can be directly stored to the corresponding address without affecting subsequent calculations. The output map of the last convolution block is directly used for the global average pooling layer output. In order to take advantage of latency and power consumption, data loading of the BRAM is completed in the fast clock domain of 50MHz, while the processing engine is executed in the slow clock domain of 10MHz.
[0071] 3. Experimental Results
[0072] A. Data Collection
[0073] The present invention performs EEG acquisition on six subjects. A number clicking task is designed to mark the subject's sustained attention level. There is a 7×7 random number matrix on the screen. The subject is required to click on a number from 1 to any large number. Once the subject clicks on the target number, the number matrix is refreshed and the target number increases by 1. Each subject is required to complete a 12-minute trial. Each trial includes four 3-minute stages. In the first and third stages, the subject is required to relax as much as possible, while in the second and fourth stages, the subject is required to complete the number clicking task. The data collected from the number clicking stage is marked as a high attention level. In the designed 12-minute trial, seven prefrontal electroencephalogram (EGG) channels of each subject are recorded. The EEG acquisition device is a module based on a neural signal recorder.
[0074] B. Sustained attention level assessment
[0075] During the evaluation process, we randomly selected one subject as the test set and the other five subjects as the training set. After training with a batch size of 16 times, the CNN network of the present invention achieved a subject-independent (SI) accuracy of 89.8%. In order to meet the requirements of real-time processing, we fixed the running mean and variance of the BN layer and evaluated the CNN network of the present invention using a batch size of 1, and still achieved a high SI accuracy of 85.9%. Figure 4 The comparison between the proposed CNN network of the present invention and several state-of-the-art attention evaluation works is shown. Compared with other works, the proposed CNN network provides an end-to-end CNN solution on FPGA with high SI accuracy and is more likely to be integrated into portable BMI.
[0076] C. Compression of CNN Network
[0077] 1) Group convolution and global average pooling layer: Figure 5 As shown, the group convolution and global average pooling layers compress the model size by 13.9 times without significant degradation of the evaluation accuracy. It should be emphasized that the global average pooling layer not only reduces the model size, but also suppresses the overfitting problem to ensure that the CNN network of the present invention achieves a higher evaluation accuracy.
[0078] 2) Pruning and quantization: The results of near-zero pruning (NZP) and bias-driven pruning (BDP) are shown in Figure 6 As shown in Figure 2, the near-zero pruning thresholds of the four convolutional layers are set to 0.017, 0.014, 0.015, and 0.014, respectively. The average compression ratio reaches 2.36 times. Figure 2 In the analysis in , bias-driven pruning is used on the first three convolutional blocks with thresholds of -0.061, -0.046, and -0.183, respectively. The results show that after near-zero pruning, the proposed bias-driven pruning can further reduce 2.62k parameters, corresponding to a compression ratio of 1.26x. There is no loss in accuracy in the pruning step.
[0079] After weight clustering and quantization, the bit widths of convolution weights, convolution biases, BN weights, BN biases, linear weights, and linear biases are set to 7, 8, 16, 14, 8, and 11, respectively. This step achieves a compression ratio of 4.43x compared to the standard 32-bit representation of non-zero network parameters. During the quantization phase, the accuracy drops from 85.9% to 84.2%, which is still reliable for attention evaluation applications.
[0080] and Figure 5 Compared with the baseline model in , the total compression ratio of the present invention is 183.11 times after group convolution, gap, near-zero pruning, bias-driven pruning, weight clustering and quantization.
[0081] D. Hardware Implementation
[0082] The CNN network of the present invention consumes 65731 LUTs, 35199 FFs, 736 DSPs and 68 BRAMs on a Xilinx Artix-7 FPGA using the Vivado 2017.4 tool. Figure 7 The figure shows the comparison between the CNN accelerator of the present invention and several state-of-the-art FPGA solutions in the field of physiological signal processing. Since the sustained attention level assessment task is insensitive to sub-second delays, the CNN network of the present invention makes a compromise between delay and power consumption. Compared with other solutions, the CNN network of the present invention has a competitive design power consumption of 0.11W and an energy efficiency of 8.19GOps / W.
[0083] IV. Conclusion
[0084] This paper proposes an end-to-end CNN network for sustained attention level assessment using 5-channel prefrontal EEG. The CNN network of the present invention is compressed 183.11 times using group convolution, gap, near-zero pruning, weight clustering, quantization, and suggested bias-driven pruning. The compressed CNN network of the present invention achieves a state-of-the-art subjective independent sustained attention level classification accuracy of 84.2%. This is the first FPGA-based end-to-end CNN solution for sustained attention level assessment, which consumes a design power of 0.11W and an energy efficiency of 8.19Gops / W.
[0085] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. An end-to-end CNN accelerator for performing a bias-driven pruning model compression method, characterized in that: The sustained attention level is evaluated based on EEG, and the accelerator is implemented by FPGA, specifically including: Controllers, on-chip block random access memories, and processing engine arrays connected to each other in pairs; The weights of the CNN network and the outputs of the first three convolutional blocks are stored in the on-chip block random access memory, and the output map of the last convolutional block is directly used for the global average pooling layer output; the inference of the four convolutional blocks requires 5016, 5076, 5016 and 5024 processing engine cycles respectively, and the inference of the linear layer only requires one processing engine cycle; The processing engine array includes 16 processing engines, each of which occupies 128 multipliers, 64 adders and 128 registers; The pruning model compression method performed by the accelerator comprises the following steps: The CNN model is subjected to group convolution, global average pooling, near-zero pruning, bias-driven pruning, weight clustering and quantization steps in sequence to obtain a compressed CNN model; The CNN model consists of four convolution blocks, a global average pooling layer and a linear layer, each convolution block includes a one-dimensional convolution layer, a one-dimensional batch normalization layer and a rectified linear unit layer; The batch normalization layer in the CNN model is shown in equation (5): Among them, γ and β are two learnable parameters; E[χ] BN and Var[χ] BN Represents the mean and variance of each channel in the mini-batch, which are calculated and stored before the inference phase; The processing engine architecture according to formula (6) is designed for one-dimensional convolutional blocks and linear layers; When used in a convolutional block, w represents the weight of the convolutional layer, while b and w BN Calculated by formulas (7) and (8): bibles-E BN (7) Where bias represents the convolutional layer bias; when the processing engine is used for inference of the linear layer, w and b in equation (6) represent the weight and bias of the linear layer, and w BN and β are set to 1 and 0, respectively.
2. The CNN accelerator according to claim 1, characterized in that The kernel size of each convolutional layer is 16.
3. The CNN accelerator according to claim 2, characterized in that The group convolution is shown in equation (1), The boundary points s and e in grouped convolution are calculated by formulas (2) and (3), where g is the group number; Cout and Cin represent the channel numbers of the output mapping and the input mapping respectively, and g is a constant.
4. The CNN accelerator according to claim 3, characterized in that The near-zero pruning includes: removing a preset number of weights with smaller absolute values.
5. The CNN accelerator according to claim 4, characterized in that: The bias-driven pruning includes: in the CNN model, three different thresholds are set for the first three batch normalization layers, and for each channel output by the batch normalization layer, when the BN deviation is less than the threshold, the convolution weight of the channel is pruned.
6. The CNN accelerator according to any one of claims 1 to 5, characterized in that: The weight clustering includes: performing K-means clustering on the weights of each convolutional layer.
Citation Information
Patent Citations
Convolution streaming engine for deep neural networks
US20200349426A1
A neural network model for cochlear mechanics and processing
WO2020249532A1
Neural network search method and apparatus
WO2021018163A1
