Brain-computer interface classification system and method based on FPGA acceleration

Abstract

The invention discloses a brain-computer interface classification system and method based on FPGA acceleration, and relates to the field of brain-computer interface classification. The method solves the problems that in the prior art, classification accuracy of single-mode electroencephalogram signals is low, feature information contained in the single-mode signals is little and not comprehensive, classification performance cannot be well achieved, and edge end deployment resources are limited. The method comprises the steps that preprocessing and feature extraction are conducted on the electroencephalogram signals and functional near-infrared spectrum signals, a multi-modal brain-computer interface classification network is established, lightweight compression is performed on a network model, and a lightweight method of reinforcement learning channel selection, QAT quantization, channel pruning and BN layer fusion is adopted in training, so that model parameter quantities are remarkably compressed on the premise of ensuring model classification precision. And finally, on the basis of an FPGA accelerated brain-computer interface classification system, an interlayer pipeline design is adopted, so that the delay of network reasoning acceleration is reduced. Reasoning of a network model after light weight is accelerated, and a real-time low-delay and low-power-consumption brain-computer interface classification method is achieved at the edge end.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface classification technology, specifically to a brain-computer interface classification system and method based on FPGA acceleration. Background Technology

[0002] With the improvement of computing power and storage, deep learning algorithms have ushered in unprecedented development, among which Convolutional Neural Networks (CNNs) are widely used in feature extraction, signal classification, and object detection. Especially in the field of Brain-Computer Interface (BCI), multimodal signal classification based on electroencephalograms (EEG) and functional near-infrared spectroscopy (fNIRS) has become a research hotspot for BCI systems. To realize the widespread application of BCI systems in real-world scenarios, it is crucial to transfer the analysis and processing of EEG and fNIRS data from offline to online environments. Therefore, deploying neural networks on edge devices to build BCI classification systems is essential. However, central processing units (CPUs) cannot handle such computationally intensive model inference; graphics processing units (GPUs) have power consumption limitations for edge device scenarios; and application-specific integrated circuits (ASICs) hardware design suffers from long development cycles. In contrast, Field Programmable Gate Arrays (FPGAs) can meet the requirements of massive parallel computing for convolutional neural networks, while also being adaptable to low-power, high-efficiency edge device scenarios, thus offering greater flexibility.

[0003] To address the issue of low classification accuracy of single-modal EEG signals, which suffer from limited and incomplete feature information and thus poor classification performance, as well as resource constraints at the edge, and the large number of parameters in convolutional neural networks requiring additional off-chip storage resources, this invention proposes a brain-computer interface classification system and method based on FPGA acceleration. This effectively solves the problems of low power consumption and low latency in implementing brain-computer interface classification systems on edge devices. Summary of the Invention

[0004] To address the problem of low classification accuracy of single-modal EEG signals in existing technologies, this invention proposes a brain-computer interface classification system and method based on FPGA acceleration. Single-modal signals contain limited and incomplete feature information, which cannot achieve good classification performance. Furthermore, the limited resources of edge devices and the large number of parameters in convolutional neural networks require a large amount of additional off-chip storage resources, leading to bandwidth limitations in memory access.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Option 1: This invention proposes a brain-computer interface classification system based on FPGA acceleration, the system comprising: The peripheral data path module includes an AXI-DMA interface and a DRAM interface. The AXI-DMA interface is used to transfer feature maps and weights to the DDR3 interface mounted on the PL end via the AXI interface and the host ARM processor. The off-chip memory controller is used to control the distribution of weights to each convolutional layer. The AXI interface is used to access the off-chip DRAM interface via the DDR3 interface for weight transfer. The feature extraction module consists of cascaded convolutional processing units with multiple convolutional kernels of different sizes. Each convolutional processing unit integrates convolution, activation, and caching functions. The classification module, including fully connected layers, activation functions, and Softmax, is used to transform the extracted high-level features into the final classification result. The convolution module is used to perform convolution operations on the weight cache data of each layer of the feature extraction module and the classification module, and return the operation results to the off-chip DRAM interface through the storage controller; The fusion module is used to send the fused feature results to the classification module, and the final classification result is displayed on the ARM terminal.

[0006] Furthermore, a preferred embodiment is provided, wherein the method for performing convolution operations on the weight cache data of each layer of the feature extraction module and the classification module in the convolution module is as follows: Feature map data is piped through a three-level row buffer. Taking a 3*3 convolution window as an example, when a new data row is loaded from the ping-pong buffer to the first-level buffer, the previous data row is entering the convolution window, and the historical data rows have already been multiplied in the computation unit. In the parallel multiplication array, the 9 pixels of the input window and the 9 feature values ​​preloaded in the weight cache are multiplied synchronously. After optimization and accumulation by the addition tree, the multiplication result is used to obtain the convolution output through the three-level accumulation architecture.

[0007] Option 2: A brain-computer interface classification method based on FPGA acceleration, the method being implemented based on the classification system described in Option 1, and the method comprising the following steps: Step 1: Preprocess and extract features from EEG signals and functional near-infrared spectroscopy signals to build a multimodal brain-computer interface classification network; Step 2: Based on the multimodal brain-computer interface classification network built in Step 1, perform lightweight compression on the constructed multimodal brain-computer interface classification network model; Step 3: An FPGA-accelerated brain-computer interface classification system is developed. The interlayer pipeline design method is used to accelerate the inference of the lightweight compressed multimodal brain-computer interface classification network model in Step 2, and the brain-computer interface system is implemented on the edge device.

[0008] Furthermore, a preferred embodiment is provided, wherein the method for lightweight compression of the multimodal brain-computer interface classification network model constructed in step 1 in step 2 includes the following steps: Step 2.1: Channel pruning based on BN layer. Select the channels that need to be pruned in each layer according to the BN layer scaling factor and the set pruning rate, and fine-tune the training to restore accuracy. Step 2.2: Extract BN layer parameters from the trained multimodal brain-computer interface classification network model. The BN layer parameters include: , , and For each channel of the convolutional layer, the fused weights and biases are calculated separately. Step 2.3: Based on the feature channel selection algorithm of reinforcement learning, after 30 rounds of pre-training in the same time window, the models with and without reinforcement learning are trained to 100 rounds respectively. Step 2.4: Use QAT quantization during training to fine-tune the quantized network, so that the parameters of the multimodal brain-computer interface classification network model can adapt to the quantization error.

[0009] Furthermore, a preferred embodiment is provided, wherein the method for calculating the fused weights and biases in step 2.2 is as follows:

[0010]

[0011] in, The weights of the convolutional layer For bias, For the new weights, For the new bias, and For trainable parameters, It is a constant.

[0012] Furthermore, a preferred embodiment is provided in which the feature channel selection algorithm based on reinforcement learning in step 2.3 further includes the step of adding an agent between the feature extraction and classification networks. The agent is used for feature selection, and adds a mask of 0 to the features that are not selected. This channel does not participate in the pooling operation.

[0013] Furthermore, a preferred embodiment is provided, wherein the QAT quantization method used in step 2.4 is as follows:

[0014] Where q is an integer, r is a real number, and S is a scaling constant. Z It is a zero-point constant.

[0015] Furthermore, a preferred embodiment is provided, wherein the interlayer pipeline design method in step 3 includes the following steps: Step 3.1: After implementing the address control function of each layer by pre-setting the access address offset, the accelerator reads memory data, processes the data, and writes the data back to memory according to the relevant parameters; Step 3.2: When reading or writing multiple blocks of data, the data blocks cached on the FPGA are reused before the next read is performed, ensuring that each block of data is read from off-chip only a small number of times to reduce memory access frequency and data transfer volume; Step 3.3: For each piece of data retrieved, read the data from outside the chip into the input buffer, and process it according to the layer type by the convolution, pooling, and fully connected modules; write the processed data back to the output buffer, and write it back to the external storage after obtaining the final result.

[0016] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 2.

[0017] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 2.

[0018] The advantages of this invention are: This invention discloses a brain-computer interface (BCI) classification system and method based on FPGA acceleration. First, preprocessing and feature extraction of EEG and functional near-infrared spectroscopy signals are performed to construct a multimodal BCI classification network. Further, the network model is lightweighted and compressed. Specifically, during training, a lightweight method is employed, including reinforcement learning channel selection, QAT quantization, channel pruning, and BN layer fusion, significantly reducing the number of model parameters while maintaining classification accuracy. Finally, an accelerator architecture is explored on the FPGA side, employing an inter-layer pipeline design, which significantly reduces latency in network inference acceleration. Accelerating the inference of the lightweight network model enables a real-time, low-latency, low-power BCI classification algorithm at the edge.

[0019] This invention designs a lightweight algorithm for classification network models, specifically consisting of pruning and quantization. Addressing the limitations of other hardware such as CPUs, GPUs, and ASICs in deploying specific neural networks, FPGAs, with their extremely high parallelism, high energy efficiency, and low development cost, are the preferred choice for this invention. Finally, by building a hardware-software interactive environment, communication between the ARM (Advanced RISC Machines) core and the FPGA is achieved, and hardware implementation and system verification are completed on a ZYNQ 7015. Attached Figure Description

[0020] Figure 1 This is a flowchart of the BN layer pruning process described in this invention.

[0021] Figure 2 This is a schematic diagram of the channel selection strategy based on reinforcement learning described in this invention.

[0022] Figure 3 This is a flowchart of the channel selection process based on reinforcement learning as described in this invention.

[0023] Figure 4 This is a flowchart of the QAT quantization training process described in this invention.

[0024] Figure 5 This is a schematic diagram illustrating the principle of the FPGA-accelerated brain-computer interface classification system described in this invention.

[0025] Figure 6 This is a flowchart of the convolution parallel multiply-accumulate unit described in this invention.

[0026] Figure 7 This is a flowchart illustrating the design of the convolution row buffer pipeline described in this invention.

[0027] Figure 8 This is a flowchart illustrating the design of the pooled row buffer pipeline described in this invention.

[0028] Figure 9 This is a flowchart of the ReLU calculation unit described in this invention.

[0029] Figure 10 This is a schematic diagram comparing the convergence performance of the reinforcement learning-based model described in this invention.

[0030] Figure 11 This is a schematic diagram illustrating the influence of the quantization bit depth on the detection results as described in this invention.

[0031] Figure 12 This is a schematic diagram of the execution scheduling process described in this invention.

[0032] (a) is a schematic diagram between layers, (b) is a schematic diagram between blocks, and (c) is a schematic diagram within a block.

[0033] Figure 13 This is a flowchart illustrating the hardware quantization process described in this invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0035] Implementation Method 1: This implementation method proposes a brain-computer interface classification method based on FPGA acceleration. The method specifically includes: I. Compression Design of Classification Network Model To compress parameters in a classification network model, the first pruning method used is as follows: Figure 1 As shown, by setting an initial pruning rate, the scaling factor is associated with each channel in the convolutional layer. Channels that need to be pruned in each layer are identified according to the scaling factor, sorted from smallest to largest. Specifically, a threshold is selected in the scaling factor sorting based on the pruning rate. When the scaling factor γ is larger than the threshold, it indicates that the channel needs to be retained, and the mask is set to 1; otherwise, the channel is pruned, and the mask is set to 0. After all channels have been pruned, a new model structure is constructed based on the masks of each layer. The pruned model is then fine-tuned to restore accuracy. If suitable accuracy is not achieved, the pruning rate is lowered, and the pruning process is repeated.

[0036] Batch Normalization (BN) layer fusion refers to merging the parameters of the BN layers in a trained neural network into the parameters of the preceding convolutional (or fully connected) layer, thereby simplifying the inference process and improving computational efficiency. This is especially important during the inference stage because it reduces computational and memory access requirements. The following are the principles and specific steps of BN layer fusion, including normalization:

[0037] in These are input features. and These are the mean and variance of the mini-batch input, respectively. It is a small constant to prevent the denominator from being zero. Scaling and translation:

[0038] in, and These are trainable parameters used to recover the model's expressive power.

[0039] Extracting parameters from the BN layer of the trained model: , , and For each channel of the convolutional layer (or fully connected layer), calculate the fused weights and biases separately. Assume the weights of the convolutional layer are... , bias is Then there will be new weights. and new bias By fusing BN layers, the model can run more efficiently during the inference phase while maintaining the accuracy during the training phase.

[0040]

[0041]

[0042] According to the network used in this embodiment, after multiple convolutional operations, it needs to pass through an average pooling layer, and then the pooled features are fed into a fully connected layer for classification. This embodiment uses a reinforcement learning-based method to select pooling channels, which not only reduces the number of pooling channels and computational load to speed up inference, but also accelerates the network's convergence speed. Specifically, as... Figure 2 As shown, during network training, an agent is added between the feature extraction and classification networks for feature selection. Unselected features are masked with 0, preventing those channels from participating in subsequent pooling operations. Taking the pooling layer of the EEG feature extraction network as an example, previously, each average pooling operation required processing 30 feature vectors. After channel selection through reinforcement learning, feature vectors with distinct characteristics can be selected. This reduces the computational cost of pooling operations and allows the model to converge faster.

[0043] The basic flowchart of the feature selection algorithm is as follows: Figure 3 The agent is controlled by a policy network π, interacts with the environment, and collects state (classification accuracy after masking some channels), action (whether to mask the current channel), and reward (the difference in classification accuracy between masked and unmasked channels). The policy network is state-based.s t (Classification accuracy after masking some channels) Take action a t (Whether to mask the current channel: 0 / 1). Value network is based on... s t Next moment state s t+1 Rewards in the current state r t Calculate the TD error δ t (Reward obtained after performing the action + value of the next state - value of the current state). Policy network (relying on...) δ t To judge the quality of one's own actions, one can improve one's acting skills (i.e., parameter θ) and obtain the optimal strategy π, which can maximize value (classification accuracy).

[0044] In addition, to further compress the network model, this implementation uses QAT quantization, the process of which is as follows: Figure 4 As shown. The quantization scheme is integer. q to real numbers r Affine mapping, i.e.

[0045] For 8-bit quantization q Quantized to an 8-bit integer (for B-bit quantization, q (Quantized to B-bit integers). Typically, it's a bias vector, quantized to a 32-bit integer, a constant. S (Scale) is an arbitrary positive real number. It is usually represented as a floating-point number in software, such as a real value. r .constant Z (Zero point) and quantized value q Belonging to the same type, they are actually quantized values ​​corresponding to the real value 0. q This enables us to automatically satisfy real values. r =0 can be precisely represented by a quantized value.

[0046] set up q For quantized fixed-point integers, r For floating-point numbers before quantization, the index w Represents weight correlation, subscript in Represents the correlation with the input feature map, subscript out Represents the output feature map, subscript bias Represents bias correlation; S and Z These represent the parameters used for mapping during the quantization process. The floating-point convolution operation can be represented as:

[0047] The quantization process for floating-point values ​​involved is represented as follows:

[0048]

[0049]

[0050]

[0051] The convolution operation involving quantization can then be represented as:

[0052] Because we need to use the quantized weights, input feature vectors, and biases to calculate the quantized output feature vector, we need to represent the quantized output result:

[0053] Using symmetric quantization ( Z =0) can reduce computational complexity, which can be obtained by expansion.

[0054] in, , , They are all of type int8. It is of type int32.

[0055] This implementation optimizes model compression, significantly reducing the number of parameters in the classification network. The overall model compression process is as follows: 1. Channel pruning based on BN layers: Identify the channels to be pruned in each layer based on the BN layer scaling factor and the set pruning rate, and fine-tune training to restore accuracy. 2. BN layer fusion: Fuse the BN layer parameters with the parameters of the previous convolutional layer. 3. Feature channel selection based on reinforcement learning: After 30 pre-training epochs within the same time window, train for 100 epochs with and without reinforcement learning respectively. The results are as follows: Figure 10 As shown, this demonstrates that introducing reinforcement learning for pooling channel selection enables the network to achieve faster convergence. 4. During training, QAT quantization is used to fine-tune the quantized network, allowing the model parameters to gradually adapt to the quantization error, resulting in better inference performance of the quantized model. This implementation explored the performance of various quantization strategies under Pot quantization in multimodal classification tasks, involving symmetric quantization, asymmetric quantization, layer-by-layer quantization, channel-by-channel quantization, PTQ quantization, QAT quantization, and a comparison of quantization bit counts. Experimental results are shown below. Figure 11 As shown.

[0056] Implementation Method 2: This implementation method proposes a brain-computer interface classification system based on FPGA acceleration. The system specifically includes: The classification system architecture designed in this embodiment is as follows: Figure 5 As shown, the system mainly consists of an external data path, a feature extraction module, and a classification module. The external data path comprises AXI DMA (Advanced eXtensible Interface Direct Memory Access) and DRAM (Dynamic Random Access Memory). AXI DMA uses the AXI interface to transfer feature maps and write weight data to the ARM processor (ARM Cortex-A9) on the PS side of the ZYNQ 7015 host, and to the DDR3 (Double Data Rate 3) mounted on the PL side. The off-chip memory controller is responsible for controlling the distribution of weights to each convolutional layer. The AXI bus accesses the off-chip DRAM through the DDR3 interface for weight transfer.

[0057] The feature extraction module is structured as follows. This module is the core computational unit of the CNN accelerator, consisting of multiple cascaded convolutional processing units (CONVs) with different kernel sizes (such as CONV9, CONV5, and CONV3). Each convolutional processing unit integrates functions such as convolution, activation, and caching.

[0058] The classification module, located after the feature extraction module, is responsible for transforming the extracted high-level features into the final classification result. It mainly consists of fully connected layers, activation functions, and a softmax layer.

[0059] During initialization, the trained weights are preloaded into DRAM via an SD card and then onto the ARM processor. During inference, the input feature map is first sent to the input buffer via AXI DMA. After computation begins, the storage controller distributes the weight data stored in the off-chip DRAM to the weight buffers of each layer in the feature extraction and classification modules. Due to the large volume of input feature map data, it is sent to the feature map buffer of the feature extraction module in blocks. After convolutional operations, the results are returned to the off-chip DRAM via the storage controller for the next layer to read. Finally, after each convolutional operation, the fusion module sends the fused feature results to the classification module, and the final classification result is displayed on the ARM processor.

[0060] (1) Convolutional module design Because convolution involves a large number of multiply-accumulate (MAC) operations, and FPGA on-chip storage and computing resources are limited, it is necessary to rationally design the parallelism of data multiplication and accumulation operations during convolution to improve computation speed. Therefore, for each layer of convolution, only on-chip caching is required. Group convolution kernels (due to the ping-pong buffer, the actual memory size will be designed to be...) In each cycle, the convolution processing unit retrieves data from the input buffer. n One input is retrieved from the weight cache. Each weight, using DSP, calculated An intermediate result. For example... Figure 6 The diagram illustrates the computation process of a parallel multiply-accumulate unit in convolution. (Example) a This indicates that the input feature map was retrieved from the buffer. n One input, b and c This indicates that it was retrieved from the weight cache. Each weight is multiplied, and the result is passed through an addition tree to obtain intermediate results for each step. These intermediate results are then accumulated, and the process is repeated for all input channels. N Finally, the final convolutional output is obtained.

[0061] Line buffer hierarchical pipeline design, such as Figure 7 As shown in the blue area, the feature map data undergoes pipelined processing through a three-level row buffer. When a new data row (Row) is added... N +1) When loading from the ping-pong buffer to the first buffer, the preceding data row (Row) N The data is entering the convolution window, while the earlier data rows (Row) are... N -1) Multiplication has already been performed in the computation unit. A parallel multiplication array is used, where the 9 pixels of the input window are multiplied synchronously with the 9 pre-loaded feature values ​​in the weight cache (green area in the diagram). Finally, after optimization and accumulation by an addition tree, the multiplication results are processed through a three-level accumulation architecture to obtain the convolutional output.

[0062] (2) Subsidiary layer unit design The pooling method used in this implementation is average pooling. The data to be pooled is divided into channels and fed into the designed average pooling unit, with pooling operations performed in parallel for the number of input channel blocks. The average pooling design implemented on FPGA employs a multi-channel parallel processing architecture, as follows: Figure 8As shown, each independent channel first performs pipelined buffering on the input feature map, extracting four pooling unit input channels in parallel from the pipeline and feeding them into the addition tree module. However, this addition tree adds an average calculation module after each addition, where the pooling window size and stride are configurable. The multi-dimensional features after pooling are fused element-wise in the accumulation unit. If the pipeline input ends, the final average calculation is enabled by a termination signal, ultimately obtaining the average pooling result of one input channel. This architecture significantly improves throughput through channel-level parallelism and pipeline design. The feature map buffering mechanism effectively balances data reuse rate and storage bandwidth requirements, making it particularly suitable for the real-time processing needs of high-dimensional feature maps in convolutional neural networks. In hardware implementation, shift-and-add operations can be used instead of division operations to optimize resource utilization while maintaining numerical accuracy.

[0063] The off-chip storage design addresses the need for block-based input feature map processing, such as reading 8 (input channel blocks) × 8 (output channel blocks) × 900 (input feature map blocks) weights. The high bandwidth transmission of the weight matrix far outweighs the computational latency, making address rearrangement for off-chip transmission essential. To reduce the additional data routing logic required to fill the buffer while maintaining a longer burst length when fetching data for the fully connected layer, we rearranged the weight matrix in external memory. We first divided the entire matrix into blocks of 64 × 9 columns and 100 rows, allowing a block to be processed within a single stage. Without this rearrangement, as shown in the figure, loading a single block would require 64 × 100 Direct Memory Access (DMA) events, with a burst length of only 9. By rearranging the data, we can load the entire block with a single DMA transaction, and the longer burst length ensures high utilization of external memory bandwidth.

[0064] The activation function is designed as follows. Since convolutional layers may be followed by activation layers to perform non-linear transformations on the data, the activation units need to be designed after obtaining the convolution result. This network uses the ReLU activation function, so a selector can be used to handle this logic. For example... Figure 9 As shown, the convolution result is split according to the output parallelism to obtain the convolution value of each output channel. The sign bit of each convolution value can be used as the selection signal of the selector. When the sign bit is 1, it indicates that the convolution value is negative, and the selector chooses to output 0; if it is 0, it indicates that the convolution value is positive, and the selector outputs the data itself, realizing hardware pipeline activation.

[0065] The inter-layer pipeline operation process is illustrated with an example: This implementation adopts an inter-layer pipeline structure, where the output of the previous layer serves as the input of the current layer, and after processing, it is passed to the next layer. To maximize the parallel computing efficiency of the FPGA and reduce off-chip memory access overhead, this system employs a block data reuse strategy, combined with multi-level cache management and modular computing units, to achieve efficient data handling and computation scheduling. The specific process is as follows: Figure 12 As shown: 1. After implementing the address control function of each layer by pre-setting the access address offset, the FPGA accelerator only needs to read memory data, process data, and write data back to memory according to relevant parameters, such as... Figure 12 As shown in (a). 2. As Figure 12 As shown in (b), each time multiple blocks of data are read or written, the data blocks in the FPGA on-chip cache are reused before the next read is performed, ensuring that each block of data is read from off-chip only a few times to reduce memory access frequency and data transfer volume. 3. For each retrieved block of data, according to Figure 12 (c) Control flow processing: Read chunks of data from off-chip into the input buffer, and then process them according to the layer type using convolution, pooling, and fully connected modules. The processed data is written back to the output buffer, and finally written back to off-chip storage after the final result is obtained.

[0066] Example of convolution module operation process: The operation flow of the convolution module in this embodiment is as follows: Figure 13 The steps are as follows: 1. Weight quantization: First, the weights of the trained FP32 model are converted to INT8 format through post-training quantization or quantization-aware training. Practice shows that properly processed 8-bit weights typically result in minimal inference accuracy loss compared to the original FP32 weights. 2. Convolution calculation and bit-width expansion: The quantized INT8 weights are convolved with the input feature map, which has also been quantized to INT8. 3. To prevent overflow of intermediate results, the bit width of the product sum is usually expanded to 16 bits or even 32 bits. 4. Bias addition: The bias is usually quantized to a 32-bit integer and then added to the convolution result after bit-width expansion. 5. Shift quantization: The 32-bit integer obtained after convolution and adding the bias needs to be scaled back to the INT8 range to be used as input for the next layer. This step is often achieved through efficient shift operations, thus avoiding floating-point operations. The scaling ratio is determined by the quantization parameters of the layer's input and weights. 6. Overflow handling and activation functions: If the value exceeds the range of INT8, it needs to be truncated to limit it to the boundary value; ReLU activation: If the ReLU activation function is used, it will set negative values ​​to zero.

[0067] In summary, this implementation proposes a solution for a brain-computer interface (BCI) classification system. First, preprocessing and feature extraction are performed on EEG and functional near-infrared spectroscopy signals to construct a multimodal BCI classification network. Further, the network model is lightweighted and compressed. During training, a lightweight method is employed, including reinforcement learning channel selection, QAT quantization, channel pruning, and BN layer fusion, significantly reducing the number of model parameters while maintaining classification accuracy. Finally, an accelerator architecture is explored on the FPGA side, employing an inter-layer pipeline design, which significantly reduces latency in network inference acceleration. This accelerates the inference of the lightweight network model, enabling a real-time, low-latency, low-power BCI classification algorithm at the edge.

[0068] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0069] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A brain-computer interface classification system based on FPGA acceleration, characterized in that, The system includes: The peripheral data path module includes an AXI-DMA interface and a DRAM interface. The AXI-DMA interface is used to transfer feature maps and weights to the DDR3 interface mounted on the PL end via the AXI interface and the host ARM processor. The off-chip memory controller is used to control the distribution of weights to each convolutional layer. The AXI interface is used to access the off-chip DRAM interface via the DDR3 interface for weight transfer. The feature extraction module consists of multiple cascaded convolutional processing units with different kernel sizes. Each convolutional processing unit integrates convolution, activation, and caching functions. The classification module, including fully connected layers, activation functions, and Softmax, is used to transform the extracted high-level features into the final classification result. The convolution module is used to perform convolution operations on the weight cache data of each layer of the feature extraction module and the classification module, and return the operation results to the DRAM interface through the storage controller; The fusion module is used to send the fused feature results to the classification module, and the final classification result is displayed on the ARM terminal.

2. The FPGA-accelerated brain-computer interface classification system according to claim 1, characterized in that, The method for performing convolution operations on the weight cache data of each layer in the feature extraction module and the classification module in the convolution module is as follows: Feature map data is piped through a three-level row buffer. When a new data row is loaded from the ping-pong buffer to the first-level buffer, the previous data row is entering the convolution window, and the historical data rows have already been multiplied in the computation unit. The parallel multiplication array performs multiplication operations synchronously on the 9 pixels of the input window and the 9 feature values ​​preloaded in the weight cache. After optimization and accumulation using an addition tree, the multiplication result is processed through a three-level accumulation architecture to obtain the convolution output.

3. A brain-computer interface classification method based on FPGA acceleration, characterized in that, The method is implemented based on the classification system of claim 1, and the method includes the following steps: Step 1: Preprocess and extract features from EEG signals and functional near-infrared spectroscopy signals to build a multimodal brain-computer interface classification network; Step 2: Based on the multimodal brain-computer interface classification network built in Step 1, construct a multimodal brain-computer interface classification network model and perform lightweight compression; Step 3: The FPGA-accelerated brain-computer interface classification system uses an interlayer pipeline design method to accelerate the inference of the lightweight compressed multimodal brain-computer interface classification network model in Step 2, and realizes the classification of brain-computer interfaces at the edge.

4. The FPGA-accelerated brain-computer interface classification method according to claim 3, characterized in that, The method for lightweight compression of the multimodal brain-computer interface classification network model built in step 1 in step 2 includes the following steps: Step 2.1: Channel pruning based on BN layer. Select the channels that need to be pruned in each layer according to the BN layer scaling factor and the set pruning rate, and fine-tune the training to restore accuracy. Step 2.2: Extract BN layer parameters from the trained multimodal brain-computer interface classification network model. The BN layer parameters include: , , and For each channel of the convolutional layer, the fused weights and biases are calculated separately. Step 2.3: Based on the reinforcement learning-based feature channel selection algorithm, after 30 rounds of pre-training within the same time window, both with and without reinforcement learning are trained to 100 rounds. Step 2.4: Use QAT quantization during training to fine-tune the quantized network, so that the parameters of the multimodal brain-computer interface classification network model can adapt to the quantization error.

5. The FPGA-accelerated brain-computer interface classification method according to claim 3, characterized in that, The method for calculating the fused weights and biases in step 2.2 is as follows: in, The weights of the convolutional layer For bias, For the new weights, For the new bias, and For trainable parameters, It is a constant.

6. The FPGA-accelerated brain-computer interface classification method according to claim 3, characterized in that, The feature channel selection algorithm based on reinforcement learning in step 2.3 also includes adding an agent between the feature extraction and classification networks. The agent is used for feature selection, adding a mask of 0 to features that are not selected. This channel does not participate in pooling operations.

7. The FPGA-accelerated brain-computer interface classification method according to claim 6, characterized in that, The method for using QAT quantization in step 2.4 is as follows: Where q is an integer, r is a real number, and S is a scaling constant. Z It is a zero-point constant.

8. The FPGA-accelerated brain-computer interface classification method according to claim 3, characterized in that, Step 3, the interlayer pipeline design method, includes the following steps: Step 3.1: After implementing the address control function of each layer by pre-setting the access address offset, the accelerator reads memory data, processes the data, and writes the data back to memory according to the relevant parameters; Step 3.2: Each time multiple data blocks are read or written, the data blocks cached on the FPGA are reused before the next read is performed, ensuring that each data block is read from off-chip only a few times to reduce the number of memory accesses and the amount of data transferred. Step 3.3: For each piece of data retrieved, read the data from outside the chip into the input buffer, and process it according to the layer type using convolution, pooling, and fully connected layers; write the processed data back to the output buffer, and write it back to the external storage after obtaining the final result.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 3-8.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 3-8.

Images