FPGA reasoning-oriented motor imagery electroencephalogram signal recognition system

Through Gram Angle Field Transformation and Quantification Technology, the feature extraction and classification process of EEG signals is simplified, and convolutional neural network inference is realized on FPGA, solving the problem of low efficiency of complex EEG signals recognition and hardware inference, and realizing the High-efficiency, low-power sports imagination EEG signal recognition system.

CN118940079APending Publication Date: 2024-11-12UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410354779.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-03-27
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to effectively process complex high-dimensional EEG signals in the recognition of motor imagination EEG signals. Especially in hardware inference, the floating-point computing of FPGA is low, requires a large amount of resources, and has a low signal-to-noise ratio feature extraction step. Complexity affects the accuracy of identification.

Method used

Preprocessing and training steps include Gram angle field conversion and convolutional neural network classification, combined with quantization technology, convert EEG signals into image data, and use PyTorch's quantization interface to quantize model parameters into signed 8-bit integers, simplifying Inference process and implement the adjusted convolutional neural network on FPGA for inference.

Benefits of technology

It improves the accuracy and efficiency of the identification of the EEG signals of the motion imagination, reduces the hardware power consumption and development cycle, provides higher energy efficiency and real-time, and provides a new method for the hardware implementation of the brain-computer interface system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940079A_ABST
    Figure CN118940079A_ABST
Patent Text Reader

Abstract

The invention particularly relates to a motor imagery electroencephalogram signal recognition system oriented to FPGA reasoning, and aims to simplify and efficiently realize the process of reasoning a motor imagery electroencephalogram signal by hardware. The system comprises three steps of preprocessing and training, quantification and FPGA data set reasoning. Wherein the preprocessing and training step comprises preprocessing, feature extraction and feature classification. The preprocessing mainly comprises the steps of reading electroencephalogram signals, filtering, rejecting bad conductors and selecting segments. Feature extraction adopts a Grubrum angle field conversion method, and time sequence data can be converted into image data. After signal data is converted into image data, feature classification needs to train a convolutional neural network after the dimension of an output layer is adjusted according to the number of motor imagery categories. According to simplification of a symmetric quantization formula, a subsequent reasoning process of a quantization step is simplified, and quantized parameters are obtained. And after quantification, motor imagery electroencephalogram signal identification and classification are carried out by using the adjusted convolutional neural network realized on the FPGA.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) signal processing, and in particular to a motor imagery EEG signal recognition system oriented to FPGA reasoning. Background Art

[0002] In the rapidly advancing fields of neuroscience and biomedical engineering, accurately identifying EEG signals during motor imagery has become a key and challenging research area. This recognition technology has great application prospects in wearable devices and human-computer interaction systems. The motor imagery EEG signal recognition system is one of the most common brain-computer interface applications. It has been widely used in the field of intelligent healthcare, such as stroke recovery treatment and the development of mobile assistive robots.

[0003] In the brain-computer interface system, the decoded brain signals are converted into digital instructions to control smart devices and interact with users. The brain-computer interface system interprets the brain's activity patterns as usable information or commands to communicate with the outside world. The specific brain-computer interface system design framework is as follows: Figure 1 shown.

[0004] The brain-computer interface system first includes the acquisition of EEG signals S10, followed by the preprocessing and training of these signals S11, and finally the use of the training results for data analysis S12 or transmission to the control system S13. The response of the control system will be fed back to the brain, thus forming a new judgment loop and entering the signal processing process again.

[0005] However, motor imagery EEG signals are complex and have a high-dimensional structure. They mainly contain four key bands: delta wave, theta wave, alpha wave and beta wave. The frequency of delta wave is 1-4Hz, the frequency of theta wave is 4-8Hz, the frequency of alpha wave is 8-13Hz, and the frequency of beta wave is 13-32Hz.

[0006] In recent years, deep learning has had a great impact on motor imagery EEG signal recognition. Therefore, in terms of software algorithms, advanced feature extraction and deep learning algorithms are needed to process and decode these complex brain data. In terms of hardware implementation, efficient hardware inference of EEG signal data with small area, low power consumption and good real-time performance is required.

[0007] Since EEG signals have the characteristic of low signal-to-noise ratio, a series of preprocessing steps are usually required to improve their signal-to-noise ratio before feature extraction.

[0008] Feature extraction is the process of extracting discriminative features from input signals, which usually relies on knowledge in a specific field. Usually, time domain features, frequency domain features, or time-frequency domain features are extracted.

[0009] Deep learning has been widely used in image classification, speech recognition, object detection, etc., and has achieved remarkable results. Convolutional neural networks, recursive neural networks and deep belief networks are the main focus of deep learning research, among which the most advanced is the convolutional neural network.

[0010] A simple convolutional neural network structure is as follows Figure 2 As shown, it includes: input layer 20, convolutional layers 21 and 23, pooling layers 22 and 24, fully connected layers 25 and 26, and output layer 27. The convolutional layer is used for feature extraction, the pooling layer is used for pixel compression, and the fully connected layer is used for classification. The convolutional layer is a computationally intensive operation, while the fully connected layer is a data-intensive operation.

[0011] Currently, due to the improvement of hardware performance, the inference process of neural network hardware implementation mainly adopts the form of CPU, GPU, FPGA and ASIC. Since FPGA has many attractive features, it has become a good platform for the hardware implementation of convolutional neural network. In general, FPGA provides higher energy efficiency than CPU and GPU, and has higher performance than CPU. Compared with GPU, the throughput of FPGA is tens of gigaflops. Compared with ASIC, it has a shorter development cycle.

[0012] FPGA itself is not suitable for floating-point operations, which will take up a lot of hardware resources and require data quantization. Quantization is a method of calculating and storing tensors under the condition of lower bit width, which not only helps to reduce the memory and storage space required for deep learning models, but also improves the calculation speed, especially in the model inference stage. Summary of the invention

[0013] In view of the above background content, the present invention proposes a motor imagery EEG signal recognition system for FPGA reasoning. Figure 3 As shown, it includes three steps: preprocessing and training S11, quantization S30 and FPGA reasoning data set S31. The preprocessing and training step includes preprocessing S110, feature extraction S111 and feature classification S112.

[0014] The steps of S110 mainly include reading EEG signals S1100, filtering S1101, removing bad conductors S1102 and selecting segments S1103. These are to improve the quality of the signal, so that the subsequent analysis and feature extraction are more accurate and effective.

[0015] S111 mainly uses the Gram angular field conversion method. Gram angular field conversion can convert time series data into image data, which not only retains the complete information of the signal, but also maintains the signal's dependence on time. After converting the signal data into image data, the advantages of convolutional neural networks in image classification and recognition can be used for label classification.

[0016] S112 mainly uses convolutional neural network classification, specifically using a convolutional neural network with 27 dimensions adjusted according to the number of motor imagery categories and training it.

[0017] S30 mainly uses PyTorch's quantization interface to quantize the parameter data obtained by S11 into signed 8-bit integers. And based on the simplification of the symmetric quantization formula, the subsequent reasoning process is simplified, such as Figure 4 The simplified process used therein includes an input feature map 40 before quantization, an input feature map 41 after quantization, a weight 42 before quantization, a weight 43 after quantization, an output feature map 48 before quantization, and an output feature map 49 after quantization.

[0018] S31 used FPGA to implement the adjusted convolutional neural network and infer the data set to achieve the purpose of correctly identifying and classifying motor imagery EEG signals.

[0019] The advantages of the present invention mainly include: in terms of software, the traditional complicated EEG processing steps are abandoned, and Gram angular field conversion, deep learning and quantization technology are directly used after preprocessing to obtain the weight parameters after convolutional neural network training. In terms of hardware, low-power and short-cycle FPGA is used to achieve correct reasoning of EEG data, which provides a new perspective and method for future research and practical application of hardware-oriented motor imagery EEG signal recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Designing frameworks for brain-computer interface systems;

[0021] Figure 2 It is a simple convolutional neural network structure;

[0022] Figure 3 It is a motor imagery EEG signal recognition system for FPGA reasoning;

[0023] Figure 4 This is the simplified quantitative reasoning process. DETAILED DESCRIPTION

[0024] The following combination Figure 3 The implementation of the present invention is described in detail:

[0025] The software development platforms used in the present invention include open source scripting language python, open source machine learning framework PyTorch, MNE library for processing EEG signal data, and Vivado design suite, etc. The specific software tools used may not be limited thereto.

[0026] S110, first call the MNE library to read the subject's EEG sampling signal S1100, followed by S1101, S1102 and S1103. Since the four main bands of EEG signals are between 1 and 32 Hz, a bandpass filter is used. For small and stable artifacts such as electrooculogram, the artifact rejection method is used to remove the electrooculogram signal channel. Since in the sample collection process, only the motor imagination stage is useful for subsequent classification processing, it is necessary to select the signals of the remaining channels and intercept the useful parts.

[0027] Take the average of the EEG segments of the subject to get the mean of the EEG signal. And regard it as a series of column vectors in a matrix X. In this process, the concept of Gram matrix G is introduced, which defines the inner product between each pair of vectors in the vector set. The formula is expressed as:

[0028] S111, the first step of the Gram angular field conversion process is signal scaling, that is, mapping the EEG signal to the coordinate system and scaling it to [-1,1]. The standardized Gram matrix is ​​expressed as follows: where φ i,j represents the angle between the two vectors i and j, and the standardized value is recorded as G represents the degree of correlation between vectors, and this degree of correlation is actually determined by the angle between the vectors.

[0029] The second step is polar coordinate mapping, which is to generate polar coordinates by taking the timestamp as the arc cosine of the radius and the scaling value. The polar angle and polar axis of the polar coordinates are shown in formula (3) and formula (4), respectively: φ i =arccosx (3) in Represents point x i timestamp, N is the number of all time points contained in the time series data. Each time series data point contains two pieces of information, one is the normalized value of the data point The other is its temporal position t iThe polar coordinate conversion encoding above includes both information without losing any information. Mathematically, it is a bijective function, that is, the independent variable and the dependent variable of this function have a one-to-one correspondence. i The temporal relationship is preserved, while the polar angle φ i The numerical relationship is preserved.

[0030] The third step is image generation, which is to convert angles into pixel intensities. The Gram angle field defines its own special inner product, that is, the inner product between two time points is the cosine of the sum of the polar angles of the two time points after polar coordinates are transformed, as shown in formula (5). This can generate the Gram angle field map corresponding to the subject's EEG signal.

[0031] Since the number of categories corresponding to the data set may be different, it is necessary to adjust the structure of the original convolutional neural network. Figure 2 As shown in the figure, for a dataset of P categories, the network output dimension of the last layer is P, and the corresponding weight dimension of this layer becomes P×84.

[0032] S112, inputting the generated GAF ​​graph of each category into the adjusted convolutional neural network for network training, and obtaining the parameters before quantization.

[0033] S30, first choose to use the post-training static quantization solution provided by PyTorch. In this process, the input feature map, output feature map and weight parameters are converted to INT8 format. Next, start the inference process using this newly converted quantized model. Figure 4 In the figure, there are processes 44, 45, 46 and 47. The bias retained in the original FP32 format is integrated into the intermediate feature map. This fusion process produces an FP32 output feature map. In order to ensure that this floating point output feature map is consistent with the rest of the quantized model, it is necessary to perform symmetric quantization on it and convert the output feature map to INT8 format.

[0034] Processes 44, 45, 46, and 47 cannot be directly used in bit-serial hardware circuits, so the existing method needs to be improved. From formula (6), it can be seen that the original quantization step can be simplified to a single INT8 quantization process. From the distribution of the bias values, it can be obtained that the values ​​are close to 0 after rounding. Therefore, these bias terms are omitted in the inference process, that is, the inference process after quantization can be simplified to 48 and 49.

[0035] Use Torch's Hook mechanism to obtain the weight parameters of each layer and the parameters of the input and output feature maps. The Hook mechanism consists of two core parts: one is to define a function for processing feature maps, and the other is to register the Hook, that is, to specify the layers on which the model uses this defined function to process feature maps.

[0036] On the FPGA, in order to simplify the quantization process after tensor accumulation and make it conform to the rules of hardware quantization, the value of the quantization ratio is preferably a power of 2. In this way, complex division operations can be replaced by simple data right shift operations. Here, the 23rd bit is selected as the highest sign bit, and the numbers from the 14th to the 8th bits are used to form the hardware quantized value. Therefore, the value of the quantization ratio is set to 256, and the other bits are discarded. At the same time, it is necessary to ensure that the newly constructed data value is within the range of -128 to 127. This processing method is intended to simplify the processing of data on the FPGA while ensuring the effectiveness and accuracy of the quantization process.

[0037] S31, first includes converting the quantized weights and feature map parameters of each layer into binary form, and then these binary data are converted into COE files, and these files are added to the RAM of the neural network accelerator. This step is necessary because COE files are often used to pass parameters in the configuration of the neural network accelerator. The COE file is an ASCII text file format, and its file header defines the cardinality of the data as 2, that is, the data is represented in binary form. In this file, the data is given in the form of vectors, each vector ends with a semicolon, and the vectors are separated by commas. In this process, the Vivado tool parses the format of the COE file and exports the corresponding MIF format file when generating the IP core. These MIF files are then used for behavioral-level simulation to ensure the accuracy and effectiveness of the entire system.

[0038] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the art can make some possible changes and modifications to the technical solution of the present invention by using the above-mentioned methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A motor imagery EEG signal recognition system for FPGA reasoning, characterized in that: The system includes three steps: preprocessing and training, quantization, and FPGA inference data set.

2. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 1 is characterized in that: The preprocessing and training steps include preprocessing, feature extraction and feature classification.

3. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 2 is characterized in that: Preprocessing includes reading EEG signals, filtering, removing bad signals and selecting segments.

4. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 2 is characterized in that: The Gram angular field transformation method was used to extract the features of the selected and averaged EEG signals.

5. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 2 is characterized in that: A convolutional neural network is used for feature classification, and the convolutional neural network is trained to adjust the output layer dimension according to the number of motor imagery categories.

6. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 1, characterized in that: According to the simplified symmetric quantization formula, the subsequent reasoning process of the quantization step is simplified by removing the bias term.

7. The motor imagery EEG signal recognition system for FPGA reasoning according to claim 1 is characterized in that: The quantized motor imagery EEG data were inferred and classified using a tuned convolutional neural network implemented on an FPGA.

Citation Information

Patent Citations

  • Fast classification method, method and device for EEG signals based on FPGA

    CN109376843A

  • Electroencephalogram data processing identification method and system

    CN114841205A

  • Motor imagery classification method and system based on electroencephalogram signals

    CN114997229A

  • Motor imagery task classification method and system based on electroencephalogram signals and deep learning

    CN116842329A

  • Space-time diagram multi-dimensional information fusion electroencephalogram domain generalization decoding method and device and storage medium

    CN117521009A