A hybrid neural network-based electroencephalogram signal hardware acceleration recognition system

Abstract

The application relates to a motor imagination electroencephalogram signal hardware acceleration recognition system based on a hybrid neural network and belongs to the technical field of robots; the system solves the problems of limited computing power and performance, high power consumption and poor real-time performance of an electroencephalogram signal recognition system in the prior art; the system comprises a signal conversion module, an embedded processor and an electroencephalogram algorithm hardware accelerator; the signal conversion module is used for carrying out digital conversion on input electroencephalogram signals to obtain digital electroencephalogram signals; the embedded processor is used for pre-processing the digital electroencephalogram signals and controlling parameters of the hardware accelerator; the electroencephalogram algorithm hardware accelerator comprises a hybrid neural network module, is used for sequentially carrying out parallel convolution operation and gate operation on the pre-processed digital electroencephalogram signals, extracting corresponding feature data of the electroencephalogram signals including spatial features and time features, and carrying out classification operation to obtain motor imagination categories corresponding to the electroencephalogram signals.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a hardware-accelerated brainwave signal recognition system based on a hybrid neural network. Background Technology

[0002] Brain-computer interfaces (BCIs), serving as a connection pathway between the brain and external devices, have significant medical applications. In the medical field, they can effectively repair damaged nerves through repetitive stimulation and also acquire and analyze electroencephalogram (EEG) signals from the brain to obtain intention information and convert it into control commands for external devices. Motor imagery (MI)-based BCI control methods use EEG signals generated by the cerebral cortex during subject imagery as input to the BCI system. The system's signal processing equipment then analyzes these signals, identifies the type of limb movement, and transmits the corresponding commands to the connected device, enabling the device to respond accordingly. With voluntary adaptive control, MI-BCI systems can help patients with damage to the nervous system or motor organs regain some simple motor abilities. For example, a robotic arm can be integrated into the MI-BCI system, allowing the patient to control it through imagery. Beyond the medical field, BCIs have wide applications in military, entertainment, and other areas.

[0003] Existing MI-BCI devices typically employ sparse multi-channel EEG acquisition equipment and deploy classic machine learning algorithms within CPUs, GPUs, or MCUs. These methods suffer from drawbacks such as poor portability, poor real-time performance, low flexibility, and high power consumption, failing to meet the miniaturization and portability requirements of medical, everyday, and industrial control applications. Research on wearable motor imagery brain-computer interfaces generally employs a method of connecting portable acquisition devices (such as microelectrode arrays connected to flexible cables) to signal processing equipment for EEG signal recognition. Among them, Luciano Prono et al. implemented EEG signal feature extraction based on a programmable counter array (PCA) on a low-end microcontroller edge device (STM32H743ZIT) and showed good performance in extracting the main subspace of EEG data. However, its classification accuracy was not outstanding and its real-time performance was poor. Moreover, due to its limited hardware resources and computing power, it could not deploy complex algorithms to improve it. Deploying improved algorithms using general-purpose AI accelerator chips requires secondary development based on the algorithm. The compatibility issues of model porting will cause a decrease in model accuracy. Furthermore, the large number of unutilized acceleration resources built into general-purpose AI accelerator chips will also lead to higher power consumption, making it difficult to adapt to the requirements of EEG signal recognition and processing in a wider range of application scenarios.

[0004] It is evident that there is an urgent need for a hardware acceleration system specifically designed for EEG motor imagery signals. Based on a hybrid neural network-based EEG signal recognition algorithm, the hardware structure should be optimized to improve the accuracy and real-time performance of the algorithm. This would meet the application requirements of wearable motor imagery brain-computer interfaces with low latency, low power consumption, and high energy efficiency for a wider range of applications, thereby enabling natural control of bionic devices. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a hardware-accelerated EEG signal recognition system based on a hybrid neural network; it solves the problems of limited computing power and performance, high power consumption and poor real-time performance of existing EEG signal recognition systems.

[0006] The objective of this invention is mainly achieved through the following technical solutions:

[0007] This invention provides a hardware-accelerated EEG signal recognition system based on a hybrid neural network, the system comprising a signal conversion module, an embedded processor, and an EEG algorithm hardware accelerator;

[0008] The signal conversion module is used to digitally convert the input EEG signal to obtain a digital EEG signal;

[0009] The embedded processor is used to preprocess the digital EEG signal and control the parameters of the hardware accelerator;

[0010] The EEG algorithm hardware accelerator includes a hybrid neural network module, which performs parallel convolution and gating operations on the preprocessed digital EEG signals to extract feature data corresponding to the EEG signals, including spatial and temporal features, and performs classification operations to obtain the motor imagery category corresponding to the EEG signals.

[0011] Furthermore, the EEG signals include multi-channel EEG signals;

[0012] The hybrid neural network module includes a convolutional neural network processing module and a long short-term memory network processing module;

[0013] The convolutional neural network processing module is used to perform feature extraction at different depths on pre-processed EEG signals from multiple channels through convolution operations based on pre-trained operational parameters, and then perform feature fusion to obtain spatial feature data corresponding to the EEG signals.

[0014] The Long Short-Term Memory Network (LSTM) processing module is used to perform gating operations and LSTM logic operations based on the spatial feature data to obtain feature data corresponding to the EEG signal, including spatial and temporal features.

[0015] Furthermore, the convolutional neural network processing module includes a PE array for feature extraction; the PE array includes n parallel channel feature extraction units, where n is the number of channels of the EEG signal; each channel feature extraction unit includes m layers of serial PE modules;

[0016] The PE module in the first layer is used to perform depth convolution and point convolution operations based on the input EEG signal and operation parameters. The PE modules in subsequent layers perform depth convolution and point convolution operations based on the output of the PE module in the previous layer and operation parameters, respectively, to obtain feature data of different depths extracted by each PE module.

[0017] Furthermore, the PE module includes a multiply-accumulate operation module and a post-processing module;

[0018] The multiply-accumulate operation module includes nine multipliers, an adder tree, and one adder. During deep convolution, the input EEG signal data and operation parameters are multiplied and added using the nine multipliers and the adder tree within one cycle to obtain the deep convolution result. During point convolution, one of the multipliers and the adder tree is selected for multiply-accumulate operation. The result of the multiply-accumulate operation is temporarily stored in a buffer, and the current multiply-accumulate result is added to the multiply-accumulate result calculated in the previous cycle using the adder until the point convolution operation of the entire input data is completed, and the final point convolution result is obtained.

[0019] The post-processing module is used to perform max pooling and activation operations on the convolution operation results to obtain feature data of different depths for the corresponding channels.

[0020] Furthermore, the Long Short-Term Memory network processing module includes a gated computation unit and an LSTM logic computation unit;

[0021] The gated computation unit includes four LSTM gate modules: a forget gate, a cell state gate, an input gate, and an output gate; the gated computation unit is based on the currently input feature data X. t The vector H calculated in the previous cycle t-1 After corresponding calculations and activations, the gate vectors of the four LSTM gate modules are obtained;

[0022] The LSTM logic computation unit is used to perform multiplication, addition, and activation operations based on the gate vectors of the four LSTM gate modules to obtain feature data corresponding to the EEG signal, including spatial and temporal features.

[0023] Furthermore, the input feature data and corresponding operation parameters of the gated computing unit are input to the four LSTM gate modules in parallel; parallel multiplication operations are performed through each LSTM gate module, and the results are summed through an addition tree to obtain the gate vector of each LSTM gate.

[0024] Furthermore, the EEG algorithm hardware accelerator also includes a parameter memory and a feature input memory;

[0025] The parameter memory is partitioned and allocated to each PE module and LSTM gate module in the hybrid neural network module to store the operation parameters of the corresponding module.

[0026] The feature input memory adopts a dual first-in-first-out queue structure, and reads the input digital EEG feature data through a sliding window. Each time three lines of data are read, they are stored in three registers of size 3. When the registers are full, a 3*3 feature data matrix is ​​generated and sent to the PE module for processing.

[0027] Furthermore, the feature fusion includes:

[0028] The feature data of different depths extracted by the convolutional neural network processing module are sequentially stored in the external memory DDR, and the feature data of the l-th and m-th layers of n channels are concatenated to obtain two sets of feature vectors of different depths, where l <m;

[0029] The two sets of feature vectors with different depths are fused by element-wise multiplication, and the fused feature vector is represented as follows:

[0030] o i =o cli ·o cmi ;

[0031] Where i is the feature index, o cli For the feature vector extracted from the l-th layer, o cmi This is the feature vector extracted from the m-th layer.

[0032] Furthermore, the hardware accelerator also includes a quantization processing module. This module performs quantization operations on each PE module and LSTM gate module after calculation. The quantization operations include: mapping the data to be quantized through a right shift to map the original data to the required data size; performing rounding during the shift process; and determining the size of the last digit of the shift operation. If the last digit is greater than 1, incrementing it by 1; otherwise, leaving it unchanged. Finally, truncating the bit by CLP to obtain the required number of bits for the output.

[0033] Furthermore, the computational parameters include weight parameters and bias parameters; the computational parameters are obtained by iteratively training the EEG recognition network model using a loss function on a pre-constructed EEG dataset, wherein the EEG dataset includes EEG signal data and the corresponding motor imagery category labels.

[0034] The beneficial effects of the technical solution of this invention:

[0035] 1. Based on a heterogeneous computing platform, this invention designs a dedicated hardware acceleration architecture for a hybrid neural network motion imagery EEG recognition model. It proposes a hardware acceleration structure that is parallel between channels and parallel within convolutional kernels. The designed hardware architecture is highly parameterizable and has low resource utilization, with high energy efficiency and performance, filling the current technological gap in motion imagery wearable devices.

[0036] 2. This invention performs signal processing and feature extraction on the motor imagery signals collected by the 16-64 channel EEG cap, and optimizes the motor imagery EEG signal recognition algorithm model structure based on hybrid neural network for wearable application scenarios. By fusing features of time and space signals, it can perform high-precision recognition of motor imagery signals for different actions, and control the bionic robotic arm to perform corresponding actions through action command encoding.

[0037] 3. This invention is the first to use a hardware accelerator of a hybrid neural network for the natural control task of motor imagery EEG signals, forming a portable wearable device that solves the shortcomings of current traditional devices, such as high power consumption, low accuracy, and high latency, and realizes real-time natural control of motor imagery signals on bionic devices.

[0038] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0040] Figure 1 This is a block diagram of a hardware-accelerated EEG signal recognition system based on a hybrid neural network, according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the electroencephalogram (EEG) signal acquisition and recognition system according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the feature input memory structure according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the hardware accelerator structure for the EEG algorithm according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the PE module structure according to an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the gated computing unit according to an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of the LSTM logic computing unit according to an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of the quantization module structure according to an embodiment of the present invention;

[0048] Figure 9 This is a schematic diagram of the EEG recognition network model according to an embodiment of the present invention. Detailed Implementation

[0049] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0050] This embodiment presents a hardware-accelerated EEG signal recognition system based on a hybrid neural network, such as... Figure 1 As shown, it includes a signal conversion module, an embedded processor, and a hardware accelerator for EEG algorithms;

[0051] The signal conversion module is used to digitally convert the input EEG signal to obtain a digital EEG signal;

[0052] Embedded processors are used for preprocessing digital EEG signals and for parameter control of hardware accelerators;

[0053] The EEG algorithm hardware accelerator includes a hybrid neural network module, which performs parallel convolution and gating operations on the preprocessed digital EEG signals to extract feature data including spatial and temporal features corresponding to the EEG signals, and performs classification operations to obtain the motor imagery category corresponding to the EEG signals.

[0054] Specifically, the EEG signals received by the hardware-accelerated EEG signal recognition system in this embodiment can be acquired through an EEG acquisition electrode cap, such as... Figure 2As shown, the acquired EEG signals are input into the EEG signal hardware acceleration recognition system of this embodiment for recognition processing via flexible cables. The electrode cap is a multi-channel EEG electrode that can be directly attached to the surface of the human scalp. The EEG signal hardware acceleration recognition system can be made into different shapes of processing circuits and fixed near the EEG cap according to actual applications to perform recognition processing of motor imagery EEG signals.

[0055] The signal conversion module is used to perform signal filtering, signal amplification, and analog-to-digital conversion on the multiple channels of EEG signals acquired by the multi-channel EEG electrodes to obtain digital EEG signals. Specifically, the EEG signals acquired by the EEG electrodes include multi-channel (16-64 channels) EEG signals and reference electrode signals. The signals are first denoised by a second-order anti-aliasing filter circuit. The denoised signals are then amplified by an operational amplifier. After amplification, the input analog signals are converted into digital signals by a high-precision analog-to-digital converter. The converted digital signals are in matrix form, where rows represent channels and columns represent sampled values. In this embodiment, the analog-to-digital converter used has a precision greater than 24 bits, preferably the ADS1299 chip. After analog-to-digital conversion, the digital EEG signals are sent to the embedded processor via the SPI bus for further processing.

[0056] The embedded processor is used to preprocess the digitally converted EEG signals and to change the value of the parameter register via the bus interface to control the structural parameters of the EEG algorithm hardware accelerator.

[0057] Preferably, after receiving the digitally converted EEG signal, the embedded processor first performs preprocessing on the digital EEG signal, including:

[0058] Signal filtering: In this embodiment, an FIR bandpass filter is used for bandpass filtering. The passband frequencies are selected from the 8Hz and 30Hz bands, which are most relevant to EEG data and motor imagery. The FIR bandpass filter has arbitrary characteristics, strict linear phase frequency characteristics, and a finite-length unit sample response, resulting in a relatively stable signal after processing.

[0059] Independent Component Analysis (ICA): In this embodiment, ICA is used to remove artifacts from the data. In practical applications, the EEGLAB toolbox in MATLAB can be used for ICA decomposition to evaluate the probability of EEG components in each channel, remove EEG artifacts, and retain the EEG signal with EEG artifacts removed; the preprocessed digital EEG signal is then used for subsequent convolution operations.

[0060] Furthermore, the EEG algorithm hardware accelerator includes an internal storage module and a hybrid neural network module, wherein the hybrid neural network module includes a convolutional neural network processing module and a long short-term memory network processing module.

[0061] Preferably, the internal memory module of the EEG algorithm hardware accelerator in this embodiment includes a parameter memory and a feature input memory. The parameter memory occupies the largest space in the internal memory and stores all parameters used in the operations of its layer. The parameter memory is partitioned and allocated to each PE module and LSTM gate module of the convolutional neural network processing module to achieve efficient parallel multiply-accumulate operations. The feature input memory structure is as follows: Figure 3 As shown, a pipelined sliding window can be formed using a dual first-in-first-out (FIFO) queue structure to read input signals. After system initialization, both FIFO rows are empty. Each FIFO row is used to buffer one line of EEG data. The EEG data is first input into FIFO1, and the data read from FIFO1 is written into FIFO2. When FIFO2 and FIFO1 have buffered one line, the module is initialized and enters the working state when the third line of data is input. Each time, three lines of EEG data are read simultaneously and stored in three registers of size 3. When the registers are full, a 3*3 EEG data matrix is ​​generated, and the module output is controlled by the doone signal configured by the control port.

[0062] Furthermore, the convolutional neural network processing module is used to extract features of different depths from the pre-processed EEG signals of multiple channels through convolution operations based on the pre-trained operational parameters, and then perform feature fusion to obtain the spatial feature data corresponding to the EEG signals.

[0063] The Long Short-Term Memory (LSTM) network processing module is used to perform gating operations and LSTM logic operations based on spatial feature data to obtain feature data corresponding to EEG signals, including spatial and temporal features.

[0064] Specifically, the convolutional neural network processing module includes a PE array for feature extraction; such as Figure 4 As shown, the PE array includes n parallel channel feature extraction units; each channel feature extraction unit includes m layers of serial PE modules;

[0065] The PE module in the first layer is used to perform depthwise convolution and pointwise convolution operations based on the input time-frequency map feature data and operation parameters. The PE modules in subsequent layers perform depthwise convolution and pointwise convolution operations based on the output results and operation parameters of the previous PE module, respectively, to obtain the feature vector output by each PE module.

[0066] Specifically, in the convolutional neural network processing module, each layer of the PE module performs convolution operations to obtain spatial feature data of different depths and spatial dimensions of the corresponding EEG signal. The lower-level PE modules extract the shallow features of the corresponding EEG signal, and the higher-level PE modules extract the deep features of the EEG signal. The outputs of the same layer of PE modules in the n channel feature extraction units are sequentially stored in the external memory DDR as feature data of different depths of the corresponding EEG signal. This data is then sent to the long short-term memory network processing module for feature extraction after subsequent fusion processing. The long short-term memory network is a recurrent neural network, mainly used to extract features from the time series of EEG signals.

[0067] Preferred, such as Figure 5 As shown, each PE module has the same structure, divided into two parts: a multiply-accumulate module and a post-processing module. The multiply-accumulate module includes nine multipliers, an adder tree, and one adder. During 3x3 depthwise convolution, the input EEG feature data and weight parameters are multiplied and added using the nine multipliers and the adder tree within one cycle to obtain the depthwise convolution result. During pointwise convolution with a kernel size of 1, one of the multipliers and the adder tree is selected for multiply-accumulate. The result of the multiply-accumulate operation is temporarily stored in a buffer, and the current result is added to the result of the multiply-accumulate operation calculated in the previous cycle using the adder until the pointwise convolution operation of the entire input data is completed, obtaining the final pointwise convolution result.

[0068] It should be noted that, in order to achieve real-time EEG processing while minimizing resource usage, two parallel optimization strategies were implemented in this embodiment. First, this embodiment introduces an internal parallelization strategy for convolutional kernels to compute each convolutional kernel in parallel. Second, the aforementioned parallel channel feature extraction unit implements an output channel parallelization strategy, achieving parallel processing of multiple channels. The two parallelization strategies mentioned above have no data dependencies, are easy to implement, and greatly improve the efficiency of EEG recognition.

[0069] Furthermore, the post-processing module consists of a bias adder, a nonlinear layer (Leaky-ReLU), a shifter, a clipping limiter (CLP), and a max pooling (MP) module. After the convolution operation, an activation operation is performed through the nonlinear layer, and the activated data is then truncated using CLP. Upon receiving the last set of data, a control signal is used to activate the max pooling module instead of CLP, resulting in the output of the convolutional neural network module.

[0070] Preferably, after the convolutional neural network processing module extracts feature data of different depths, the data of different channels of the same layer are sequentially stored in the external memory DDR for subsequent processing.

[0071] The EEG algorithm hardware accelerator in this embodiment further includes a fusion module, which is used to retrieve the feature data of the l-th layer and the m-th layer stored in the DDR respectively for data splicing, so as to obtain two sets of feature vectors with different depths, where l < m; in this embodiment, l and m are respectively 2 and 4. After splicing the feature data of the 2nd layer and the 4th layer respectively, two sets of feature vectors with different depths are obtained, that is, the feature vectors extracted from the 2nd layer and the 4th layer are obtained. The two sets of feature vectors with different depths are multiplied element by element through a multiplier by the fusion module to obtain a fused feature vector, which is expressed as:

[0072] o i = o cli · o cmi ;

[0073] where i is the feature index, o cli is the feature vector extracted from the l-th layer, and o cmi is the feature vector extracted from the m-th layer.

[0074] The EEG signal feature vector after feature fusion is mainly the feature data of the channel and spatial dimensions. In order to improve the accuracy of EEG signal motor imagery classification, this embodiment further extracts the features in the EEG signal time series from the spatially fused feature data through a long short-term memory network processing module.

[0075] The long short-term memory network processing module includes a gating calculation unit and an LSTM logic calculation unit; among them, the gating calculation unit includes four LSTM gate modules: a forget gate, a cell state gate, an input gate and an output gate; the gating calculation unit is based on the currently input feature data X t and the vector H t-1 calculated in the previous cycle, and through corresponding calculations and activations, the gate vectors of the four LSTM gate modules are obtained; the LSTM logic calculation unit is used to perform multiplication, addition and activation operations based on the gate vectors of the four LSTM gate modules to obtain the feature data corresponding to the EEG signal including spatial features and time features.

[0076] Specifically, the core of the Long Short-Term Memory (LSTM) network processing module includes an LSTM gated computation unit and an LSTM logic computation module. The LSTM gated computation unit comprises four gate modules: the Forget Gate, the Cell Gate, the Input Gate, and the Output Gate. Through data control, the fused EEG signal feature data stored in the DDR (Memory Memory) is first read into the feature input memory. Then, it is input into the four LSTM gate modules to execute the LSTM network's inference. The results are then transmitted to the LSTM logic computation module to perform logical calculations, obtaining feature vectors corresponding to the EEG signals, including spatial and temporal features. These feature vectors are then stored in external memory for subsequent classification operations.

[0077] Special, such as Figure 6 As shown, the input feature data and corresponding operational parameters of the gated computation unit are input to the four LSTM gate modules in parallel. Parallel multiplication operations are performed within each LSTM gate module, and the results are summed using an addition tree to obtain the gate vector for each LSTM gate. That is, within each gate module, the gate vector is computed using a tiling scheme, and the tiled input vector and corresponding parameters are transmitted in parallel to the LSTM gate module for inference. Within each LSTM gate module, all multiplications between input elements and parameters are performed in parallel. The results are then summarized using an addition tree to minimize latency. The entire architecture employs a pipelined approach to further improve throughput.

[0078] The input to the gated computation unit is the feature input data X at the current moment. t The feature data H passed down from the previous cycle t-1 The concatenated data is processed through four different gates, undergoing corresponding calculations and activation operations to obtain the corresponding output. The calculation formula for each gate is as follows:

[0079] Z = tanh(W × [X]) t H t-1 ]);

[0080] Z o = sigmod(W o ×[X t H t-1 ]);

[0081] Z i = sigmod(W i ×[X t H t-1 ]);

[0082] Z f = sigmod(W f×[X t H t-1 ]);

[0083] Among them, Z and Z o Z i Z f These are the outputs of the cell state gate, output gate, input gate, and forget gate, respectively; W, W o W i W f These are the weight parameters for the cell state gate, output gate, input gate, and forget gate, respectively; X t Input data for the features at the current time; H t-1 These are the feature data passed down from the previous cycle.

[0084] LSTM logic computing unit such as Figure 7 As shown, its function is to perform element-wise multiplication and addition of the gate vector and activate it to obtain the final output result. Its calculation formula is as follows:

[0085] C t =Z f ⊙C t-1 +Z i ⊙Z;

[0086] H t =Zo⊙tanh(C t );

[0087] Y t =sigmod(W′H t );

[0088] Among them, C t For the current cell state, C t-1 H represents the cell state at the previous time step. t For the current feature data, Y t For module output, Z i Z is the input gate output, Z is the cell state gate output, ⊙ indicates the dot product operation, and W′ is the weight parameter.

[0089] Furthermore, the EEG algorithm hardware accelerator also includes a classification module. Data processed by the Long Short-Term Memory (LSTM) network module is then classified by this module to obtain the motor imagery category corresponding to the EEG signal. The classification module comprises two fully connected layers. It sequentially reads data stored in the attention module from the external memory (DDR) and performs classification operations to obtain the action command category corresponding to the EEG signal.

[0090] Preferably, the hardware accelerator in this embodiment further includes quantization operations after each layer of PE module, LSTM gate module, and LSTM logic computation. To further reduce hardware resource consumption, this embodiment uses shifting and truncation to implement the quantization operations. Figure 8 As shown, each different layer corresponds to a scaling factor, which determines the shift size. First, the data to be quantized is mapped by right shifting to the desired data size. During the shift, rounding is performed to further improve quantization accuracy. The size of the last digit in the shift operation is checked; if it is greater than 1, it is incremented by 1, otherwise it remains unchanged. Finally, a truncation operation is performed using CLP to obtain the desired number of bits in the output, completing the quantization operation.

[0091] The hardware accelerator architecture in this embodiment has high compatibility with different networks and processing layers, and can largely utilize the inherent pipelining characteristics of hardware. The DSP is highly efficient, with fast computation speed, achieving higher throughput. In practical applications, a highly parameterized hardware accelerator architecture suitable for hybrid neural networks can be built based on a heterogeneous computing platform with the ZYNQ architecture, enabling high-performance parallel computation of the network model and obtaining motion imagery classification results. Due to the large number of network layers, the large computational load, and the irregular network structure, the computational load and data cache of different layers and branches can be evenly allocated during the forward computation process. The computational part of the network mainly uses DSP resources, while the data cache mainly uses BRAM and LUT-RAM resources. Efficient use and scheduling of hardware resources can enable the EEG algorithm hardware accelerator to achieve better performance and throughput.

[0092] Furthermore, the EEG signal hardware acceleration recognition system in this embodiment also includes a power supply module. This power supply module is responsible for powering the entire system. It is powered by a 3.7V lithium battery, and the voltage is converted through a boost / buck circuit and a voltage regulator circuit to provide the required voltage for other modules in the system. The signal conversion module requires three voltages: 3V, 3.3V, and 5V. 3V is used as the analog power supply in the analog-to-digital conversion, 3.3V as the digital power supply, and 5V to power the operational amplifier. The data processing module requires both 3.3V and 5V. 3.3V powers the embedded processor, and 5V powers the EEG algorithm hardware accelerator.

[0093] Specifically, in this embodiment, the embedded processor can be selected as a processor with a main frequency greater than 1.2 GHz and a floating-point arithmetic unit. The EEG signal data after signal preprocessing and time-frequency conversion are all stored in external memory (DDR) via bus interconnection. DDR can be third-generation DDR (DDR3) or higher, with a capacity greater than 1 GB. The external memory (DDR) is also used to store the input signals, input signal parameters, and output data of each layer of the EEG algorithm hardware accelerator. Finally, the EEG algorithm hardware accelerator of this embodiment reads the preprocessed EEG data for data processing and analysis, parses the corresponding EEG signals, determines the type of motor imagery command, and writes the results into DDR via bus interconnection. Then, the embedded processor reads the results and sends control commands to the robotic arm via serial port to complete real-time natural control.

[0094] Preferably, the computational parameters in this embodiment include weight parameters and bias parameters; the computational parameters are obtained by iteratively training the EEG recognition network model using a pre-constructed EEG dataset and a loss function. The EEG dataset includes EEG signal data and corresponding motor imagery instruction category labels.

[0095] This embodiment uses four object grasping action categories—extending arm, bending arm, grasping, and relaxing—as classification targets to make it more closely resemble real-life scenarios.

[0096] First, EEG signals for motor imagery were collected to construct a dataset. Before the start of the EEG signal collection experiment, there was a 5-second preparation time during which a "+" symbol was displayed on the screen, indicating that the experiment was about to begin and that the subject was ready to perform motor imagery. A complete collection process lasted 10 seconds and was divided into two phases. From 0 to 7.5 seconds, a random image prompt for motor imagery was displayed on the screen, depicting one of four actions: extending the arm, bending the arm, grasping an object, or releasing the hand. The subject performed the corresponding motor imagery action based on the image. At 7.5 seconds, the image disappeared, and the subject stopped imagining the action. During the imagination process, the data was recorded and labeled accordingly. From 7.5 to 10 seconds, the screen remained blank, prompting the subject to relax their mind. Each subject had 10 EEG recordings for each type of motor imagery, for a total of 40 recordings, with a 5-minute rest period after each recording. This embodiment collected EEG signals from 10 subjects, obtaining the initial EEG signal dataset.

[0097] After acquiring the initial EEG signal dataset, the EEG signals are converted from analog to digital and preprocessed using the methods described above to obtain a digital EEG signal dataset with noise reduction and removal of electrooculogram artifacts.

[0098] Because the subject's state (such as fatigue, emotional fluctuations, etc.) needs to be considered during the acquisition of EEG signals, the experimental process of acquiring EEG signals should not be too long. The acquisition of EEG data is time-consuming, labor-intensive, and difficult to obtain, while deep learning methods require a large amount of training data. Existing methods for expanding training datasets often use methods such as cropping, amplitude perturbation, signal time-frequency domain segmentation, and reconstruction to augment the data. However, these methods are merely simple transformations of the original data and have low generalization ability.

[0099] This embodiment constructs a generator network G and a discriminator network D based on a convolutional neural network (CNN) to achieve data augmentation. The generator network includes two fully connected layers and two convolutional layers; the training objective of the generator is to generate EEG signal data identical to real EEG signals. The discriminator network consists of a convolutional layer, a max-pooling layer, another convolutional layer, a max-pooling layer, an average pooling layer, and finally two fully connected layers. The outputs of the generator and discriminator use the TanH activation function, while other layers use the Leak ReLU activation function, and each layer is normalized using BatchNormalization.

[0100] Specifically, firstly, random samples are taken from a normal distribution with a mean of 0 and a standard deviation of 1 as random noise data. These random noise data are labeled to generate sample labels. The labels are concatenated with the random noise to establish a mapping relationship, and then input into the network model of the generator G for learning to obtain generated samples. After obtaining the generated samples, the generated sample data, along with pre-collected real EEG data of motor imagery, are input into the discriminator D. The discriminator D calculates the probability that the input motor imagery EEG data originates from real data. If the sample comes from real training data, a high probability is output; otherwise, a low probability is output. The amplified samples corresponding to the EEG data in the sample set are obtained through iterative updates using the cross-entropy loss function. The objective function of the GAN model for sample amplification is as follows:

[0101]

[0102] Where x~p data (x) represents the distribution of the real data x, z ~ p z (z) represents the distribution of generated data z, D(x) represents the real data, and G(z) represents the generated data.

[0103] Preferably, this embodiment also uses the cross-entropy loss function to calculate the discriminator network loss and the generator network loss, and their objective functions are as follows:

[0104] Loss D = -logD(x) - log(1 - D(G(z));

[0105] Loss G = -logD(G(z));

[0106] Among them, Loss D Loss is the loss of the discriminator network. G The generator network loss.

[0107] During training, one network in the generator and discriminator is fixed while the parameters of the other network are updated, and this process is repeated iteratively. Ultimately, the generator G can estimate the distribution of real motor imagery EEG data, generating highly realistic motor imagery data, thus expanding and enhancing the dataset and constructing the final EEG data sample set. After constructing the EEG data sample set, deep learning algorithms can be used to train the EEG recognition network model. The trained and tested EEG recognition network model algorithm code is ported to the embedded processor of the data processing module in a heterogeneous computing architecture. The trained model weight parameters and bias parameters are written to the parameter memory of the EEG algorithm hardware accelerator via the embedded processor and bus interconnection. The configuration registers in the EEG algorithm hardware accelerator are then modified via the embedded processor and bus interconnection to configure the parameters, completing the model deployment.

[0108] In practical applications, real-time acquired EEG signals are sent to an embedded processor via a differential SPI bus for preprocessing and calculation using a deep learning EEG algorithm hardware accelerator. Finally, the classification results of motor imagery commands are obtained and encoded into control commands by the embedded processor and sent to the bionic robotic arm for control.

[0109] like Figure 9 As shown, the EEG recognition network model includes a feature extraction module, a feature fusion module (LSTM module), and a classification output module. The feature extraction module comprises two feature extraction pathways, each using a convolutional neural network (CNN) of different depths, trained to obtain the channel and spatial features of the EEG signals. After training, the multi-channel feature data output from the two CNNs of different depths are concatenated and then input into the feature fusion module for feature fusion. Finally, the fused spatial features are fed as a time series into the LSTM to extract the temporal features of the signal. Finally, the classification output module classifies the motor imagery EEG signals.

[0110] Specifically, in this embodiment, the feature extraction module uses two pathways: one is a shallow CNN with 1-3 convolutional layers, which can be flexibly adjusted based on the actual training effect to extract simple, shallow features from the motor imagery EEG signal. The other pathway is a deep CNN with 4-5 convolutional layers, which can also be flexibly adjusted based on the actual training effect to extract complex, deep features from the motor imagery EEG signal. ReLU is used as the activation function. The output of each convolutional layer is:

[0111] o c =max(0,conv(W) c ,x c )+b c );

[0112] Where conv represents the convolution operator, x c W c ,b c These are the input signal, weight parameters, and bias parameters of the convolutional layer, respectively.

[0113] The two feature extraction pathways are iteratively trained using a loss function. After training, the feature vectors output by the two pathways are fused by element-wise multiplication. The fused features are:

[0114] o i =o c1i ·o c2i ;

[0115] Where i represents the index of the feature, o c1i To extract features for shallow CNNs, o c2i Extracting features for deep CNNs.

[0116] The fused spatial features are then fed into an LSTM as a time series to extract the temporal features of the signal. The tanh activation function is used, and dropout is employed to avoid overfitting. Finally, the output of the LSTM unit is:

[0117] o t =σ(W o ·[hl t-1 ,x t ]+b o );

[0118] hl t =o t ×tanh(C t );

[0119] Where σ() is the sigmoid function, W o and b o These are the weight parameters and bias parameters, respectively. tC represents the current time characteristics of the signal. t This indicates the current state of the updated LSTM cell.

[0120] Finally, a fully connected layer is used as a classification output module to achieve four-class classification of motor imagery EEG signals.

[0121] The cross-entropy function is used as the optimization objective, and iterative training is performed. The loss function is:

[0122]

[0123] Among them, y p For the predicted value, y l This is the actual label value.

[0124] In this embodiment, a learning rate of lr = 0.0001 is defined during training, and the Adam algorithm is used as the optimization method. The parameters are updated during the iteration process until the final stable EEG recognition network model and the iteratively optimized operation parameters are obtained.

[0125] In summary, this invention addresses the shortcomings of current integrated wearable EEG methods and devices for motor imagery recognition, as well as the drawbacks of traditional detection systems, such as poor classification ability, low classification accuracy, poor real-time performance, and inability to achieve real-time movement control of bionic devices. It proposes a portable wearable EEG motor imagery recognition and control device. This invention employs non-invasive multi-channel synchronous EEG acquisition technology to achieve high-performance acquisition of motor imagery EEG signals. By preprocessing the signals to remove environmental noise and interference, a deep learning model based on hybrid neural networks is used to classify and recognize the motor imagery signals. Simultaneously, addressing the insufficient computing power of portable devices, this invention utilizes a heterogeneous computing platform based on the ZYNQ architecture, building a highly parameterized hardware accelerator architecture suitable for hybrid neural networks. This enables high-performance parallel computing of the network model, yielding motor imagery classification results. Finally, the corresponding actions are encoded into action commands to achieve corresponding natural control. Compared to existing acquisition, processing, and recognition systems, the recognition system of this invention is significantly smaller, with power consumption reduced by approximately 95%, and both computational accuracy and speed are significantly improved. The processing time for each frame of signal can reach less than 50 microseconds, while maintaining a recognition accuracy of over 90%.

[0126] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0127] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A hardware-accelerated EEG signal recognition system based on a hybrid neural network, characterized in that, The system includes a signal conversion module, an embedded processor, and a hardware accelerator for EEG algorithms; The signal conversion module is used to digitally convert the input EEG signal to obtain a digital EEG signal; the EEG signal includes a multi-channel EEG signal. The embedded processor is used to preprocess the digital EEG signal and control the parameters of the hardware accelerator; The EEG algorithm hardware accelerator includes a hybrid neural network module, which performs parallel convolution and gating operations on the preprocessed digital EEG signals to extract feature data including spatial and temporal features corresponding to the EEG signals, and performs classification operations to obtain the motor imagery category corresponding to the EEG signals. The hybrid neural network module includes a convolutional neural network processing module and a long short-term memory network processing module. The convolutional neural network processing module is used to extract features of different depths from pre-processed EEG signals of multiple channels through convolution operations based on pre-trained operational parameters, and then perform feature fusion to obtain spatial feature data corresponding to the EEG signals. The convolutional neural network processing module includes a PE array for feature extraction. The PE array includes n parallel channel feature extraction units, where n is the number of channels of the EEG signal; each channel feature extraction unit includes m layers of serial PE modules; The PE module in the first layer is used to perform depth convolution and point convolution operations based on the input EEG signal and operation parameters. The PE modules in subsequent layers perform depth convolution and point convolution operations based on the output of the PE module in the previous layer and operation parameters, respectively, to obtain feature data of different depths extracted by each PE module. The Long Short-Term Memory (LSTM) network processing module is used to perform gating operations and LSTM logic operations based on the spatial feature data to obtain feature data corresponding to the EEG signal, including spatial and temporal features. The LSTM network processing module includes a gating computation unit and an LSTM logic computation unit. The gating computation unit includes four LSTM gate modules: a forget gate, a cell state gate, an input gate, and an output gate. The gating computation unit is based on the currently input feature data. The vector calculated from the previous period After corresponding calculations and activations, the gate vectors of four LSTM gate modules are obtained; the LSTM logic calculation unit is used to perform multiplication, addition and activation operations based on the gate vectors of the four LSTM gate modules to obtain feature data corresponding to the EEG signal, including spatial and temporal features.

2. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 1, characterized in that, The PE module includes a multiply-accumulate operation module and a post-processing module; The multiply-accumulate operation module includes nine multipliers, an adder tree, and one adder. During deep convolution, the input EEG signal data and operation parameters are multiplied and added using the nine multipliers and the adder tree within one cycle to obtain the deep convolution result. During point convolution, one of the multipliers and the adder tree is selected for multiply-accumulate operation. The result of the multiply-accumulate operation is temporarily stored in a buffer, and the current multiply-accumulate result is added to the multiply-accumulate result calculated in the previous cycle using the adder until the point convolution operation of the entire input data is completed, and the final point convolution result is obtained. The post-processing module is used to perform max pooling and activation operations on the convolution operation results to obtain feature data of different depths for the corresponding channels.

3. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 1, characterized in that, The input feature data and corresponding operation parameters of the gated computing unit are input into the four LSTM gate modules in parallel; parallel multiplication operations are performed through each LSTM gate module, and the results are summed through an addition tree to obtain the gate vector of each LSTM gate.

4. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 1, characterized in that, The EEG algorithm hardware accelerator also includes a parameter memory and a feature input memory; The parameter memory is partitioned and allocated to each PE module and LSTM gate module in the hybrid neural network module to store the operation parameters of the corresponding module. The feature input memory adopts a dual first-in-first-out queue structure, and reads the input digital EEG feature data through a sliding window. Each time three lines of data are read, they are stored in three registers of size 3. When the registers are full, a 3*3 feature data matrix is ​​generated and sent to the PE module for processing.

5. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 1, characterized in that, The feature fusion includes: The feature data of different depths extracted by the convolutional neural network processing module are sequentially stored in the external memory DDR, and the feature data of the l-th and m-th layers of n channels are concatenated to obtain two sets of feature vectors of different depths. ; The two sets of feature vectors with different depths are fused by element-wise multiplication, and the fused feature vector is represented as follows: ； Where i is the feature index. The feature vector extracted from the l-th layer. This is the feature vector extracted from the m-th layer.

6. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 4, characterized in that, The hardware accelerator also includes a quantization processing module, which performs quantization operations on each PE module and LSTM gate module after calculation. The quantization operation includes: mapping the data to be quantized by right shifting to map the original data to the required data size; performing rounding during the shifting process; and judging the size of the last digit of the shift operation. If it is greater than 1, it is incremented by 1, otherwise it remains unchanged. Finally, the required number of bits is obtained by truncating the bit using CLP.

7. The hardware-accelerated EEG signal recognition system based on a hybrid neural network according to claim 1, characterized in that, The computational parameters include weight parameters and bias parameters; the computational parameters are obtained by iteratively training the EEG recognition network model using a loss function on a pre-constructed EEG dataset, wherein the EEG dataset includes EEG signal data and the corresponding motor imagery category labels.

Images