A hardware acceleration platform and method for electroencephalogram epilepsy data classification

Abstract

The application discloses a kind of electroencephalogram epilepsy data classification hardware acceleration platform and method, it is related to electroencephalography technical field, including: the PC end, accelerator and off-chip memory connected in turn;PC end is deployed with image processor, image processor is used to obtain electroencephalogram signal, and electroencephalogram signal is converted into gram angle field image;Off-chip memory is used to store the target weight parameter of improved Spikformer model;Accelerator is used to load target weight parameter, and electroencephalogram epilepsy data classification is carried out based on gram angle field image according to improved Spikformer model, and output classification result.Based on the above scheme, after electroencephalogram signal is converted into gram angle field image, accelerator is used to carry out electroencephalogram epilepsy data classification according to improved Spikformer model, and the calculation complexity can be reduced, so as to reduce the calculation power consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram, and particularly relates to a hardware acceleration platform for electroencephalogram epilepsy data classification. BACKGROUND

[0002] Epilepsy is a common neurological disease, and electroencephalogram (EEG) can record the changes of brain electrical signals, and is an important tool for epilepsy data classification to realize diagnosis and monitoring of epilepsy. In order to improve the detection efficiency, deep learning is currently mainly applied to electroencephalogram epilepsy data classification.

[0003] The existing electroencephalogram epilepsy data classification based on deep learning usually adopts a Transformer model. The Transformer is a neural network architecture based on self-attention mechanism, which allows the model to automatically focus on different parts of the sequence when processing sequence data, and gives different weights according to the importance of each part. It is suitable for processing time series data such as electroencephalogram. Although the Transformer architecture is powerful, the self-attention mechanism has high complexity in the case of long sequence. The implemented self-attention mechanism has problems such as not easy to deploy to hardware due to the calculation of multiplication, square root and division. The design is too complex, the calculation amount is large, and the resources are occupied, which leads to high power consumption when classifying electroencephalogram epilepsy data, and limits the application in resource-limited environments (such as edge devices). SUMMARY

[0004] The present application provides a hardware acceleration platform and method for electroencephalogram epilepsy data classification, which solves the technical problem of high power consumption of the existing electroencephalogram epilepsy data classification technology in electroencephalogram epilepsy data classification.

[0005] The first aspect of the present application provides a hardware acceleration platform for electroencephalogram epilepsy data classification, comprising: a PC end, an accelerator and an off-chip memory connected in sequence.

[0006] The PC end is provided with an image processor, which is used to acquire electroencephalogram signals and convert the electroencephalogram signals into Gabor angle field images.

[0007] The off-chip memory is used to store the target weight parameters of the improved Spikformer model.

[0008] The accelerator is used to load the target weight parameters and perform electroencephalogram epilepsy data classification based on the Gabor angle field images according to the improved Spikformer model, and output the classification results.

[0009] Further, the processing process of the improved Spikformer model comprises:

[0010] performing pulse depth separation convolution operation on the gram angle field image to generate a first pulse convolution feature;

[0011] performing multiple pulse depth separation convolution operations on the first pulse convolution feature to map into a pulse query matrix, a pulse key matrix and a pulse value matrix respectively;

[0012] adding the pulse query matrix and the pulse key matrix column by column according to bit phase, and adding the pulse value matrix according to bit phase after pulse nonlinear mapping to generate a pulse attention feature through pulse matrix dot product operation;

[0013] performing pulse full connection calculation on the pulse attention feature to determine a first pulse transformation feature;

[0014] connecting the first pulse convolution feature and the first pulse transformation feature by residual connection to output a bit phase addition feature;

[0015] performing pulse full connection calculation on the bit phase addition feature to construct a second pulse transformation feature;

[0016] connecting the bit phase addition feature and the second pulse transformation feature by residual connection, and performing pulse full connection calculation to output a classification result;

[0017] The pulse depth separation convolution operation comprises sequentially performing pulse depth convolution operation and pulse point convolution operation.

[0018] Further, the accelerator comprises a pulse detection model controller, a pulse matrix dot product component, a pulse convolution component and a pulse full connection component, a pulse buffer and a model parameter buffer;

[0019] The pulse detection model controller is connected with the pulse matrix dot product component, the pulse convolution component, the pulse full connection component, the pulse buffer and the PC terminal respectively;

[0020] The pulse buffer is connected with the pulse matrix dot product component, the pulse convolution component and the pulse full connection component respectively;

[0021] The model parameter buffer is connected with the pulse convolution component, the pulse full connection component and the off-chip memory respectively;

[0022] The pulse detection model controller is used to control the pulse matrix dot product component, the pulse convolution component and the pulse full connection component to operate and output a classification result according to the improved Spikformer model based on the gram angle field image;

[0023] The model parameter buffer includes a weight buffer for buffering target weight parameters and a neuron parameter buffer for buffering neuron state parameters.

[0024] The pulse convolution component is configured to load the target weight parameters and the neuron state parameters to perform pulse deep separation convolution operation and output pulse convolution features.

[0025] The pulse matrix dot product component is configured to perform pulse matrix dot product operation and output pulse attention features.

[0026] The pulse full connection component is configured to load the target weight parameters and the neuron state parameters to perform pulse full connection calculation and output pulse transformation features.

[0027] The pulse buffer is configured to buffer the pulse convolution features, the pulse attention features, the pulse transformation features and classification results, and perform feature bit-by-bit addition.

[0028] Further, the pulse matrix dot product component includes a pulse matrix dot product controller, a matrix dot product calculation core and a pulse matrix dot product buffer connected in sequence.

[0029] The pulse matrix dot product controller is connected with the pulse detection model controller.

[0030] The matrix dot product calculation core includes a pulse column summation unit and a bit-by-bit phase and calculation unit.

[0031] The pulse column summation unit is configured to perform column-by-column addition to output pulse summation features.

[0032] The bit-by-bit phase and calculation unit is configured to perform bit-by-bit phase and to output pulse phase and features.

[0033] The pulse matrix dot product buffer is configured to buffer the pulse summation features and the pulse phase and features, and perform pulse nonlinear mapping.

[0034] Further, the pulse convolution component includes a pulse convolution controller and a processing unit array.

[0035] The pulse convolution controller is connected with the pulse detection model controller and the processing unit array, respectively.

[0036] The processing unit array is connected with the pulse buffer and the model parameter buffer, respectively.

[0037] The processing unit array includes a plurality of processing units in an array, and the processing units are configured to perform event-driven pulse point convolution operation or pulse deep convolution operation.

[0038] Further, the pulse full connection component comprises a pulse full connection controller and a linear unit.

[0039] The pulse full connection controller is connected with the pulse detection model controller and the linear unit respectively.

[0040] The linear unit is connected with the pulse buffer and the model parameter buffer respectively.

[0041] The linear unit is used for pulse full connection calculation.

[0042] Further, the target weight parameter acquisition process comprises:

[0043] The improved Spikformer model to be trained is trained by using a training image set, and a trained improved Spikformer model is determined.

[0044] The trained improved Spikformer model is subjected to weight quantization training, and a target weight parameter is determined.

[0045] The second aspect of the present application provides a hardware acceleration method for electroencephalogram epilepsy data classification, comprising:

[0046] An electroencephalogram signal is acquired, and the electroencephalogram signal is converted into a Gram angle field image.

[0047] According to the improved Spikformer model, electroencephalogram epilepsy data classification is performed based on the Gram angle field image, and a classification result is output.

[0048] The third aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above aspects.

[0049] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above aspects.

[0050] The fifth aspect of the present application provides a computer program product, comprising computer programs / instructions, which are executed by a processor to implement the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above aspects.

[0051] From the above technical solutions, the present application has the following advantages:

[0052] The first aspect of the application provides a hardware acceleration platform for electroencephalogram epilepsy data classification, comprising: a PC end, an accelerator and an off-chip memory connected in sequence; the PC end is provided with an image processor, the image processor is used to acquire an electroencephalogram signal and convert the electroencephalogram signal into a Gram angle field image; the off-chip memory is used to store target weight parameters of an improved Spikformer model; the accelerator is used to load the target weight parameters and perform electroencephalogram epilepsy data classification based on the Gram angle field image according to the improved Spikformer model, and output a classification result. Based on the above scheme, after the electroencephalogram signal is converted into a Gram angle field image, the accelerator is used to perform electroencephalogram epilepsy data classification according to the improved Spikformer model, so that the calculation complexity can be reduced, thereby reducing the calculation power consumption.

[0053] The second aspect of the application provides a hardware acceleration method for electroencephalogram epilepsy data classification, comprising: acquiring an electroencephalogram signal and converting the electroencephalogram signal into a Gram angle field image; performing electroencephalogram epilepsy data classification based on the Gram angle field image according to an improved Spikformer model, and outputting a classification result. Based on the above scheme, after the electroencephalogram signal is converted into a Gram angle field image, the improved Spikformer model is used to perform electroencephalogram epilepsy data classification, so that the calculation complexity can be reduced, thereby reducing the calculation power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0055] Figure 1 A structure schematic diagram of a hardware acceleration platform for electroencephalogram epilepsy data classification provided by the embodiment of the application is provided.

[0056] Figure 2 A structure schematic diagram of training and reasoning of an improved Spikformer model provided by the embodiment of the application is provided.

[0057] Figure 3 A structure schematic diagram of an improved Spikformer model provided by the embodiment of the application is provided.

[0058] Figure 4 A data preprocessing schematic diagram of an electroencephalogram signal provided by the embodiment of the application is provided.

[0059] Figure 5 A structure schematic diagram of a pulse self-attention module provided by the embodiment of the application is provided.

[0060] Figure 6 A feature processing mapping schematic diagram of the improved Spikformer model in the accelerator is provided for the embodiment of the present application.

[0061] Figure 7 A step flowchart of the hardware acceleration method for electroencephalogram epilepsy data classification provided for the embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiment of the present application provides a hardware acceleration platform and method for electroencephalogram epilepsy data classification, which is used for solving the technical problem of high power consumption when the existing electroencephalogram epilepsy data classification technology is used for electroencephalogram epilepsy data classification.

[0063] In order to make the purpose, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the following described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0064] Term explanation:

[0065] SNN: Spiking Neural Network.

[0066] Transformer: A deep learning network architecture.

[0067] LIF: Leaky Integrate-and-Fire neuron.

[0068] BPTT: Backpropagation Through Time.

[0069] EEG: Electroencephalogram.

[0070] Spikformerr: Spiking Transformer network architecture.

[0071] MLP (Multilayer Perceptron): A feedforward artificial neural network.

[0072] Please refer to Figure 1 , Figure 1 A structural schematic diagram of the hardware acceleration platform for electroencephalogram epilepsy data classification provided for the embodiment of the present application.

[0073] The hardware acceleration platform for electroencephalogram epilepsy data classification provided by the present application comprises a PC end, an accelerator and an off-chip memory connected in sequence.

[0074] The PC end is deployed with an image processor, which is configured to acquire the electroencephalogram signal and convert the electroencephalogram signal into a GASF image;

[0075] An off-chip memory is configured to store target weight parameters of the improved Spikformer model.

[0076] An accelerator is configured to load the target weight parameters and perform electroencephalogram epilepsy data classification based on the GASF image according to the improved Spikformer model, and output a classification result.

[0077] It should be noted that the hardware acceleration platform for electroencephalogram epilepsy data classification in the embodiment performs data interaction between the PC end, the accelerator and the off-chip memory connected in sequence: first, the image processor deployed on the PC end performs normalization and polar coordinate processing on the acquired electroencephalogram signal, and then converts it into a corresponding GASF image through GASF image processing; the target weight parameters refer to the weight parameters used in actual application after the model is trained; after the accelerator receives the GASF image sent by the PC end, the accelerator loads the corresponding target weight parameters from the off-chip memory according to the hierarchical logic of the improved Spikformer model to complete the electroencephalogram epilepsy data classification, thereby outputting the classification result; storing the target weight parameters in the off-chip memory can reduce the occupation of the accelerator resources, and the hardware-friendly design of the improved Spikformer model makes it easier to be deployed in FPGA hardware.

[0078] It can be understood that the improved Spikformer model in the embodiment is improved on the basis of Spikformer; thanks to the computing characteristics of SNN, SNN has obvious advantages in power consumption compared with traditional neural networks with the same structure; similarly, Spikformer, which combines SNN and transformer, inherits the characteristics of low power consumption, high energy efficiency, and the ability to process spatio-temporal information of pulse neural networks, as well as the strong expression ability of transformer; in the improved Spikformer model, in one implementation manner, the problem that the self-attention mechanism of the traditional Transformer architecture is not easy to deploy on hardware due to the inclusion of multiplication, square root, division and other calculations can be solved by using simple operations and LIF neurons to replace complex calculations such as softmax, and hardware-friendly design is achieved.

[0079] In one specific implementation manner, the processing process of the improved Spikformer model includes:

[0080] The GASF image is subjected to pulse depth separation convolution operation to generate first pulse convolution features.

[0081] The first pulse convolution feature is subjected to a plurality of pulse depth separation convolution operations, and is respectively mapped into a pulse query matrix, a pulse key matrix and a pulse value matrix;

[0082] The pulse query matrix and the pulse key matrix are added column by column after bit phase addition, and are subjected to pulse nonlinear mapping and bit phase addition with the pulse value matrix, so as to generate a pulse attention feature through pulse matrix dot product operation;

[0083] The pulse attention feature is subjected to pulse full connection calculation to determine a first pulse transformation feature;

[0084] The first pulse convolution feature and the first pulse transformation feature are subjected to residual connection, and a bit phase addition feature is outputted;

[0085] The bit phase addition feature is subjected to pulse full connection calculation to construct a second pulse transformation feature;

[0086] After the bit phase addition feature and the second pulse transformation feature are subjected to residual connection, pulse full connection calculation is performed to output a classification result;

[0087] The pulse depth separation convolution operation includes pulse depth convolution operation and pulse point convolution operation in sequence.

[0088] It should be noted that, as shown in Figures 2-5 The improved Spikformer model includes a pulse coding module, an encoder and a classification head connected in sequence; the pulse coding module includes a pulse depth separation convolution layer; the encoder includes a pulse self-attention (SSA) module and an MLP module connected in cascade with residual connection, the MLP module includes at least one pulse full connection layer, the pulse full connection layer includes a Linear unit, a BN unit and a LIF neuron; the pulse self-attention module includes a plurality of pulse depth separation convolution layers, a LIF neuron and a pulse full connection layer; the classification head includes a pulse full connection layer;

[0089] In the specific implementation, first, the Gram angular field image of the input pulse coding module is subjected to a pulse depth separation convolution operation through a pulse depth separation convolution layer to generate a first pulse convolution feature. Then, in the encoder, the first pulse convolution feature is respectively subjected to pulse depth separation convolution operation based on multiple pulse depth separation convolution layers to be mapped into a pulse query matrix Q, a pulse key matrix K and a pulse value matrix V. The pulse query matrix and the pulse key matrix are phase-wise ANDed and then added column by column. After the pulse nonlinear mapping is performed according to the LIF neuron, the pulse is phase-wise ANDed with the pulse value matrix to generate a pulse attention feature through a pulse matrix dot product operation in the pulse self-attention module. The self-attention mechanism contained in this part allows the model to automatically pay attention to nonlinear data in the sequence when processing sequence data. The same part is divided into two parts, and different weights are assigned according to the importance of each part. This enables the model to better capture long-distance dependencies, which helps to improve the accuracy of the model in the EEG epilepsy data classification task. The pulse attention feature is calculated by the fully connected layer to determine the first pulse transformation feature. The first pulse convolution feature is residually connected with the first pulse transformation feature, and the bitwise addition feature is output. The fully connected layer of the MLP module is used to perform pulse full connection calculation on the bitwise addition feature for further high-level feature extraction, and the second pulse transformation feature is constructed. After the bitwise addition feature is residually connected with the second pulse transformation feature, the pulse full connection calculation is performed through the fully connected layer of the classification head to realize EEG epilepsy data classification and output the classification result.

[0090] Among them, the pulse depth separation convolution layer includes depth convolution, BN unit, LIF neuron, point convolution, BN unit and LIF neuron, and the processing process of the pulse depth separation convolution unit includes: performing pulse depth convolution operation on the input pulse convolution input feature through cascaded depth convolution, BN unit and LIF neuron to generate depth feature, using cascaded point convolution, BN unit and LIF neuron to perform pulse point convolution operation on the depth feature, and outputting pulse convolution output feature; therefore, more specifically, the pulse depth separation convolution operation performed by the pulse depth separation convolution layer includes sequentially performing pulse depth convolution operation and pulse point convolution operation, the pulse depth convolution operation includes sequentially performing depth convolution operation, batch normalization and pulse nonlinear mapping, and the pulse point convolution operation includes sequentially performing point convolution operation, batch normalization and pulse nonlinear mapping;

[0091] It can be understood that the pulse encoding module, the encoder and the classification head ensure that the data running therein are all pulse data by setting the pulse nonlinear mapping of the LIF neuron, thereby reducing the power consumption of network operation; at the same time, since the processed data is converted from the electroencephalogram signal to the Gram angle field image, the pulse deep separable convolution operation more suitable for image feature extraction can be used to extract the features of the image, thereby extracting the local information correlation features and spatial hierarchical structure in the image. Correspondingly, the pulse self-attention module of the embodiment no longer uses the linear layer to extract the corresponding QKV data, but uses the convolution operation more suitable for image data feature extraction. In order to reduce the parameter quantity of the model, the self-attention operation composed of the pulse deep separable convolution is used. Since the features obtained through the LIF neuron are all pulse data, the Q and K can be calculated directly by using the bitwise exclusive OR operation plus the addition operation. The pulse data obtained through the LIF neuron is only 0 and 1, so it is not necessary to limit the range of data by the Scale and Softmax operations. The pulse data is naturally limited to 0 and 1. Then, the obtained result is subjected to the bitwise exclusive OR operation with V to obtain the pulse attention feature after the weight distribution processing of the result. The pulse self-attention module of the embodiment uses the pulse deep separable convolution to replace the QKV extraction operation composed of the pulse full connection layer, and replaces the traditional matrix multiplication, Scale operation and Softmax operation. Since the elements of the pulse data are only 1 or 0, the pulse data is very sparse, and the corresponding self-attention calculation consumption is greatly reduced, thereby helping to reduce the parameter quantity of the network and speed up the calculation speed.

[0092] It is understandable that many different neuron models can be used as basic units for building SNN networks, such as the Hodgkin-Huxley (HH) model, the Izhikevich (IZH) model, the Integred-and-Fire (IF) model, and the Leaky Integred-and-Fire (LIF) model. These neuron models can realistically simulate the operating modes of biological neurons from the perspective of neuroscience. Among them, the HH model has high accuracy but is too complex. Although the IZH model is a simplified version of the HH model and its biological accuracy is close to that of the HH model, its computational complexity is still very high. The IF model is the simplest, with the smallest amount of computation and high hardware implementation efficiency. However, due to its oversimplification, it may not be able to accurately simulate the complex behaviors of some biological neurons. Although the accuracy of the LIF model is slightly lower than that of the IZH model and its computational cost is slightly greater than that of the IF model, the LIF model is simpler and more intuitive than the IZH. At the same time, its performance and biological interpretability are better than the IF model. It can also solve the problem of the lack of time dependence of the IF model and can provide the performance required by the SNN network. Therefore, it is widely used to implement various SNN networks.

[0093] LIF neurons integrate input current in a leaky manner , and at its membrane potential Crossing a fixed threshold from below When an action potential is triggered, a spike is emitted at this time, and the process is modeled as a nonlinear function, after which the membrane potential is reset to:

[0094] ;

[0095] Where, is a spike train The input signal, is a time constant The exponentially decaying membrane potential, is the membrane resistance, the spike The emission is expressed as a nonlinear function of threshold and potential.

[0096] In one embodiment, the accelerator includes a pulse detection model controller, a pulse matrix dot product component, a pulse convolution component, a pulse fully connected component, a pulse buffer, and a model parameter buffer;

[0097] The pulse detection model controller is respectively connected to the pulse matrix dot product component, the pulse convolution component, the pulse fully connected component, the pulse buffer and the PC end;

[0098] The pulse buffer is connected to the pulse matrix dot product component, the pulse convolution component, and the pulse fully connected component respectively;

[0099] The model parameter buffer is connected with the pulse convolution component, the pulse full connection component and the off-chip memory respectively.

[0100] The pulse detection model controller is configured to control the pulse matrix multiplication component, the pulse convolution component and the pulse full connection component to output a classification result according to an improved Spikformer model based on a Gram angle field image.

[0101] The model parameter buffer includes a weight buffer and a neuron parameter buffer, the weight buffer is configured to buffer target weight parameters, and the neuron parameter buffer is configured to buffer neuron state parameters.

[0102] The pulse convolution component is configured to load the target weight parameters and the neuron state parameters to perform pulse deep separation convolution operation and output pulse convolution features.

[0103] The pulse matrix multiplication component is configured to perform pulse matrix multiplication operation and output pulse attention features.

[0104] The pulse full connection component is configured to load the target weight parameters and the neuron state parameters to perform pulse full connection calculation and output pulse transformation features.

[0105] The pulse buffer is configured to buffer the pulse convolution features, the pulse attention features, the pulse transformation features and the classification result, and perform feature bitwise addition.

[0106] In a more specific embodiment, the pulse matrix multiplication component includes a pulse matrix multiplication controller, a matrix multiplication calculation core and a pulse matrix multiplication buffer connected in sequence.

[0107] The pulse matrix multiplication controller is connected with the pulse detection model controller.

[0108] The matrix multiplication calculation core includes a pulse column summation unit and a bitwise phase and calculation unit.

[0109] The pulse column summation unit is configured to perform column-by-column addition to output pulse summation features.

[0110] The bitwise phase and calculation unit is configured to perform bitwise phase and to output pulse phase and features.

[0111] The pulse matrix multiplication buffer is configured to buffer the pulse summation features and the pulse phase and features, and perform pulse nonlinear mapping.

[0112] In a more specific embodiment, the pulse convolution component includes a pulse convolution controller and a processing unit array.

[0113] The pulse convolution controller is connected with the pulse detection model controller and the processing unit array respectively.

[0114] The processing element array is connected with the pulse buffer and the model parameter buffer respectively;

[0115] The processing element array includes a plurality of processing elements of the array, and the processing elements are configured to perform event-driven pulse point convolution operations or pulse depth convolution operations.

[0116] In a more specific embodiment, the pulse full connection component includes a pulse full connection controller and a linear unit;

[0117] The pulse full connection controller is connected with the pulse detection model controller and the linear unit respectively;

[0118] The linear unit is connected with the pulse buffer and the model parameter buffer respectively;

[0119] The linear unit is configured to perform pulse full connection calculation.

[0120] It should be noted that according to the framework of the improved Spikformer model, the whole model is mainly divided into three parts, as shown in Figure 6 ① pulse coding operation, ② SSA operation and ③ MLP operation, and correspondingly, the accelerator mainly includes a controller part, a buffer part and a computing core part. The controller part includes a pulse detection model controller (Spikformer controller) and each computing core controller. The pulse detection model controller is deployed with the hierarchical logic of the improved Spikformer model, is responsible for controlling the loading of the stored data, and the start of the remaining computing core controllers and the loading and caching of the buffer data, so as to realize the correct operation of the data flow of the whole system. The computing core part includes a processing element array (PE_Array) part for implementing pulse convolution operation, a linear unit (Linear) part for implementing pulse full connection calculation and a matrix dot-product core part for implementing attention calculation. The buffer part is used to cache the neuron state parameters and the weights and intermediate output pulses and other data in the calculation process;

[0121] Specifically, the accelerator is designed to include a spike detection model controller (Spikformer controller), a spike matrix dot product component, a spike convolution component and a spike full connection component, an off-chip memory and a model parameter cache, the model parameter cache includes a weight cache and a neuron parameter cache, the spike convolution component includes a spike convolution controller and a processing unit array, the processing unit array (PE_Array) includes a plurality of processing units (PE) of the array, each processing unit (PE) is used to simulate a pulse deep convolution structure of a cascaded deep convolution-BN unit-LIF neuron or a pulse point convolution structure of a cascaded point convolution-BN unit-LIF neuron, the spike matrix dot product component includes a spike matrix dot product controller, a matrix dot product calculation core and a spike matrix dot product cache connected in sequence, and the spike full connection component includes a spike full connection controller and a linear unit; in specific implementation:

[0122] Firstly, the spike encoding operation of the pulse encoding module is performed, that is, ①-(a) pulse deep separation convolution operation: the pulse detection model controller calls the pulse convolution controller, the pulse convolution controller loads the corresponding target weight parameter from the off-chip memory through the weight cache, and loads the membrane potential and other neuron state parameters from the neuron parameter cache; based on the Gram angle field image, the data is sorted according to the deep convolution, and then input into the processing unit array for pulse deep convolution operation and cache the operation result in the pulse cache; at the same time, in the operation process of the LIF neuron, the new neuron state parameter is output and cached in the neuron parameter cache; then, the operation result of the deep convolution is called from the pulse cache and the data is arranged according to the point convolution and input into the processing unit array for pulse point convolution operation, and the output pulse convolution feature is the first pulse convolution feature of the pulse encoding module and is cached in the pulse cache; at the same time, the plurality of processing units (PE) of the processing unit array can operate in parallel, and are designed to realize the response to only non-zero pulses in cooperation with event-driven, that is, if the input pulse data is a non-zero pulse, the corresponding neuron weight and neuron state are loaded for calculation, and if the input is 0, no data is loaded for corresponding operation, thereby accelerating the processing speed and energy consumption of the network;

[0123] Then, in the encoder part: 1) performing the operation of the SSA module: the pulse detection model controller controls the pulse convolution controller to load the first pulse convolution feature into the pulse cache for three times of ①-(a) pulse depth separation convolution operation ②-(a) pulse depth separation convolution, so as to obtain the QKV cache to the pulse cache; the pulse matrix point multiplication controller loads the QKV pulse into the pulse matrix point multiplication cache based on the matrix point multiplication calculation core to perform ②-(b) pulse matrix point multiplication operation, which includes: performing the bitwise AND operation by the bitwise AND calculation unit using Q and K to obtain the pulse AND feature cache to the pulse matrix point multiplication cache, performing the column-wise summation operation by the pulse column-wise summation unit using the pulse AND feature to output the pulse summation feature cache to the pulse matrix point multiplication cache, and performing the pulse nonlinear mapping on the pulse summation feature in the pulse matrix point multiplication cache, and then performing the bitwise AND operation by the bitwise AND calculation unit using the pulse summation feature after the nonlinear mapping and V to obtain the new pulse summation feature, i.e., the pulse attention feature extracted by the pulse self-attention mechanism, and cache it to the pulse cache; the pulse full connection controller loads the pulse attention feature, the corresponding target weight parameter and the neuron state parameter based on the linear unit to perform ②-(c) pulse full connection operation, which simulates the pulse full connection calculation, i.e., the linear transformation of the Linear unit, the batch normalization of the BN unit and the pulse nonlinear mapping of the LIF neuron in sequence, and outputs the pulse transformation feature, i.e., the corresponding first pulse transformation feature, to the pulse cache, so as to complete the SSA operation; 2) performing the feature bitwise phase addition on the first pulse convolution feature and the first pulse transformation feature in the pulse cache to obtain the bitwise phase addition feature, which is similar to the residual connection operation; 3) controlling the pulse full connection controller to load the bitwise phase addition feature to perform ③-(a) pulse full connection operation, so as to complete the pulse full connection calculation of the MLP module, and output the new pulse transformation feature, i.e., the second pulse transformation feature, to the pulse cache;

[0124] Finally, the control pulse full connection controller is controlled to load the second pulse transformation feature to perform ③-(a) pulse full connection operation, so as to complete the pulse full connection calculation of the classification head, and output the new pulse transformation feature, i.e., the classification result, to the pulse cache.

[0125] In one specific embodiment, the accelerator is further used to send the classification result to the PC end for display. More specifically, the PC end is connected with the pulse cache.

[0126] In one specific embodiment, the target weight parameter acquisition process includes:

[0127] The improved Spikformer model to be trained is trained using the training image set, and the trained improved Spikformer model is determined.

[0128] The improved Spikformer model trained is subjected to weight quantization training to determine the target weight parameters.

[0129] It should be noted that in order to enable the hardware acceleration platform based on the improved Spikformer model to exhibit better performance, the improved Spikformer model is trained in the following manner: Figure 2 As shown in the figure, after the improved Spikformer model is built, the training image set is used in the training part of the Spikformer model, and the Multi-Gaussian function is used as the substitute function of the BPTT algorithm to train the improved Spikformer model to be trained, and after the trained improved Spikformer model is obtained, the pre-trained weight parameters are used for weight quantization processing, and the weight quantization training is used to fine-tune the trained improved Spikformer model, which is conducive to the improvement of hardware inference accuracy, and after the quantization training is completed, the quantized weight parameters, i.e. the target weight parameters, can be loaded into the hardware architecture.

[0130] The weight quantization training aims to quantize the weight tensor W of the model by using a quantization function The weight tensor W of the model is subjected to quantization processing, including: first, a quantization factor n is set, the continuous decimal value domain is converted into a discrete integer value set by multiplying x by n and performing a rounding operation; then, the obtained integer value is divided by n to reduce the value to 1 / n times; finally, the clip() function is used to limit the range of the reduced value, ensuring that all values are within the interval [a, b], where values less than a are adjusted to a, and values greater than b are adjusted to b; through this quantization process, the value originally represented by a 32-bit floating point number can be effectively mapped to an 8-bit integer value within the interval [a, b], thereby significantly reducing the computational complexity and improving the processing efficiency.

[0131] In the embodiment of the present application, the electroencephalogram signal is converted into a Gram angle field image for a simple data compression, and then the electroencephalogram epilepsy data classification is performed through the improved Spikformer model. In the feature processing process, the feature maintains the pulse data form. The pulse depth separation convolution is used to effectively capture the local feature and spatial hierarchy. The pulse self-attention mechanism is used to replace the traditional self-attention mechanism, which can capture long-distance dependencies. The pulse self-attention mechanism uses binary pulse signals for calculation, which can convert all matrix multiplications related to the pulse matrix into sparse additions. Thus, the complex multiplication calculation and matrix calculation in the self-attention mechanism are converted into simple column-by-column addition operation and bitwise AND operation. In addition, the pulse data is sparse in nature, which can make the model more efficient when processing large-scale sparse data. Combined with the event-driven characteristics, the hardware friendliness can be realized as a whole. Thus, the hardware acceleration platform supporting the improved Spikformer model based on FPGA is built to accelerate the electroencephalogram epilepsy data classification, realize high-performance and low-power electroencephalogram classification. In addition, due to the low energy consumption, the improved Spikformer model can be deployed on the edge device and applied to the wearable device.

[0132] Please refer to Figure 7 , Figure 7 The step flow chart of the hardware acceleration method for electroencephalogram epilepsy data classification provided by the embodiment of the present application is shown in the figure.

[0133] The hardware acceleration method for electroencephalogram epilepsy data classification provided by the present application comprises the following steps:

[0134] Step 701: Obtain the electroencephalogram signal and convert the electroencephalogram signal into a Gram angle field image.

[0135] It should be noted that when the electroencephalogram signal is obtained, the electroencephalogram signal is normalized and polar coordinate processed, and then converted into the corresponding Gram angle field image through Gram angle field imaging processing.

[0136] Step 702: Perform electroencephalogram epilepsy data classification based on the Gram angle field image according to the improved Spikformer model, and output the classification result.

[0137] It should be noted that the improved Spikformer model comprises a pulse coding module, an encoder and a classification head. The pulse coding module comprises a pulse depth separation convolution layer. The encoder comprises a pulse self-attention module and an MLP module. The MLP module comprises at least one pulse full connection layer. The pulse full connection layer comprises a Linear unit, a BN unit and a LIF neuron. The pulse self-attention module comprises a plurality of pulse depth separation convolution layers, LIF neurons and pulse full connection layers. The classification head comprises a pulse full connection layer.

[0138] The processing procedure of the improved Spikformer model comprises: performing pulse depth separation convolution operation on the Gram angle field image to generate first pulse convolution features; performing multiple pulse depth separation convolution operations on the first pulse convolution features to respectively map into a pulse query matrix, a pulse key matrix and a pulse value matrix; performing bitwise phase addition on the pulse query matrix and the pulse key matrix column by column, and performing pulse nonlinear mapping and then performing bitwise phase addition with the pulse value matrix to generate pulse attention features through pulse matrix dot product operation; performing pulse full connection calculation on the pulse attention features to determine first pulse transformation features; performing residual connection on the first pulse convolution features and the first pulse transformation features to output bitwise phase addition features; performing pulse full connection calculation on the bitwise phase addition features to construct second pulse transformation features; performing residual connection on the bitwise phase addition features and the second pulse transformation features, and then performing pulse full connection calculation to output a classification result; the pulse depth separation convolution operation comprises sequentially performing pulse depth convolution operation and pulse point convolution operation.

[0139] The pulse depth separation convolution layer comprises depth convolution, a BN unit, an LIF neuron, point convolution, a BN unit and an LIF neuron; and the processing procedure of the pulse depth separation convolution unit comprises: performing pulse depth convolution operation on the input pulse convolution input features through the cascaded depth convolution, the BN unit and the LIF neuron to generate depth features, and performing pulse point convolution operation on the depth features through the cascaded point convolution, the BN unit and the LIF neuron to output pulse convolution output features.

[0140] In the embodiment of the present application, the electroencephalogram signal is converted into a Gram angle field image for simple data compression, and then the improved Spikformer model is used for electroencephalogram epilepsy data classification. In the feature processing procedure, the features remain in the form of pulse data, the local features and spatial hierarchical structure are effectively captured by using the inherent locality of the pulse depth separation convolution, the complex multiplication calculation and matrix calculation in the self-attention mechanism are converted into simple column-by-column addition operation and bitwise phase addition operation, and high-performance and low-power electroencephalogram epilepsy data classification can be realized.

[0141] The embodiment of the present application also provides a computer device comprising a memory and a processor, and the memory stores a computer program; when the computer program is executed by the processor, the processor executes the steps of the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above embodiments.

[0142] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above embodiments are implemented.

[0143] The embodiment of the present application further provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the hardware acceleration method for electroencephalogram epilepsy data classification according to any one of the above embodiments.

[0144] In several embodiments provided in the present application, it should be understood that the disclosed hardware acceleration platform and method can be implemented in other manners. For example, the above-described hardware acceleration platform embodiments are merely illustrative, and the division of the units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0145] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments.

[0146] In addition, each function unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software function unit.

[0147] When the integrated unit is implemented in the form of software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various media that can store program codes.

[0148] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hardware acceleration platform for electroencephalogram epilepsy data classification, characterized in that, The method comprises the following steps: sequentially connecting a PC end, an accelerator and an off-chip memory; the PC end is provided with an image processor, which is used to acquire electroencephalogram signals and convert the electroencephalogram signals into Gram angle field images; the off-chip memory is used to store target weight parameters of an improved Spikformer model; the accelerator is used to load the target weight parameters and perform electroencephalogram epilepsy data classification based on the Gram angle field images according to the improved Spikformer model, and output a classification result; the processing process of the improved Spikformer model comprises the following steps: performing pulse depth separation convolution operation on the Gram angle field images to generate first pulse convolution features; performing multiple pulse depth separation convolution operations on the first pulse convolution features to respectively map them into a pulse query matrix, a pulse key matrix and a pulse value matrix; performing bitwise phase sum of the pulse query matrix and the pulse key matrix column by column, and performing pulse nonlinear mapping and bitwise phase sum with the pulse value matrix to generate pulse attention features through pulse matrix dot product operation; performing pulse full connection calculation on the pulse attention features to determine first pulse transformation features; performing residual connection of the first pulse convolution features and the first pulse transformation features to output a bitwise phase addition feature; performing pulse full connection calculation on the bitwise phase addition feature to construct second pulse transformation features; performing pulse full connection calculation on the bitwise phase addition feature and the second pulse transformation feature after residual connection to output a classification result; the pulse depth separation convolution operation comprises sequentially performing pulse depth convolution operation and pulse point convolution operation.

2. The hardware acceleration platform for electroencephalogram epilepsy data classification of claim 1, wherein, The accelerator comprises a pulse detection model controller, a pulse matrix dot product component, a pulse convolution component and a pulse full connection component, a pulse buffer and a model parameter buffer; the pulse detection model controller is connected with the pulse matrix dot product component, the pulse convolution component, the pulse full connection component, the pulse buffer and the PC end respectively; the pulse buffer is connected with the pulse matrix dot product component, the pulse convolution component and the pulse full connection component respectively; the model parameter buffer is connected with the pulse convolution component, the pulse full connection component and the off-chip memory respectively; the pulse detection model controller is used to control the pulse matrix dot product component, the pulse convolution component and the pulse full connection component to operate and output a classification result based on the Gram angle field images according to the improved Spikformer model; the model parameter buffer comprises a weight buffer and a neuron parameter buffer, the weight buffer is used to buffer target weight parameters, and the neuron parameter buffer is used to buffer neuron state parameters; the pulse convolution component is used to load the target weight parameters and the neuron state parameters to perform pulse depth separation convolution operation and output pulse convolution features; the pulse matrix dot product component is used to perform pulse matrix dot product operation and output pulse attention features; the pulse full connection component is used to load the target weight parameters and the neuron state parameters to perform pulse full connection calculation and output pulse transformation features. The pulse buffer is used for buffering pulse convolution features, pulse attention features, pulse transformation features and classification results, and performing feature bit-by-bit addition.

3. The hardware acceleration platform for electroencephalogram epilepsy data classification of claim 2, wherein, The pulse matrix point multiplication component includes a pulse matrix point multiplication controller, a matrix point multiplication calculation core and a pulse matrix point multiplication buffer connected in sequence. The pulse matrix point multiplication controller is connected with the pulse detection model controller. The matrix point multiplication calculation core includes a pulse column summation unit and a bit-by-bit phase and calculation unit. The pulse column summation unit is used for performing column-by-column addition to output pulse summation features. The bit-by-bit phase and calculation unit is used for performing bit-by-bit phase and to output pulse and features. The pulse matrix point multiplication buffer is used for buffering pulse summation features and pulse and features, and performing pulse nonlinear mapping.

4. The hardware acceleration platform for electroencephalogram epilepsy data classification of claim 2, wherein, The pulse convolution component includes a pulse convolution controller and a processing unit array. The pulse convolution controller is connected with the pulse detection model controller and the processing unit array respectively. The processing unit array is connected with the pulse buffer and the model parameter buffer respectively. The processing unit array includes a plurality of processing units in an array, and the processing units are used for performing event-driven pulse point convolution operation or pulse deep convolution operation.

5. The hardware acceleration platform for electroencephalogram epilepsy data classification of claim 2, wherein, The pulse full connection component includes a pulse full connection controller and a linear unit. The pulse full connection controller is connected with the pulse detection model controller and the linear unit respectively. The linear unit is connected with the pulse buffer and the model parameter buffer respectively. The linear unit is used for pulse full connection calculation.

6. The hardware acceleration platform for electroencephalogram epilepsy data classification of claim 1, wherein, The target weight parameter acquisition process includes: Training the improved Spikformer model to be trained using a training image set to determine a trained improved Spikformer model; Performing weight quantization training on the trained improved Spikformer model to determine the target weight parameter.

7. A method of hardware acceleration for electroencephalogram epilepsy data classification, the method comprising: The hardware acceleration platform applied to the electroencephalogram epilepsy data classification of any one of claims 1 to 6, comprising: Obtaining an electroencephalogram signal and converting the electroencephalogram signal into a Gram angle field image; Classifying electroencephalogram epilepsy data based on the Gram angle field image according to the improved Spikformer model, and outputting a classification result.

8. A computer device, comprising: The computer program / instructions are executed by the processor to implement the steps of the method of claim 7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of claim 7.

Images