EEG motor imagery classification system using spectrogram images and deep learning method for rehabilitation applications

Abstract

TITLE: EEG MOTOR IMAGERY CLASSIFICATION SYSTEM USING SPECTROGRAM IMAGES AND DEEP LEARNING METHOD FOR REHABILITATION APPLICATIONS APPLICANT: KARUNYA INSTITUTE OF TECHNOLOGY AND SCIENCES ABSTRACT The present invention discloses an EEG motor imagery classification system for neuro-rehabilitation of stroke patients. The system of the present invention comprises of an EEG device coupled to a processing device and an output device, characterized in that • the EEG device comprises of 64-electrode adapted to be placed on scalp and configured to acquire electroencephalogram (EEG) signals; • the processing device adapted to receive EEG signals and o inbuilt with Short Term Fourier Transform (STFT) configured to extract spectrogram images; o inbuilt with Inception v3 network model adapted configured to classify left fist and right fist from the spectrogram images; • the output device adapted to display the spectrogram images and the classified images of left fist / right fist received from the processing device for assisting neuro-rehabilitation of stroke patients.

Description

FIELD OF THE INVENTION:The present invention generally relates to a non-invasive EEG-based brain-computerinterface system. More particularly, the present invention relates to EEG motorimagery classification system for neuro-rehabilitation of stroke patients.BACKGROUND OF THE INVENTION:Brain computer interfaces (BCIs) function as a direct communication pathwaybetween a human brain and an external device. Furthermore, BCI systems can alsoprovide an important test-bed for the development of mathematical methods andmulti-channel signal processing to derive command signals from brain activities. Asit directly uses the electrical signatures of the brain's activity for responding toexternal stimuli, it is particularly useful for paralyzed people who suffer from severeneuromuscular disorders and are hence unable to communicate through the normalneuromuscular, pathway. The electroencephalogram (EEG) is one of the widely usedtechniques out of many existing brain signal measuring techniques due to itsadvantages such as its non-invasive nature and its low cost.In addition to rehabilitation, BCI applications include, but are not limited to,communication, control, biofeedback and interactive computer gaming andentertainment computing.For example, currently, stroke rehabilitation commonly involves physical therapy byhuman therapists. Alternatively, robotic rehabilitation may augment human therapistsand enable novel rehabilitation exercises which may not be available from humantherapists.There are reports available in the state of art revealing the existence ofdevices / systems for rehabilitation of patients.WO2008101205A2 discloses an 8+2 degrees of freedom (DOF) intelligentrehabilitation robot capable of controlling the shoulder, elbow, wrist and fingersindividually and allowing functional arm movements with accompanying trunk andscapular motions. The rehabilitation robot uses the following integrated rehabilitationapproach: 1) it has unique diagnostic capabilities to determine patient-specificmultiple joint and / or multiple DOF biomechanical and neuromuscular changes; 2) itstretches the stiff joints / DOFs under intelligent control to loosen up the specific stiffjoints and to reduce excessive cross-coupling torques / movements between thespecific joints / DOFs, which can be done based on the above diagnosis for subjectspecifictreatment; 3) the patients practice voluntary reaching and some functionaltasks to regain / improve their motor control capability, which can be done after thestretching loosened up the stiff joints; and 4) the outcome will be evaluatedquantitatively at the levels of individual joints, multiple joints / DOFs, and the wholearm.WO2010105773A1 discloses a robot motor rehabilitation device, especially suitablefor being used in motor rehabilitation activities on patients suffering fromneurological and / or orthopaedic damages of a various nature such as outcomes ofictus, head traumas, vertebral column injuries, as well as motor rehabilitation onpatients with a hip bone or knee prosthesis, comprising a portal structure fixed to abase structure, a motor-driven belt suitable for allowing the walk of the patient, abodice basically provided with four straps or braces that can be hooked to the devicestructure and worn by the patient for the support thereof, a first exoskeleton and asecond exoskeleton articulated and independent for the passive and assistedmovement of the lower limbs of the patient, the exoskeletons comprising automaticactuation and movement means, with said rehabilitation device that further comprisesmeans for lifting the patient and for the automatic and continuous control of theamount of weight of the same to be used in the therapy.WO2005074373A2 discloses a method of rehabilitation using an actuator type thatincludes a movement mechanism capable of applying a force that interacts with amotion of a patient's limb in a volume of at least 30 cm in diameter, in at least threedegrees of freedom of motion of the actuator and capable of preventing substantialmotion in any point in any direction in the volume, including exercising a patient at afirst place of rehabilitation selected from a bed, a wheel-chair, a clinic and a home,using an actuator of said actuator type which interacts with a motion of the patient ata second place of rehabilitation selected from a bed, a wheel-chair, a clinic and ahome using a second actuator of the actuator type which interacts with a motion ofthe patient; wherein the first exercising and the second exercising utilize a samemovement mechanism design for moving the actuator.US20130138011A1 discloses a system and method for brain-computer interface(BCI) based interaction. The method comprising the steps of: acquiring a person'sEEG signal; processing the EEG signal to determine a motor imagery of the person;detecting a movement of the person using a detection device; and providing feedbackto the person based on the motor imagery, the movement, or both; wherein providingthe feedback comprises activating a stimulation element of the detection device forproviding a stimulus to the person. The system comprising: means for acquiring aperson's EEG signal; means for processing the EEG signal to determine a motorimagery of the person; means for detecting a movement of the person using adetection device; and means for providing feedback to the person based on the motorimagery, the movement, or both; wherein the means for providing the feedbackcomprises a stimulation element of the detection device for providing a stimulus tothe person.The above disclosed prior arts mainly focus on either robotic rehabilitation orphysical training and only few discloses a system and method for brain-computerinterface (BCI) based interaction. Typically, robotic rehabilitation includesrehabilitation based solely on movement repetition. In other words, a robot assists thepatient even if the patient is not attentive towards therapy and robot assistance istriggered if no movement detected, for example, after a period of 2 seconds.Moreover, robots deliver standardized rehabilitation, unlike human therapists whocan deliver individualized rehabilitation based on the condition and progress of thestroke patient. Further, the use of a robot may not be suitable for home-basedrehabilitation where there are cost and space concerns. In addition, the main form offeedback to the patient is visual feedback provided through a screen, which may beinsufficient. Furthermore the efficiency and accuracy of the disclosed system basedBCI is not adequate.Thus there exists a need in the state of art for an alternative system for neuro20rehabilitation of stroke patients that had enhanced efficiency and accuracy.Hence, an attempt has been made to develop an EEG motor imagery classificationsystem for neuro-rehabilitation of stroke patients overcoming the above saiddrawbacks.OBJECT OF THE INVENTION:The main object of the present invention is to develop an efficient and accuratesystem for assisting patients undergoing neuro-rehabilitation.Another object of the present invention is to develop an EEG motor imageryclassification system for neuro-rehabilitation of stroke patients.Yet another object of the present invention is to develop EEG motor imageryclassification system comprises of an EEG device coupled to a processing device andan output device.Further object of the present invention is to utilize the developed system in neurorehabilitationof stroke patients.BRIEF DESCRIPTION OF THE DRAWINGS:Figure 1 represent a flow diagram of a method performed by the pre – trainedmodelto classify the images.Figure 2 represent visual representation of flow diagram.SUMMARY OF THE INVENTION:The present invention discloses an EEG motor imagery classification system forneuro-rehabilitation of stroke patients. The system of the present invention comprisesof an EEG device coupled to a processing device and an output device, characterizedin that- the EEG device comprises of 64-electrode adapted to be placed on scalp andconfigured to acquire electroencephalogram (EEG) signals;- the processing device adapted to receive EEG signals ando inbuilt with Short Term Fourier Transform (STFT) configuredto extract spectrogram images;o inbuilt with Inception v3 network model adapted configured toclassify left fist and right fist from the spectrogram images;- the output device adapted to display the spectrogram images and the classifiedimages of left fist / right fist received from the processing device for assistingneuro-rehabilitation of stroke patients.DETAILED DESCRIPTION OF THE DRAWINGS:The present invention discloses an EEG motor imagery classification system forneuro-rehabilitation of stroke patients.The system of the present invention comprises of an EEG device coupled to aprocessing device and an output device.Characterization in the system of the present invention:The EEG device comprises of 64-electrode to place on scalp and to acquireelectroencephalogram (EEG) signals. The processing device receives EEG signalsand inbuilt with Short Term Fourier Transform (STFT) to extract spectrogram imagesand inbuilt with Inception v3 network model adapted to classify left fist and right fistfrom the spectrogram images. The output device displays the spectrogram images andthe classified images of left fist / right fist received from the processing device forassisting neuro-rehabilitation of stroke patients.The spectrogram images obtained by dividing the each acquired EEG signal intooverlapping segments followed by applying Fourier transform to each of the segmentand plotting magnitude of resulting complex coefficients as function of time andfrequency to obtain 2D representation of the signal frequency content over timespectrogram image.The Inception v3 network model is pre-trained employing ImageNet data set andrefinedbydeleting a last-layer of the pre-training model, addition of global averagepooling a layer, addition of first fully connected layer next to the global averagepooling layer, addition of second fully connected layer next to the first fullyconnected layerfollowed by transformation of activation function of the saidInception v3 network to a Tanh-ReLU function and finally tuning to obtain Inceptionv3 network for image classification.The system is further subjected to experimental analysis to ascertain its efficiency onclassification.The EEG signal data information is obtained from PhysioNet datasetcollected fromBCI competition. Each signal consist of 64 channel readings from: Left fist and Rightfist. Each trial lasted for 4 seconds, which was followed by a short duration of noactivity. On average 150 EEG trials have been obtained from each user with roughlyequal distribution of the left, right, or rest labels. The spectrogram images wereobtained from the EEG signal.The images are fed into pre-trained inceptionV3network model for training and testing. 56832 images for each class is given as input.A network of data sets, the data containing training information and a testinformation. The activation function of the network is Tanh-ReLU function; thenetwork later the activation function is enhanced is tuned to the training information,and refinement training is accomplished to attain sufficient modification; the testinformation is input to the tuned trained system to classify the right and left fistsignals.The pre-training of the network involves: deleting a last-layer of the pre-trainingmodel, and addition of global average pooling a layer, and addition of first fullyconnected layer next to the global averaged pooling layer, and addition of secondfully connected layer next to the first fully connected layer, so obtaining the network;where the first fully connected layer encloses 1024 nodes, Relu is the activationfunction, processed by Dropout, the probability is 0.5; the Softmax is the activationfunction of the second fully connected layer, and the output node is 2 classes. Theinception V3 model has 316 layers which takes the obtained images as input through11 inception module (three Inception module groups).Table 1: Description of Inception V3 Model with layers and input sizeType Patch / Stride Size Input SizeConv 3x3 / 2 299x299x3Conv 3x3 / 1 149x149x32Conv padded 3x3 / 1 147x174x32Pool 3x3 / 2 147x174x64Conv 3x3 / 1 73x73x64Conv 3x3 / 2 71x71x80Conv 3x3 / 1 35x35x1923xInception Module 1 35x35x2885xInception Module 2 17x17x7682xInception Module 3 8x8x1280Pool 8x8 8x8x2048Linear Logits 1x1x2048Softmax Classifier 1x1x1000The first Inception module group contains three similar Inception modules. Thefirst Inception module has 4 branches, the first branch has a 1×1 convolution of 64output channels, and the second branch has 48 output channels. The 1×1 convolutionis connected with a 5×5 convolution of 64 output channels; the third branch has a 1×1convolution of 64 output channels, and then continuously connects two 96-channel3×3 convolutions. The branches are 3 × 3 average pooled, connected with a 1×1convolution of 32 output channels. The last four branches are merged on the outputchannel to generate the final output of the Inception module.The only difference between the second Inception module and thefirst Inception module is that the fourth branch is connected to a 1×1 convolution ofthe 64 output channels. The final output third Inception module is the same as thesecond one.The second Inception module group contains five similar Inception modules. Thefirst Inception module has three branches. The first branch has a 3×3 convolution of384 output channels. The second branch has three layers. Is a 1×1 convolution of 64output channels and 3×3 convolution of two 96 output channels; the third branch is3×3 maximum pooling; the last 3 branches are merged on the output channel togenerate this Inception The final output of the module. The second Inception modulehas 4 branches, which combines 4 branches and outputs. The three Inception modulesare similar, which can enrich convolution and non-linearization, and refine thefeatures.The third Inception module group contains three similar Inception modules. Thefirst Inception module has three branches. The first branch has a 1×1 convolution of192 output channels and a 3× connection with 320 output channels. 3 convolution;the second branch has 4 convolutional layers, 1 × 1 convolution of 192 outputchannels, 1 × 7 convolution of 192 output channels, 7 × 1 convolution of 192 outputchannels, and 192 outputs 3 × 3 convolution of the channel; the third branch is thelargest pool of 3 × 3; the last 3 branches are merged on the output channel to generatethe final output of the Inception module. The second Inception module has 4branches, The first branch is a 1×1 convolution of 320 channels; the second branchand the third branch are both. The fourth branch is in a 3×3 average pooling layerfollowed by a 192 channel 1×1 Convolution; the last 4 branch output channels aremerged to get. The third Inception module is identical to thesecond Inception module.The steps based final layers for classification:1. Module Cluster with convolutional layers extract features from the input image.2. Feature Map: The output of the module cluster is a feature map, which is a tensorcontaining the extracted features from the input image. Each element of the featuremap represents a specific feature or pattern detected in the image.3. Pooling Layer: The feature map is passed through a pooling layer, which reducesthe spatial dimensions of the feature map. Pooling helps in down sampling thefeatures and capturing the most important information while reducing computationalcomplexity.4. Higher-Order Abstract Features: The pooling layer transforms the spatialinformation from the feature map into higher-order abstract feature information. Byreducing the spatial dimensions, the pooling layer emphasizes more on the dominantfeatures in the feature map.5. Global Average Pooling: After the pooling layer, a global average poolingoperation is applied to the feature map. This operation calculates the average value ofeach feature map channel, resulting in a fixed-length feature vector. It aggregates thehigh-level abstract features from the entire feature map.6. Linear 1 × 1 × 2048 Layer: A linear layer (also known as a fully connected layer)with a size of 1 × 1 × 2048 is applied to the output of the global average pooling. Thislayer performs a linear transformation on the features, mapping them to a newrepresentation.7. Fully Connected Layer: The output of the linear layer is then used as the input to afully connected layer. The fully connected layer connects every neuron from theprevious layer to each neuron in this layer, allowing for complex nonlineartransformations.8. Feature Constraints: Feature constraints regularize constraints imposed on thefeatures to enhance their properties.9. Softmax Layer: The output of the fully connected layer is passed through asoftmax layer. The softmax function converts the final layer activations intoprobability scores, indicating the likelihood of each class. This allows forclassification prediction.Pre-trained model network:The basic InceptionV3 network is trained on massive information to attain a pre20trained network; The input size of the picture is 299*299*3 in the network.Inception-V3 network trained by Google - ImageNet data set. The Inception V3 model has 96convolutional layers, and number of inception modules. Refining the pre-trainingmodel to attain an enhanced network apt for signal categorization, the data baseconsists of training and testing information. Transforming the activation function ofthe transferred model to a Tanh-ReLU function centered on the alteration of the Tanhand ReLU functions, and attaining a network model later the activation function isenhanced. Tuning the pre-trained network after the activation function is improved tothe training data base, performing tuning training, and obtaining a network aftertuning training. The test data fed into the tuned trained network for imageclassification.Refining the pre-training network to attain enhanced network appropriate for imageclassification information includes removal of the past layer of the network, adding alayer of the global average pooling layer, adding the first all-connected layer after theglobal average pooling layer, and adding the first all-connected layer two fullyconnected layers to obtain the improved network.The transferring of the network to the training information for training for whichAdam optimizer training parameters are used. In training process, when the gradientfalls, each batch contains 32 samples, and the number of iterations is set to 50. Thetotal parameters used 21,950,242 out of which 147,458 are trainable parameters. Thelearning rate is set to 0.001, the momentum parameter is set to 0.9, and the lossfunction uses the categorical- cross entropy loss function.Judging the proportion of samples correctly classified by the model through theconfusion matrix;formulaWhere,Tp is true positive, Fp is false positive, Fn is false negative, Tn is truenegative,Deploying the trained model to the Jetson Nano developmentKit. Feeding the test image into the model, and classify and predictwhether the fedimage belongs to left fist or right fist.The accuracy obtained from the system of thepresent invention was found to be88.06%.Thus, from the above it can be concluded that the system of present invention utilizeCNN design for image classification, incorporating convolutional layers, pooling,global average pooling, fully connected layers, and a softmax layer for classificationwhich subsequently results in high accuracy. The accuracy obtained from the systemof the present invention was found to be88.06%. Hence, it would be a promisingalternative to classify the images either left or right fist for assisting neurorehabilitationof stroke patients.In one of the preferred embodiments, the present invention shall discloses an EEGmotor imagery classification system for neuro-rehabilitation of stroke patients. Thesystem of the present invention comprises of an EEG device coupled to a processingdevice and an output device, characterized in that- the EEG device comprises of 64-electrode adapted to be placed on scalp andconfigured to acquire electroencephalogram (EEG) signals;- the processing device adapted to receive EEG signals ando inbuilt with Short Term Fourier Transform (STFT) configuredto extract spectrogram images;o inbuilt with Inception v3 network model adapted configured toclassify left fist and right fist from the spectrogram images;- the output device adapted to display the spectrogram images and the classifiedimages of left fist / right fist received from the processing device for assistingneuro-rehabilitation.As per the invention, in the system of the present invention, the spectrogram imagesobtained by dividing the each acquired EEG signal into overlapping segmentsfollowed by applying Fourier transform to each of the segment and plottingmagnitude of resulting complex coefficients as function of time and frequency toobtain 2D representation of the signal frequency content over time spectrogramimage.In accordance with the invention, in the system of the present invention, the Inceptionv3 network model is pre-trained employing ImageNet data set and refined by deletinga last-layer of the pre-training model, addition of global average pooling a layer,addition of first fully connected layer next to the global average pooling layer,addition of second fully connected layer next to the first fully connected layerfollowed by transformation of activation function of the said Inception v3 network toa Tanh-ReLU function and finally tuning to obtain Inception v3 network for imageclassification.In accordance with the invention, in the system of the present invention, theprocessing device is Jetson Nano Developer Kit.In accordance with the invention, in the system of the present invention, the outputdevice is computer, laptop and the like.Although the invention has now been described in terms of certain preferredembodiments and exemplified with respect thereto, one skilled in art can readilyappreciate that various modifications, changes, omissions and substitutions may bemade without departing from the scope of the following claim.

Claims

1. An EEG motor imagery classification system for neuro-rehabilitation of stroke patients, comprises of an EEG device coupled to a processing device and an output device, characterized in that - the said EEG device comprises of 64-electrode adapted to be placed on scalp and configured to acquire electroencephalogram (EEG) signals; - the said processing device adapted to receive EEG signals and - inbuilt with Short Term Fourier Transform (STFT)and configured to extract spectrogram images; - inbuilt with Inception v3 network model and configured to classify left fist and right fist from the said spectrogram images; - the said output device adapted to display the said spectrogram images and the classified images of left fist / right fist received from the said processing device for assisting neuro-rehabilitation of stroke patients.

2. The EEG motor imagery classification system as claimed in claim 1 wherein the said spectrogram images obtained by dividing the each acquired EEG signal into overlapping segments followed by applying Fourier transform to each of the segment and plotting magnitude of resulting complex coefficients as function of time and frequency to obtain 2D representation of the signal frequency content over time spectrogram image.

3. The EEG motor imagery classification system as claimed in claim 1 wherein the said Inception v3 network model is pre-trained employing ImageNet data set and refinedby deleting a last-layer of the pre-training model, addition of global average pooling a layer, addition of first fully connected layer next to the global average pooling layer, addition of second fully connected layer next to the first fully connected layerfollowed by transformation of activation function of the said Inception v3 network to a Tanh-ReLU function and finally tuning to obtain Inception v3 network for image classification.

4. The EEG motor imagery classification system as claimed in claim 1 wherein the said processing device is Jetson Nano Developer Kit.

5. The EEG motor imagery classification system as claimed in claim 1 wherein the said output device is computer, laptop and the like.