Apparatus using lightweight End-to-End Neural Networks for Decoding of Motor Imagery Brain Signal

Abstract

An apparatus using a lightweight terminal neural network for decoding motion image (MI) brain signals is disclosed. The apparatus using a lightweight terminal neural network for decoding motion image (MI) brain signals comprises: an initial inception module that receives and learns raw EEG data from LiteEEG-ISENet; an intermediate inception block including five intermediate inception modules excluding the initial inception module and an SE Block, a residual module, and an summer; a block that compresses channel-side information using Global Average Pooling and converts multidimensional input data into a one-dimensional vector (Flatten); and a fully connected layer (FC layer) that classifies and outputs the flattened one-dimensional features into four types of motion images—left hand, right hand, both feet, and tongue. Variants of convolutional neural networks (CNNs) in motion image-based brain-computer interfaces (MI-BCI) have relatively excellent decoding performance. However, the availability of actual MI-BCI systems is limited by the increase in network size required for high decoding performance and the inefficient procedures of BCI systems. To address these issues, an end-to-end neural network called the Lightweight EEG Inception Squeeze-and-Excitation Network (LiteEEG-ISENet) was proposed.

Description

Technology Field

[0001] The present invention relates to a device using lightweight end-to-end neural networks for decoding motor imagery brain signals in the study of a motor imagery-based brain-computer interface (MI-BCI).

[0002] This study was conducted with funding from the Ministry of Science and ICT (No. 2021-0-00034, Analysis of Clustering Techniques for Fragmented Data for Time-Based Data) and support from the Institute of Information and Communications Technology Planning and Evaluation (IITP). Background Technology

[0003] Brainwave measurement technology is primarily used in the fields of medicine, brain science, and cognitive neuroscience, and EEG-based brain-computer interface (BCI) technology is applied to the brain's neural networks. Electroencephalography (EEG) measures electrical potential differences occurring on the surface of the brain using multiple EEG electrodes; brainwave analysis is then performed after amplification (analog amplifier), BPF filtering, ADC, and, in the time domain, FFT transformation or wavelet transformation.

[0004] Brainwaves are classified into delta-δ waves (0.5 to 3.99 Hz), theta-θ waves (4 to 7.99 Hz), alpha-α waves (8 to 12.99 Hz), beta-β waves (13 to 29.99 Hz), and gamma-γ waves (30 to 50 Hz) depending on the range of vibrating frequencies.

[0005] Classification of brainwave signals Frequency characteristic Delta-δ wave 0.5 ~ 3.99 Hz It has a frequency of 0.2–0.4 Hz and an amplitude of 20–200 V, and mainly occurs in the deep sleep state of normal individuals or in newborns. The delta wave state indicates the production of a large amount of growth hormone. Theta-θ wave 4 ~ 7.99 Hz They have a frequency of 4–8 Hz and an amplitude of 20–100 V, and appear in an emotionally stable state or before bedtime (before falling asleep). Theta waves occur at the boundary state between perception and dreams. alpha-α waves 8 ~ 12.99 Hz It has a frequency of 8–13 Hz and an amplitude of 20–60 V, appears in a relaxed state such as meditation, and helps relieve stress and improve concentration. Beta-β waves 13 ~ 29.99 Hz It has a frequency of 13–30 Hz and an amplitude of 2–20 V, and appears during daily activities such as states of tension or excitement. It helps improve athletic performance and is a brainwave that occurs when consciousness is awake. Gamma-γ waves 30~50 Hz It has a frequency of 30–50 Hz and an amplitude of 2–20 V, and mainly appears when excited

[0006] FIG. 1 is a block diagram of a conventional brainwave measuring device. The brainwave measuring device (EEG measuring device) comprises: a brainwave detection unit (10) for detecting brainwave data; a brainwave amplification unit (20) for amplifying brainwave data measured from the brainwave detection unit (10); an A / D conversion unit (30) for converting the amplified brainwave data from analog to digital; a brainwave processing unit (40) for obtaining relative output values ​​for each time period and frequency considering the subject's baseline state using the digitally converted brainwave data, determining whether noise is mixed therein, and stopping the output of brainwave data when noise is mixed therein; and a display unit (50) for displaying the results of the brainwave processing unit (40).

[0007] The brainwave detection unit (10) is an EEG electrode for measuring brainwave data, and this electrode is attached to the subject's scalp to detect brainwave data.

[0008] The brainwave amplification unit (20) amplifies the brainwave data measured from each EEG electrode by a certain amount from the brainwave detection unit (10). In addition, the brainwave amplification unit (20) filters the amplified brainwave data, for example, in the 60Hz band.

[0009] The brainwave processing unit (40) includes a Fourier transform unit (410) that separates brainwave data amplified from the brainwave amplification unit (20) into sinusoidal waves, a data storage unit (420) that stores the brainwaves of the subject's baseline state, and a control unit (430) that calculates relative output values ​​by frequency over time, taking into account the subject's baseline state, using the brainwave data converted by the Fourier transform unit (410) and the brainwave data of the subject's baseline state.

[0010] The Fourier transform unit (410) uses the Fast Fourier Transform.

[0011] The data storage unit (420) storing the baseline brainwaves stores, for example, brainwave data measured when the subject is in an eye-closed and relaxed state or during the interval before stimulus presentation.

[0012] Additionally, the control unit (430) includes a first operation unit (432) that calculates a frequency-specific output value after a Fourier transform, a second operation unit (434) that calculates a time-specific and frequency-specific output value by repeatedly performing the first operation unit (432) over time, a third operation unit (436) that calculates a time-specific and frequency-specific relative output considering the baseline state by dividing the output of the second operation unit (434) by the frequency-specific output of each subject's baseline state, and a comparison unit (438) that compares the output of the third operation unit (436) with a predetermined value.

[0013] The first operation unit (432) calculates the magnitude of the absolute value (a² + b²) of the real part and the imaginary part (a+bi), i.e., the output value, from the Fourier transform unit (410).

[0014] The second operation unit (434) repeats the first operation unit (432) over time to calculate output values ​​for each frequency at regular intervals. The time interval may be, for example, about 0.375 seconds, and the frequency interval is determined by the time interval through Fourier transform.

[0015] The third operation unit (436) divides the output values ​​for each specific time period and frequency obtained from the second operation unit (434) into the output values ​​for each frequency of the brainwave data in the baseline state of each subject. Accordingly, relative output values ​​for each specific time period and frequency considering the baseline state are obtained.

[0016] The comparison unit (438) compares the relative output value (output of the third calculation unit) for each time period and frequency, considering the baseline state of each subject, with a predetermined constant value. At this time, the predetermined constant value is approximately 3, and if it is less than or equal to this, it is determined that the measured brainwave is not mixed with noise. Here, the predetermined constant value is the target for comparison with the output value of the third calculation unit (436), and ultimately represents the output value when noise is mixed. Generally, when noise is mixed, the brainwave suddenly increases by 2-3 times, so in this embodiment, the predetermined constant value is set to 3, for example. Accordingly, if the output of the third calculation unit (436) is 3 or higher, it is considered that noise is mixed, and the brainwave data is not output.

[0017] The display unit (50) displays the magnitude of the relative output value by time and frequency by changing the color.

[0018] 1. Introduction

[0019] A motor image-based brain-computer interface (MI-BCI) is a system that communicates with and controls various surrounding devices by decoding brainwave (EEG) signals based on the user's implicit intent. In such MI-BCI systems, decoding user intent has been an open challenge because EEG signals possess statistical complexity and are easily distorted by artifacts, movements, and eye blinking [1]. Recently, numerous neural networks based on the original Convolutional Neural Network (CNN) have demonstrated record-breaking performance compared to existing methods [2-4]. Generally, the implementation of a BCI system consists of signal collection, preprocessing, feature extraction, classification, and the transmission of the output as a control command. However, selecting suitable methods for high feature extraction and classification, as well as classification accuracy in dynamic background environments, is a time-consuming and difficult task.

[0020] Recently, with the advent of deep learning, several end-to-end CNN-based neural networks with high performance and simple architectures have been introduced [5, 6]. Recently, Zhang et al. presented a CNN model for MI-BCI based on an inception network with efficient and accurate time-series decoding capacity called EEG-inception [7]. This network is designed to capture high-quality time-series features using deep blocks and multi-branch inception structures corresponding to multiple time windows. However, large-scale neural networks including EEG inception cannot be used in actual MI-BCI systems due to limited computing resources.

[0021] As prior art related to this, Patent Publication No. 10-2022-0051613 discloses "a robust Lp norm-based common spatial pattern configuration method for an electroencephalogram-based motion imagery brain-computer interface."

[0022] A robust Lp-norm-based common spatial pattern construction method for an electroencephalogram-based motion imagery brain-computer interface

[0023] (a) In an electroencephalogram (EEG)-based BCI study, measuring an electroencephalogram (EEG) by an EEG measuring device during motor imagery (MI) without physical movement of the subject; and

[0024] (b) For electroencephalogram (EEG) data measured during motor imagery (MI) occurring in specific frequency bands, namely the alpha band (8–13 Hz) and the beta band (13 Hz–30 Hz), the method includes a motion imagery EEG analysis step in which the Lp-norm-based CSP-Lp, which is robust against noise such as muscle movement compared to the existing CSP and CSP-L1, is applied to the existing Common Spatial Patterns (CSP), which are sensitive to noise such as muscle movement, in order to reduce performance degradation due to noise during accurate pattern analysis of the data.

[0025] To improve the sensitivity of the above Common Spatial Patterns (CSP) to noise We use CSP-Lp based on Lp-norm using parameter P of CSP-Lp of the range.

[0026] As prior art related to this, Patent No. 10-2192211 discloses "an efficient adversarial generative neural network using depth-separable convolution and channel attention for image transformation."

[0027] In the field of deep learning, image transformation is achieving significant results by utilizing artificial neural networks. The area of ​​image transformation research that is yielding the greatest achievements is research using Generative Adversarial Networks (GANs). Image transformation using GANs includes converting image styles into the form of images in a different domain (Image-to-Image Translation), generating new high-resolution images from existing ones (Super-resolution), and filling in damaged areas (Image inpainting) or restoring images (Image restoration). These applications are being used across various industries, such as improving autonomous driving datasets, restoring and enhancing medical images, and augmenting class-imbalanced datasets. However, GANs are composed of multiple conventional convolutional neural networks, which results in a heavier structure and higher computational load compared to other artificial neural networks. Consequently, the number of parameters representing the complexity of the deep learning model is extremely high.

[0028] Figure 2 is a configuration diagram of an image conversion device for an efficient generative adversarial network (GAN) using conventional depth-separable convolution and channel attention.

[0029] The image transformation method of an efficient adversarial generative neural network using depthwise separable convolution and channel attention according to Prior Art 2 includes a step of applying depthwise separable convolution to reduce the number of parameters, and a step of applying channel attention to compensate for information loss caused by applying depthwise separable convolution to balance the quality of the output image with computational cost.

[0030] It consists of two sessions: a depthwise separation section (210) (depthwise separable session) and a channel attention section (220) (channel attention session).

[0031] The depth-separable separation unit (210) applies depth-separable convolution to reduce the number of parameters.

[0032] The depth-wise separation unit (210) passes through the depth-wise separable convolution block, the instance normalization block, the ReLU block, the depth-wise separable convolution block, and the instance normalization block, and applies depth-wise convolution using the kernel size, width, height, input channels, and feature map of the depth convolution, and applies point convolution that is performed only on the channels without handling spatial features.

[0033] Information loss occurs when reducing the number of parameters by applying depth-separable convolution. Although this information loss can be overcome through residual connections, it is not mitigated during image-to-image transformations, resulting in the model's output not matching the image quality of the original model. Information loss occurs because the loss of features present in the overall structure is inevitable since depth-separable convolution is performed for each channel. Therefore, to strike the optimal balance between output image quality and computational cost, techniques to compensate for information loss had to be added. Consequently, channel attention is applied to retain as much necessary information as possible and to prevent information loss even in deep layers.

[0034] The channel attention unit (220) applies channel attention to compensate for information loss caused by applying depth-separable convolution through a GAP (Global Average Pooling) block, a Convolution block, a ReLU block, a Convolution block, and a Sigmoid block, in order to balance the quality of the output image and the computational cost.

[0035] The channel attention unit (220) utilizes the interdependence between channels to focus on necessary input feature points, compresses channel information using Global Average Pooling (GAP), and restores input feature points through a convolution layer. Then, to strengthen and restore input feature points for parts to be focused on within the network, it extracts channel statistics through a gating mechanism.

[0036] In Brain-Computer Interface (BCI) technology, two measurement methods are used depending on the location of brain signal measurement. These include the invasive method, which extracts brain signals by directly attaching electrodes to the brain through surgical procedures, and the non-invasive method, which extracts brain signals by attaching electrodes to the scalp. Electroencephalography (EEG) is the most representative measurement method among non-invasive techniques and refers to the flow of electrical current generated when brain neurons exchange information. Because EEG allows for the easy measurement of brain activity without separate surgery, it is widely used in neuroengineering applications such as BCI research; however, the attached electrodes are susceptible to noise generated by muscle movements.

[0037] In EEG-based BCI research, motor imagery (MI) appears in specific frequency bands, namely the alpha band (8–13 Hz) and the beta band (13–30 Hz). Motor imagery is a model that utilizes similar EEG patterns when a subject moves a part of their body (hands, arms, legs, etc.) and when they imagine the movement. In 2010, KJ Miller's research team analyzed the Power Spectral Density (PSD) of actual body movements and imagined movements, demonstrating that characteristic EEG patterns occur in the same Sensory Motor Cortex. Regarding the classification of these motor imagery patterns, Common Spatial Patterns (CSP), proposed by Koles et al., serve as a representative spatial filter for BCI and demonstrate excellent performance in analyzing the spatial patterns of motor imagery.

[0038] Recently, variations of convolutional neural networks (CNNs) in motor imagery-based brain-computer interfaces (MI BCIs) have been attracting attention for their relatively superior decoding performance.

[0039] However, the availability of actual MI-BCI systems is limited by the increase in network size for high decoding performance and inefficient procedures of BCI systems. Prior art literature

[0040] Patent Publication No. 10-2022-0051613 (Publication Date: April 26, 2022), "Robust Lp norm-based common spatial pattern construction method for EEG-based motion imagery brain-computer interface," Kwangwoon University Industry-Academic Cooperation Foundation Patent Registration No. 10-2192211 (Registration Date: December 10, 2020), "Efficient adversarial generative neural network using depth-separable convolution and channel attention for image transformation," Inha University Industry-Academic Cooperation Foundation

[0041] F. Lotte et al., "A review of classification algorithms for EEG-based brain computer interfaces: A 10 year update," J. Neural Eng., vol. 15, no. 3, Apr. 2018.H. K. Lee and Y.-S. Choi, "Application of continuous wavelet transform and convolutional neural network in decoding motor imagery brain Computer Interface," Entropy, vol. 21, no. 12, 2019.X. Xiao and Y. Fang, "Motor imagery EEG signal recognition using deep convolution neural network," Front Neurosci. vol. 15. 2021.B. Sun, X. Zhao, H. Zhang, R. Bai, and T. Li, "EEG motor imagery classification with sparse spectrotemporal decomposition and deep learning," IEEE Trans. Autom. Sci. Eng., vol. 18, no. 2, pp. 541?551, Apr. 2021.V. J. Lawhern et al., "EEGNet: A compact convolutional neural network for EEG-based brain computer interfaces,” J. Neural Eng., vol. 15, no. 5, Oct. 2018.J. Liu, F. Ye, and H.Xiong, "Multi-class motor imagery EEG classification method with high accuracy and low individual differences based on hybrid neural network,” J. Neural Eng., vol. 18, no. 4, Aug. 2021.C. Zhang, Y.-K. Kim, and A. Eskandarian, "EEG-inception: an accurate and robust end-to-end neural network for EEG-based motor imagery classification," J. Neural Eng., vol. 18, no. 4, Mar. 2021.A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017.H. Zhang, X. Zhao, Z. Wu, B. Sun, and T. Li, "Motor imagery recognition with automatic EEG channel selection and deep learning," J. Neural Eng., vol. 18, no. 1, Feb. 2021.J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proc. Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 7132?7141.Wu, A. Fan, A. Baevski, Y. Dauphin, and M.Auli, “Pay less attention with lightweight and dynamic convolutions,” in Proc. Int. Conf. Learn. Representations (ICLR), 2019. The problem to be solved

[0042] The objective of the present invention to solve the above problems is to propose an end-to-end neural network called a lightweight EEG-inception squeeze-and-excitation network (LiteEEG-ISENet) to solve these problems in MI-BCI research, and to provide a device using a lightweight end-to-end neural network for decoding motor imagery brain signals.

[0043] The present invention presents a lightweight end-to-end neural network while improving decoding performance, called a lightweight EEG-inception squeeze-and-excitation network (LiteEEG-ISENet). means of solving the problem

[0044] To achieve the objective of the present invention, an apparatus using a lightweight terminal neural network for decoding Motor Imagery (MI) brain signals comprises: an Initial Inception Module that receives and learns raw EEG data from a LiteEEG-ISENet on a computer; an Intermediate Inception Block comprising five Intermediate Inception Modules excluding the Initial Inception Module and an SE Block (Squeeze and Excitation Block), a residual module, and an aggregator; and a block that compresses channel-side information using Global Average Pooling and flattens multidimensional input data into a one-dimensional vector. and decoding motion image (MI) brain signals using a Lightweight EEG-Inception Squeeze-and-Excitation Network (LITEEEG-ISENet) which includes a fully connected layer (FC layer) that classifies and outputs the flattened 1-dimensional features into four types of motion images—left hand, right hand, both feet, and tongue—and in the structure of the Lightweight EEG-ISENet, the network comprises an initial Inception Module, five intermediate Inception Modules, an intermediate Inception Block comprising an SE Block (Squeeze and Excitation Block), a residual Module, and a summerizer, and has channels of input electroencephalogram data composed of 228 channels, and the SE Block (Squeeze and Excitation Block) is a depthwise convolution;It is characterized by applying Channel Attention of the SE Block proposed in SENet (Squeeze and Excitation Network) to compensate for the decrease in model performance caused by information loss due to the reduction in the number of parameters in terms of channels by applying DC).

[0045] In the structure of the above-described lightweight terminal neural network LiteEEG-ISENet, the initial inception module, in addition to the initial inception module, five intermediate inception modules and an intermediate inception block comprising the SE Block (Squeeze and Excitation Block), a residual module, and an summerator are provided.

[0046] It is equipped with channels for input electroencephalogram data consisting of 228 channels.

[0047] The above intermediate inception block includes the intermediate inception module and the SE block (Squeeze and Excitation Block).

[0048] The Inception module of the above-mentioned lightweight terminal neural network LiteEEG-ISENet is divided into two forms: the Initial Inception Module and the Intermediate Inception Module. It learns various time-series information of one-dimensional electroencephalogram (EEG) signals for the Motor Imagery task through convolution and max pooling operations corresponding to different ranges of time-series windows (0.1 sec, 0.3 sec, 0.5 sec, 0.7 sec, 0.9 sec). It efficiently learns not only time-series information but also spatial information by generating weights for each channel of the feature map through the use of depthwise convolution. Furthermore, the depthwise convolution operation is applied to reduce the number of model parameters.

[0049] In the next layer, the individual feature maps are combined at the channel level through concatenation, followed by batch normalization and the ReLU (Rectified Linear Unit) activation function.

[0050] The above SE Block (Squeeze and Excitation Block) applies Depthwise Convolution (DC) and applies Channel Attention, proposed in SENet (Squeeze and Excitation Network), to compensate for the decrease in model performance caused by information loss resulting from the reduction in the number of parameters in terms of channels.

[0051] The above SENet (Squeeze-and-excitation networks) includes a squeeze stage that compresses channel information through Global Average Pooling (GAP) to model correlations between channels, and reshapes the feature map by applying a gating mechanism to increase non-linearity in terms of channels within the model and assign weights to each channel.

[0052] The above Channel Attention applies Global Average Pooling (GAP), a Fully Connected Layer, a ReLU activation function, a Fully Connected Layer, and a Sigmoid activation function, and then identifies the information of each channel through input features and element-wise operations.

[0053] The above Channel Attention compresses information on the channel side by the above Global Average Pooling (GAP), and after converting multidimensional input data into a 1-dimensional vector by the above Fully Connected Layer (Flatten), classifies the converted 1-dimensional features into four types of motion images (left hand, right hand, both feet, tongue).

[0054] The above residual module consists of a 1D convolutional layer, a batch normalization layer, and a ReLU activation function to compensate for the degradation problem that occurs as the layers of the designed model deepen. Effects of the invention

[0055] The device using a lightweight terminal neural network for decoding Motor Imagery Brain Signals according to the present invention, in the MI-BCI study, 1) adopted depthwise convolution to reduce the computational complexity of the network and train unique features for each channel of the MI dataset, and 2) adaptively rebalanced the channel-wise feature response using a squeeze-and-excitation (SE) block in addition to the previous mechanism. Experimental results on widely used public datasets in the MI-BCI study demonstrated that the proposed method is superior to existing methods in terms of decoding performance and neural network memory efficiency. Brief explanation of the drawing

[0056] Figure 1 is a block diagram of a conventional brainwave measurement device. Figure 2 is a configuration diagram of an image conversion device for an efficient generative adversarial network (GAN) using conventional depth-separable convolution and channel attention. Figure 3a shows the overall visualization process of the LiteEEG-ISENet architecture according to the present invention. FIG. 3b is a structural diagram of the SE Block, and FIG. 23c is a structural diagram of the residual module. Figure 4 shows the overall architecture of the initial module and intermediate module proposed in LiteEEG-ISENet (where the input data depth of the initial and intermediate inception modules is 2, respectively). Specific details for implementing the invention

[0057] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings, including the structure and operation of the invention.

[0058] The present invention is not limited to the disclosed embodiments but can be implemented in various different forms by those skilled in the art. In the description of the present invention, detailed descriptions of related known technologies or known configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Additionally, the same reference numerals are assigned in different drawings when indicating the same configuration.

[0059] This study is not limited to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0060] Variants of convolutional neural networks (CNNs) in motor imagery-based brain-computer interfaces (MI-BCIs) provide relatively superior decoding performance.

[0061] However, the availability of actual MI-BCI systems is limited by the increase in network size for high decoding performance and the inefficient procedures of BCI systems. To address these issues, an end-to-end neural network called the Lightweight EEG Inception Squeeze-and-Excitation Network (LiteEEG-ISENet) was proposed. The architecture was constructed to address the following two parts.

[0062] 1) Depthwise convolution was adopted to reduce the computational complexity of the network and to train unique features for each channel of the MI dataset.

[0063] 2) In addition to the previous motivation, the channel-wise feature response was adaptively resized using a squeeze-and-excitation (SE) block. Experimental results on public datasets widely used in MI-BCI research demonstrated that the proposed method outperformed existing methods in terms of decoding performance and neural network memory efficiency.

[0064] This study proposes a lightweight end-to-end neural network called the lightweight EEG-inception squeeze-and-excitation network (LiteEEG-ISENet) while improving decoding performance. We focus on two problems.

[0065] (1) A memory-efficient neural network architecture in a resource-constrained environment, considering the problem of relatively low performance in lighting models,

[0066] (2) To solve the problem of the former (1), depthwise convolution was adopted, which performs operations on each channel of the input data and saves the calculations [8]. Also, from a neurophysiological perspective, some channels have a more significant impact on MI tasks [9].

[0067] In the latter problem (2), the attention mechanism of the SE (squeeze-and-excitation) block is utilized to further strengthen channel-wise dependency, which captures channel-wise features related to MI tasks and contributes to improving classification accuracy

[10] . The proposed method was evaluated using the BCI Competition IV dataset 2a, which is widely used in MI-BCI research. Experimental results demonstrated the superiority of LiteEEG-ISENet by showing high classification accuracy in most subjects despite having 10 times fewer parameters.

[0068] This section explains the proposed method and background. First, the depthwise convolution and SE block (Squeeze and Excitation block) used in this study are introduced. The overall structure of the proposed model, based on the previous method and summarized in Figure 3a, is described.

[0069] A. Depthwise Convolution

[0070] Depthwise Convolution is the most frequently used method to reduce the number of parameters in general convolution-based neural networks and handle costs.

[0071] Here, convolution operations are performed separately on each channel of the input data. Therefore, the number of model parameters is It can be reduced from dk, where k is the kernel width. The output of depthwise convolution is provided in “Mobilenets: Efficient convolutional neural networks for mobile vision applications”[8] and “Pay less attention with lightweight and dynamic convolutions”

[11] .

[0072] (1)

[0073] Here, X is input data (EEG data), w is weight, k is kernel width (j = 1...K), i is element, c is output resource, is the weight of component i (i=1,2,3..), and Wc is the output dimension weight, output size.

[0074] Figure 3a shows the overall visualization process of the LiteEEG-ISENet architecture according to the present invention. Figure 3b is a structural diagram of the SE Block, and Figure 3c is a structural diagram of the residual module.

[0075] The device using a lightweight terminal neural network for decoding motion image brain signals of the present invention

[0076] In a computer, in LiteEEG-ISENet (Lightweight EEG-Inception Squeeze-and-Excitation Network), an Initial Inception Module that receives raw EEG data as input and learns;

[0077] Among a total of 6 Inception Modules, the remaining 5 Intermediate Inception Modules excluding the Initial Inception Module, the Intermediate Inception Block including the SE Block (Squeeze and Excitation Block), the Residual Module, and the Summerizer;

[0078] A block that compresses channel-side information using Global Average Pooling and flattens multidimensional input data into a one-dimensional vector; and

[0079] The method is characterized by decoding motion image (MI) brain signals using a lightweight terminal neural network (LiteEEG-ISENet) that includes a fully connected layer (FC layer) that flattens multidimensional input data into a one-dimensional vector and classifies the flattened one-dimensional features into four types of motion images—left hand, right hand, both feet, and tongue—and applying the channel attention of the SE Block proposed in SENet (Squeeze and Excitation Network) to compensate for the decrease in model performance caused by information loss due to the reduction in the number of parameters in terms of channels by applying depthwise convolution (DC) to the SE Block (Squeeze and Excitation Block).

[0080] In the structure of the above-described lightweight terminal neural network LiteEEG-ISENet, the system includes the initial inception module, the intermediate inception module, an intermediate inception block comprising an SE Block (Squeeze and Excitation Block), a residual module, and an summerator.

[0081] In addition to the above initial inception module, it is equipped with five intermediate inception modules and an intermediate inception block having an SE Block, a residual module and a summer, and has channels of input electroencephalogram data consisting of 228 channels.

[0082] Raw EEG data is input as initial training data into the lightweight terminal network LiteEEG-ISENet.

[0083] Inception Module

[0084] The Inception module of the proposed lightweight longitudinal neural network LiteEEG-ISENet is divided into two forms: an Initial Inception Module and an Intermediate Inception Module. It learns various time-series information of electroencephalogram (EEG) signals for a Motor Imagery task through convolution and max pooling operations corresponding to different ranges of time-series windows (0.1 sec, 0.3 sec, 0.5 sec, 0.7 sec, 0.9 sec) for 1-dimensional EEG signals. A Bottleneck Layer prior to the convolution operation is applied to form the same channel dimension, and by utilizing Depthwise Convolution, weights are generated for each channel of the feature map to efficiently learn not only time-series information but also spatial information. In addition, it is applied to reduce the number of model parameters through depthwise convolution operations.

[0085] In the next layer, the individual feature maps are combined at the channel level through concatenation, and then Batch Normalization and the ReLU (Rectified Linear Unit) activation function are applied.

[0086] A time series is defined as a series of sequential data collected over a certain period of time. In this invention, time series information utilizes time series electroencephalogram (EEG) data extracted from a user performing motor imagery as input, and a technology for predicting and recognizing the user's intention from sequential EEG data has been invented. Therefore, the time series information refers to EEG data corresponding to a series of processes performed by the user over a certain period of time, such as rest, preparation for motor imagery, motor imagery, and rest.

[0087] The electroencephalogram data, which is the input data of the present invention, is composed of three dimensions (1 x E x T), where C represents an electrode and T represents the number of samples. Here, the spatial information refers to spatial information of the brain, and this was conceived based on the fact that the active areas of the brain differ depending on specific activities and behaviors (upper limb movement behavior, language activity, etc.).

[0088] Depthwise convolution performs independent convolution operations for each channel of the input feature map. It is a widely applied technique used to reduce the number of model parameters and was utilized in the Depthwise Separable Convolution applied in MobileNet. In this invention, it is used to efficiently learn spatial information of the brain while performing standard convolution operations in existing artificial neural network models.

[0089] * Initial Inception Module (see Fig. 3a): This refers to the module that receives raw EEG data from the proposed LiteEEG-ISENet and is the first module deployed among the six Inception Modules.

[0090] The structure and design purpose of the lightweight terminal neural network LiteEEG-ISENet are the same as the Inception Module mentioned above, and the difference from the Intermediate Inception Module is that the channel dimension of the input electroencephalogram data, which consists of 22 channels, has been expanded to 228 channels.

[0091] * Intermediate Inception Module: Represents the remaining modules excluding the Initial Inception Module among the total of 6 Inception Modules.

[0092] * Squeeze and Excitation Block (SE Block)

[0093] Figure 3b is a structural diagram of the SE Block.

[0094] To compensate for the performance degradation of the model caused by information loss resulting from the reduction in the number of parameters (computational cost) achieved by applying Depthwise Convolution (DC), Channel Attention from the SE Block proposed in SENet (Squeeze and Excitation Network) is applied to address this from a channel perspective.

[0095] Channel Attention applies Global Average Pooling (GAP), a fully connected layer, a ReLU activation function, a fully connected layer, and a sigmoid activation function, and then determines the relative importance of each channel through input features and element-wise operations.

[0096] Channel Attention offers significant model performance improvement relative to the increase in the number of parameters and has the advantage of being applicable to any structure because the input and output dimensions are the same.

[0097] Channel information is compressed using Global Average Pooling (GAP), and input feature points are restored through a convolution layer.

[0098] Residual Module

[0099] FIG. 3c is a diagram illustrating the configuration of the invented residual module.

[0100] The Residual Module was applied to solve the degradation vanishing problem that occurs as the layers of the designed model deepen, and

[0101] The above residual module is a one-dimensional convolutional layer that learns time-series information from input features; and for each layer of the model being trained, the output values ​​of the i-th convolutional layer, which are the normalization components ( It consists of a batch normalization layer that calculates the mean and variance of and applies batch normalization; and a ReLU activation function that receives the output of the batch normalization layer and applies it to the ReLU activation function to output only feature values ​​greater than 0.

[0102] Batch Normalization is For each layer of the model being trained, the regularization component (output values ​​of the convolution layer ( It refers to a technique that controls the modified distribution of input data by calculating the mean and variance of the input data.

[0103]

[0104] In the above formula represents the output of the i-th convolution layer, and and is the mean and variance of the output values ​​of the i-th convolution layer, and is the output with batch normalization applied.

[0105] The ReLU activation function is currently the most widely used activation function for hidden layers.

[0106] ReLU(x) = max (0, x)

[0107] ReLU(x) is 0 if x ≤ 0, and if input x > 0, that is, if the input is greater than 0, the input is output as is.

[0108] Global Average Pooling (GAP) is The input data is compressed into a single average value (1 x 1 x C) per channel to provide representativeness to the data. Here, C represents the number of channels.

[0109] Average Pooling

[0110] - Compresses channel-side information using Global Average Pooling (GAP).

[0111] * Flatten

[0112] - Converts multidimensional input data into a one-dimensional vector.

[0113] Fully Connected Layer (FC)

[0114] - Multidimensional input data is converted into a 1-dimensional vector (Flatten), and the converted 1-dimensional features are used to classify four types of movement images (left hand, right hand, both feet, tongue).

[0115] B. Squeeze-and-Excitation Network (SENet)

[0116] According to Squeeze-and-excitation networks

[10] , the Squeeze-and-excitation network (SENet) consists of SE blocks that adaptively recalibrate. By compressing features using global average pooling (GAP), which has shown strong performance on the ImageNet database, to obtain channel relationships, SENet consists of two stages: squeeze and excitation. In the first stage, all features along the channel axis are converted into one-dimensional vectors with condensed channel-wise representations. Then, multiple fully connected layers in the excitation step can obtain channel-wise attention from the output of the squeeze step. A more detailed explanation is provided in "Squeeze-and-excitation networks"

[10] .

[0117] Figure 4 shows the overall architecture of the initial module and intermediate module proposed in LiteEEG-ISENet (where the input data depth of the initial and intermediate inception modules is 2, respectively).

[0118] C. Lightweight EEG-Inception Squeeze-and-Excitation Network (LiteEEG-ISENet)

[0119] We propose a lighter and more robust end-to-end neural network task by introducing depthwise convolution and SE blocks. Like our initial network, EEG-inception [7], the LiteEEG-ISENet proposed in Fig. 2 consists of one inception module, five intermediate modules, five SE blocks, and five residual modules. Each inception module includes a bottleneck layer, multiple 1D convolutional layers, a 1D max pooling layer, a batch normalization, and a ReLU activation function. The overall approach uses the same structure as “EEG-inception: an accurate and robust end-to-end neural network for EEG-based motor imagery classification” [7]. The bottleneck layer with a [1 x 1] kernel matrix extends the dimension of the time-series EEG input data from 22 to 48. Subsequently, to reduce the computational complexity of the network, all standard 1D convolution layers after the bottleneck layer are 1D depth convolution layers, and the detailed architecture is illustrated in Fig. 2. The five depthwise convolution operations include kernel sizes of [25 x 1], [75 x 1], [125 x 1], [175 x 1], and [225 x 1], respectively. Each kernel size represents the size of a window along the time axis of 0.1 sec, 0.3 sec, 0.5 sec, 0.7 sec, and 0.9 sec.A pooling layer is applied to collect various features by downsampling for features. Different features extracted from multiple branches are concatenated, followed by batch normalization and the ReLU activation function.

[0120] Table 2 shows a comparison of three neural networks.

[0121]

[0122] Table 2 shows the subject-independent classification accuracy (%) by different networks for BCI Competition IV Dataset 2A.

[0123] Finally, the temporal dimensionality reduction of the output of the fifth intermediate block is performed by average pooling. Following average pooling is a fully connected layer with four outputs corresponding to four MI tasks.

[0124] III. Experimental Results

[0125] A. Dataset

[0126] To evaluate the proposed method, the BCI competition IV dataset 2a was used. Nine subjects participated in an experiment consisting of two sessions with 288 trials per session. Each trial had 22 EEG channels, and a sampling rate of 250 Hz was used. Four MI tasks (left hand, right hand, foot, and tongue) were performed. Here, to classify the four MI tasks, only MI durations between 3 and 6 seconds were used.

[0127] In the experiment, the evaluation dataset was split into 5 folds (4 for training and 1 for testing). To train the proposed method, the training dataset was subjected to 10-fold cross-validation.

[0128] B. Evaluation Performance

[0129] All experiments were conducted on Ubuntu 18.04.6 LTS, CPU (AMD Ryzen Threadripper 3960X 24-core processor), GPU (Nvidia Geforce RTX 3090), RAM (64GB), Python 3.8.12, and PtTorch 1.10.1.

[0130] Table 1 reports the results for the number of parameters of the proposed method with computational and input sizes for EEG-inception. Here, FLOP represents floating-point operations. For each result, the proposed LiteEEG-ISENet showed values ​​90.33% and 90.8% lower than EEG-inception. Notably, regarding the number of parameters and computations, the proposed LiteEEG-inception has lower decoding performance than other networks.

[0131] LiteEEG-ISENet is shown in Table 3.

[0132]

[0133] Table 2 summarizes the results of subject-independent classification accuracy. From the table, the superiority of the proposed LiteEEG-ISENet can be confirmed, as it provides high classification accuracy results for most subjects. In particular, it demonstrates improved classification accuracy for Subjects S5 and S7. Furthermore, in terms of average classification accuracy, LiteEEG-ISENet achieved a result 4.01% higher than the reference method.

[0134] IV. Conclusion

[0135] We presented LiteEEG-ISENet, which is motivated to capture channel-wise dependencies while reducing the size of the neural network. Experimental results demonstrated the potential of the proposed method by achieving improved classification accuracy across various topics on the public MI evaluation dataset. Furthermore, the proposed LiteEEG-ISENet reduces the computational cost of the EEG-inception base model by approximately 10 times and eliminates the need for cumbersome procedures such as preprocessing and feature extraction. Consequently, the proposed end-to-end neural network can provide a tool more suitable for actual MI-BCI systems.

[0136] Embodiments according to the present invention are implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. The computer-readable recording medium may include magnetic media such as storage, hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices configured to store and execute program instructions on storage media such as ROM, RAM, flash memory, storage, etc. Examples of program instructions may include machine code as well as high-level language code that can be executed by a computer using an interpreter. The hardware devices may be configured to operate as one or more software modules to perform the operation of the present invention.

[0137] As described above, the method of the present invention can be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) in a form that can be read using computer software.

[0138] Although the present invention has been described with reference to specific embodiments, the present invention is not limited to the same configuration and operation as the specific embodiments used to illustrate the technical concept as described above, and may be implemented with various modifications within the limits of the technical concept and scope of the present invention, and the scope of the present invention should be determined by the claims set forth below.

Claims

Claim 1 In a computer, in LiteEEG-ISENet, an Initial Inception Module that receives and learns raw EEG data; an Intermediate Inception Block including the remaining five Intermediate Inception Modules excluding the Initial Inception Module and an SE Block (Squeeze and Excitation Block), a Residual Module, and an Adder; and a Block that compresses channel-side information using Global Average Pooling and flattens multidimensional input data into a one-dimensional vector; and decoding motion image (MI) brain signals using a Lightweight EEG-Inception Squeeze-and-Excitation Network (LITEEEG-ISENet) which includes a fully connected layer (FC layer) that classifies and outputs the flattened 1-dimensional features into four types of motion images—left hand, right hand, both feet, and tongue—and in the structure of the Lightweight EEG-ISENet, the intermediate inception block comprising the initial inception module, five intermediate inception modules, and the SE Block (Squeeze and Excitation Block), the residual module, and the summerator, and the channels of the input electroencephalogram data composed of 228 channels, and the SE Block (Squeeze and Excitation Block) is depthwise convolution;A device using a lightweight terminal neural network for decoding motion image brain signals, which applies Channel Attention of the SE Block proposed in SENet (Squeeze and Excitation Network) to compensate for the phenomenon of model performance degradation caused by information loss due to the reduction in the number of parameters in the channel side by applying DC). Claim 2 delete Claim 3 delete Claim 4 In claim 1, the Inception module of the light terminal neural network LiteEEG-ISENet is divided into two forms: the Initial Inception Module and the Intermediate Inception Module. It learns various time-series information of one-dimensional electroencephalogram (EEG) signals for a Motor Imagery task through convolution and max pooling operations corresponding to different ranges of time-series windows (0.1 sec, 0.3 sec, 0.5 sec, 0.7 sec, 0.9 sec). It efficiently learns not only time-series information but also spatial information by generating weights for each channel of a feature map through depthwise convolution. The depthwise convolution operation is applied to reduce the number of model parameters. In the next layer, after combining each feature map at the channel level through concatenation, batch A device using a lightweight terminal neural network for decoding motion image brain signals using Batch Normalization and ReLU (Rectified Linear Unit) activation functions. Claim 5 delete Claim 6 A device using a lightweight terminal neural network for decoding motion image brain signals, wherein the SENet (Squeeze-and-excitation networks) comprises a squeeze step that compresses channel information through Global Average Pooling (GAP) to model the correlation between channels, and a step of readjusting a feature map by applying a gating mechanism to increase non-linearity in terms of channels within the model and assign weights to each channel. Claim 7 A device for decoding motion image brain signals using a lightweight terminal neural network, wherein the channel attention is applied to Global Average Pooling (GAP), the Fully Connected Layer, the ReLU activation function, the Fully Connected Layer, and the Sigmoid activation function, and then identifies information of each channel through input features and element-wise operations. Claim 8 A device using a lightweight terminal neural network for decoding motion image brain signals, wherein the channel attention compresses channel-side information by the global average pooling (GAP), and the fully connected layer flattens multidimensional input data into a one-dimensional vector, and classifies the flattened one-dimensional features into four types of motion images—left hand, right hand, both feet, tongue. Claim 9 In claim 1, the residual module comprises a one-dimensional convolutional layer that learns information according to a time series from input features; and output values ​​of the i-th convolutional layer, which is a normalization component for each layer of the model being learned ( A device using a lightweight terminal neural network for decoding motion image brain signals, comprising: a batch normalization layer that calculates the mean and variance of ) and applies batch normalization; and a ReLU activation function that receives the output of the batch normalization layer and applies it to the ReLU activation function to output only feature values ​​greater than 0.

Images