A method and system for motion imagery recognition based on multi-scale convolution and Transformer

Abstract

The present invention belongs to the field of electroencephalographic (EEG) signal technology, and specifically relates to a motor imagery recognition method and system based on multi-scale convolution and Transformer, as well as a decoding device for motor imagery EEG signals. The recognition method comprises the following steps: S1: bandpass filtering and z-score normalization of the original EEG signal. S2: extracting spatial features using a CSP spatial domain filter. S3: constructing a spatial self-attention mechanism and implementing feature channel weighting. S4: processing the weighted data using channel convolution and multi-scale convolution. S5: using an average pooling layer to divide the fused features into multiple feature slices. S6: constructing a multi-head attention mechanism to extract global features. S7: performing global average pooling on the output of the self-attention module to obtain refined features. S8: performing layer normalization on the output and connecting it to a fully connected layer, and obtaining predicted probabilities for different categories using a Softmax function. The present invention solves the problems of low accuracy and insufficient real-time performance in existing motor imagery EEG signal decoding.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram (EEG) signals, and in particular relates to a motor imagery recognition method and system based on multi-scale convolution and Transformer, as well as a decoding device for motor imagery EEG signals. Background Art

[0002] Motor imagery (MI) refers to the neural activity generated when the brain imagines movements of parts of the body, such as the limbs or tongue. By decoding the motor imagery electroencephalogram (MI-EEG), information related to the movement of the corresponding parts can be analyzed. Brain-computer interface (BCI) technology based on MI-EEG has a wide range of applications in sports rehabilitation, smart device control, and new smart games. However, the low signal-to-noise ratio and low spatial resolution of MI-EEG make it difficult to accurately analyze and classify motor imagery.

[0003] In recent years, with the introduction of technologies such as artificial intelligence and machine learning, motor imagery decoding has made great progress. However, commonly used machine learning methods such as linear discriminant analysis and support vector machines are still very limited in the feature information they can mine. With the development of deep learning, convolutional neural networks have been applied to a large number of offline MI-EEG studies, significantly improving decoding performance. However, a single convolution scale may not be suitable for effectively capturing the distinguishable features of various non-overlapping typical frequency bands of EEG signals, nor is it sufficient to effectively perceive the global dependencies of EEG signals. At the same time, complex network structures and deep computations often place high demands on the quantity and quality of training samples. The MI-EEG acquisition process is easily affected by the experimental environment and the subject's own state, and the number of high-quality training samples is usually very limited.

[0004] Representative deep-level motion-related feature components can improve detection performance, and designing an effective method for directly extracting deep-level features from raw EEG signals is crucial. Therefore, those skilled in the art urgently need to provide a novel motor imagery classification and detection method to address the aforementioned issues, overcoming the drawbacks of existing solutions, such as low accuracy, insufficient real-time performance, and high requirements for the quantity and quality of training samples. Summary of the Invention

[0005] In order to solve the problems of low accuracy and insufficient real-time performance of existing motor imagery EEG signal decoding, the present invention provides a motor imagery recognition method and system based on multi-scale convolution and Transformer, as well as a motor imagery EEG signal decoding device.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] A motor imagery recognition method based on multi-scale convolution and Transformer is used to online identify the corresponding motor imagery category based on input EEG signals. The motor imagery recognition method includes the following steps:

[0008] S1: Obtain the original EEG signal to be identified, perform bandpass filtering and z-score normalization on it, and obtain the preprocessed EEG signal.

[0009] S2: Extract the spatial features of the preprocessed EEG signal through a feature extraction module composed of multiple CSP spatial filters.

[0010] S3: Use the channel attention module in Transformer to build a spatial self-attention mechanism, perform feature analysis on the feature map in the previous step, and implement feature channel weighting.

[0011] S4: Use multiple convolution modules to perform convolution processing on the weighted two-dimensional data. The convolution processing process is as follows:

[0012] S41: Perform inter-channel convolution on the feature map output by the channel attention module. The convolution kernel size of the inter-channel convolution is (Channels, 1); where Channels represents the number of channels of the original EEG signal, the convolution stride is 1, and the data after convolution will be compressed from two-dimensional data to one-dimensional data.

[0013] S42: A multi-scale convolution module with three convolution kernels of different scales is used to perform one-dimensional convolution on the input data in the time domain; the convolution kernel shapes of the multi-scale convolution module are (1, 3), (1, 5), and (1, 7), respectively.

[0014] S43: Horizontally concatenate the local features captured by convolution kernels of different scales to obtain fused features.

[0015] S5: Use the average pooling layer to divide the one-dimensional fusion feature into multiple feature slices according to the preset data length.

[0016] S6: Use the self-attention module in Transformer to build a multi-head attention mechanism, perform feature analysis on the feature slices in the previous step, and extract the global features contained therein.

[0017] S7: Use a global average pooling module to process the output of the self-attention module to obtain refined features after global average pooling.

[0018] S8: The output of the global average pooling module is normalized and then connected to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function, and the category of the EEG signal is generated and identified.

[0019] As a further improvement of the present invention, in step S1, the preprocessing process of the EEG signal is:

[0020] S11: Based on the characteristic frequency band of the movement-related components mu / beta rhythm, a wide-bandpass time-domain filter of 8-30 Hz is designed.

[0021] S12: Use a wide-bandpass time-domain filter to perform noise reduction on the original EEG signal to remove low-frequency and high-frequency noise in the original EEG signal.

[0022] S13: Use the following z-score normalization function to process the denoised EEG signal to alleviate signal fluctuations and the non-stationary characteristics of the EEG signal:

[0023]

[0024] Where x is the input signal; μ is the mean value of the input signal; σ is the variance of the input signal; the output signal X obtained after normalization conforms to the normal distribution with a mean of 0 and a standard deviation of 1.

[0025] As a further improvement of the present invention, any CSP spatial domain filter is used to realize the spatial domain feature extraction of a binary classification target, and its expression is as follows:

[0026]

[0027] In the above formula, X i represents the preprocessed EEG matrix of the target of the i-th category, C i represents the spatial covariance matrix corresponding to the target of the i-th category; X represents the input preprocessed EEG signal, Z is the result obtained after CSP spatial domain filtering; W represents the corresponding spatial domain coefficient matrix.

[0028] When multi-classification is required in the recognition task, multiple CSP spatial domain filters are cascaded to form the required feature extraction module.

[0029] As a further improvement of the present invention, the expression of the channel attention module in step S3 is as follows:

[0030]

[0031] Among them, Attention(Q,K,V) is the weighted feature representation; Q, K, V are the corresponding feature vectors after the original data is mapped to the three vector spaces through linear transformation, d k Indicates the scaling factor.

[0032] As a further improvement of the present invention, in step S5, the data of the feature slices after the average pooling layer is segmented is expressed as (L, V), wherein the pooling layer shape is set to (1,75), the stride is (1,15), L represents the length of the data after segmentation, and V represents the number of segmented segments; the length of the data after segmentation L is adaptively adjusted according to the number of time domain sampling points of the original EEG signal.

[0033] As a further improvement of the present invention, the expression of the multi-head attention mechanism constructed in step S6 is as follows:

[0034]

[0035] In the above formula, head i represents the feature vector of the i-th feature slice after being processed by the self-attention module; MHA represents the multi-head attention mechanism; [head0; ...; head h-1 ] represents the feature vector head0~head h-1 Perform the connection operation; Q i , K i 、V i They represent the three feature vectors obtained after linear transformation of i feature slices.

[0036] The present invention also includes a motor imagery recognition system based on multi-scale convolution and Transformer, which is used to create a network model for analyzing and identifying EEG signals. The network model uses the aforementioned multi-scale convolution and Transformer-based motor imagery recognition method to output a prediction result of the corresponding motor imagery category based on the input raw EEG signal. The motor imagery classification detection system includes a preprocessing module, a feature extraction module, a channel attention module, a convolution module, a feature segmentation module, a multi-head attention module, a feature refinement module, and a classification module.

[0037] The preprocessing module includes a noise reduction unit and a normalization unit. The noise reduction unit uses an 8-30Hz wideband-pass time-domain filter to reduce the noise of the original EEG signal, removing low-frequency and high-frequency noise. The normalization unit uses the z-score normalization function to process the noise-reduced EEG signal to reduce signal fluctuations and non-stationary characteristics of the EEG signal.

[0038] The feature extraction module uses multiple cascaded CSP spatial domain filters to extract features from the preprocessed EEG signals and obtain the spatial domain features contained therein.

[0039] The channel attention module is used to perform feature analysis and weighting on the spatial features of the input EEG signal to obtain weighted two-dimensional features.

[0040] The convolution module includes a channel convolution unit, a multi-scale convolution unit, and a feature splicing unit. The channel convolution unit is used to perform inter-channel convolution operations on the output of the channel attention model; the multi-scale convolution unit is used to perform synchronous convolution processing on the output of the channel convolution unit using three convolution kernels of different scales; and the feature splicing unit is used to horizontally splice the output of the multi-scale convolution unit to obtain the corresponding fusion features.

[0041] The feature segmentation module is used to divide the one-dimensional fusion features output by the convolution module into multiple feature slices according to the preset data length using the average pooling layer.

[0042] The multi-head attention module is used to perform feature analysis on the feature slices using a multi-head attention mechanism to extract the global features contained therein.

[0043] The feature refinement module is used to process the output of the self-attention module using the global average pooling module to obtain refined features after global average pooling.

[0044] The classification module is used to normalize the output of the global average pooling module and then connect it to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function to generate the category of the recognized EEG signal.

[0045] As a further improvement of the present invention, the design process of the network model includes the following steps:

[0046] S01: Design an 8-30 Hz wide-bandpass time domain filter for denoising the original EEG signal, and a normalization filter using the z-score normalization function.

[0047] S02: Design a corresponding feature extraction module according to the number of classification results. The feature extraction module is composed of a cascade of multiple CSP spatial domain filters.

[0048] S03: Design a convolution module that includes a channel convolution unit, a multi-scale convolution unit, and a feature splicing unit; the convolution module is used to generate one-dimensional fusion features from the input two-dimensional features.

[0049] S04: Use a wideband-pass time-domain filter and a normalization filter to form the required pre-processing module.

[0050] S05: Build a network model that includes a preprocessing module, a feature extraction module, a channel attention module, a convolution module, a feature segmentation module, a multi-head attention module, a feature refinement module, and a classification module.

[0051] S06: Obtain a large number of EEG signal samples and add category labels to the samples to form the required data set, and divide the data set into a training set and a test set.

[0052] S07: The network model is trained using the training set and the test set until the preset training target is met and the model parameters are retained to obtain the required network model for analyzing and identifying EEG signals.

[0053] As a further improvement of the present invention, in step S07, the prediction accuracy is used as the training target, and the loss function for evaluating the training target adopts the following cross entropy loss:

[0054]

[0055] In the above formula, M is the number of EEG samples, N is the number of categories, represents the true label of the mth sample, It represents the probability that the mth sample is predicted to be the nth category.

[0056] The present invention also includes a decoding device for motor imagery EEG signals, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it creates a network model for analyzing and identifying EEG signals; the network model adopts the aforementioned motor imagery recognition method based on multi-scale convolution and Transformer, and outputs the prediction result of the corresponding motor imagery category according to the input original EEG signal.

[0057] The technical solution provided by the present invention has the following beneficial effects:

[0058] The present invention introduces a transformer into motor imagery EEG waves and, combined with data processing modules such as preprocessing and multi-scale convolution, converts multi-channel EEG signals into one-dimensional feature data suitable for transformer processing. This provides a more lightweight EEG signal recognition method, enabling local deployment and application of the system. By combining deep learning with traditional signal processing, the present invention extracts feature information from motor imagery EEG signals in multiple dimensions, both in the time and spatial domains, thereby improving EEG signal recognition accuracy.

[0059] This paper utilizes a CSP spatial domain filter to construct a feature extraction module, deeply extracting spatial features from EEG signals. It also incorporates a self-attention mechanism to weight the signals in motion-related channels within the EEG signal, reducing the influence of interference from irrelevant channels. To capture distinguishable features across various non-overlapping frequency bands typical of motor imagery, the paper also employs multi-scale convolutional blocks to extract deep temporal information.

[0060] The solution provided by the present invention can also utilize the multi-head attention mechanism to fully consider the highly context-related characteristics of EEG signals, and adopt a lightweight Transformer structure to perceive the global dependency of motor imagery EEG, thereby significantly improving the recognition accuracy of motor imagery categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of the steps of a motion imagery recognition method based on multi-scale convolution and Transformer provided in Example 1 of the present invention.

[0062] Figure 2 This is an architectural diagram of the motion imagery recognition system based on multi-scale convolution and Transformer constructed in Example 2 of the present invention.

[0063] Figure 3 This is an architecture diagram of the multi-scale convolution unit and feature splicing unit in the convolution module in Example 2 of the present invention.

[0064] Figure 4 This is a flowchart of the steps of the network model design method in Example 2 of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0066] Example 1

[0067] This embodiment provides a method for identifying motor imagery based on multi-scale convolution and Transformer, which is used to identify the corresponding motor imagery category online based on the input EEG signal. Figure 1 As shown, the motor imagery recognition method includes the following steps:

[0068] S1: Obtain the original EEG signal to be identified, perform bandpass filtering and z-score normalization on it, and obtain the preprocessed EEG signal. In this embodiment, the preprocessing process of the EEG signal is as follows:

[0069] S11: Based on the characteristic frequency band of the movement-related components mu / beta rhythm, a wide-bandpass time-domain filter of 8-30 Hz is designed.

[0070] S12: Use a wide-bandpass time-domain filter to perform noise reduction on the original EEG signal to remove low-frequency and high-frequency noise in the original EEG signal.

[0071] S13: Use the following z-score normalization function to process the denoised EEG signal to alleviate signal fluctuations and the non-stationary characteristics of the EEG signal:

[0072]

[0073] Where x is the input signal; μ is the mean value of the input signal; σ is the variance of the input signal; the output signal X obtained after normalization conforms to the normal distribution with a mean of 0 and a standard deviation of 1.

[0074] S2: Extracting spatial features of the preprocessed EEG signal through a feature extraction module composed of multiple CSP spatial filters. In this embodiment, the feature extraction module uses common spatial patterns and spatial attention blocks to model the interdependence between feature channels.

[0075] Any CSP spatial domain filter is used to extract spatial features of a binary classification target, and its expression is as follows:

[0076]

[0077] In the above formula, X i represents the EEG matrix of the target of the i-th category after preprocessing; C i represents the spatial covariance matrix corresponding to the target of the i-th category; X represents the input preprocessed EEG signal; Z is the result obtained after CSP spatial domain filtering; W represents the corresponding spatial domain coefficient matrix.

[0078] When multiple classifications are required in the recognition task, multiple CSP spatial domain filters are cascaded to form the required feature extraction module. For example, if N categories are required, it can be divided into N dual classification tasks of one category and the remaining categories, and CSP spatial domain filters are constructed for each category.

[0079] S3: Use the channel attention module in Transformer to build a spatial self-attention mechanism, perform feature analysis on the feature map in the previous step, and implement feature channel weighting.

[0080] Among them, the expression of the channel attention module is as follows:

[0081]

[0082] Among them, Attention(Q,K,V) is the weighted feature representation; Q, K, V are the corresponding feature vectors after the original data is mapped to the three vector spaces through linear transformation, d k Indicates the scaling factor.

[0083] In this embodiment, by introducing the CSP spatial filter and spatial self-attention mechanism, the model can obtain spatial information; pay more attention to channels that are highly related to motion, reduce the influence of other irrelevant channels, and thus improve the classification accuracy of EEG signals.

[0084] S4: Use multiple convolution modules to perform convolution processing on the weighted two-dimensional data. The convolution processing process is as follows:

[0085] S41: Perform inter-channel convolution on the feature map output by the channel attention module. The convolution kernel shape of the inter-channel convolution is (Channels, 1); where Channels represents the number of channels of the original EEG signal, the convolution stride is (1, 1), and the data after convolution will be compressed from two-dimensional data to one-dimensional data.

[0086] S42: A multi-scale convolution module with three convolution kernels of different scales is used to perform one-dimensional convolution on the input data in the time domain; the convolution kernel shapes of the multi-scale convolution module are (1, 3), (1, 5), and (1, 7), and the convolution stride is (1, 1).

[0087] S43: Horizontally concatenate the local features captured by the convolution kernels of different scales to obtain fused features. In this embodiment, the use of multi-scale convolution blocks can capture distinguishable features of various non-overlapping typical frequency bands in the time domain of the EEG signal, which has an important impact on improving the classification accuracy of the EEG signal.

[0088] S5: Use the average pooling layer to divide the one-dimensional fusion feature into multiple feature slices according to the preset data length.

[0089] The data of the feature slices after the average pooling layer is represented as (L, V), where the pooling layer shape is (1,75), the stride is (1,15), L represents the length of the data after segmentation, and V represents the number of segmented segments; the length of the data after segmentation L is adaptively adjusted according to the number of time domain sampling points of the original EEG signal.

[0090] In this embodiment, the shape of the data after division is (140, 20), that is, the data length of each feature slice divided is 140, and the number of segments is 20.

[0091] S6: Use the self-attention module in Transformer to build a multi-head attention mechanism, perform feature analysis on the feature slices in the previous step, and extract the global features contained therein.

[0092] The expression of the constructed multi-head attention mechanism is as follows:

[0093]

[0094] In the above formula, headi represents the feature vector of the i-th feature slice after being processed by the self-attention module; MHA represents the multi-head attention mechanism; [head0; ...; head h-1 ] represents the feature vector head0~head h-1 Perform the connection operation; Q i , K i 、V i They represent the three feature vectors obtained after linear transformation of the i-th feature slice.

[0095] In this embodiment, the introduction of the spatial self-attention mechanism and the multi-head attention mechanism can perceive the global dependency of EEG signals, make full use of the highly correlated characteristics of EEG information before and after, and extract motor imagery classification features.

[0096] S7: Use a global average pooling module to process the output of the self-attention module to obtain refined features after global average pooling.

[0097] S8: The output of the global average pooling module is normalized and then connected to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function, and the category of the EEG signal is generated and identified.

[0098] The motor imagery recognition method based on multi-scale convolution and Transformer proposed in this invention combines deep learning with traditional signal processing to extract motor imagery EEG features from multiple dimensions in time and space, thereby improving prediction accuracy.

[0099] The present invention also considers the spatial characteristics of motor imagery, focusing on EEG channels highly correlated with movement while minimizing the influence of other irrelevant channels. A spatial self-attention mechanism is employed to model the interdependencies between feature channels, enabling weighting of individual feature channels. Furthermore, to capture distinguishable features across the various non-overlapping frequency bands typical of motor imagery, the present invention employs multi-scale convolutional blocks to extract deep temporal information.

[0100] Finally, the solution provided by the present invention fully considers the highly context-related characteristics of EEG signals, and uses a lightweight Transformer structure to perceive the global dependency of motor imagery EEG, which can significantly improve the accuracy of motor imagery classification detection technology.

[0101] Example 2

[0102] Based on Example 1, this embodiment also provides a motor imagery recognition system based on multi-scale convolution and Transformer, which is used to create a network model for analyzing and identifying EEG signals; the network model adopts the motor imagery recognition method based on multi-scale convolution and Transformer as in Example 1, and outputs the prediction result of the corresponding motor imagery category according to the input original EEG signal.

[0103] like Figure 2 As shown, the motor imagery classification detection system provided in this embodiment includes: a preprocessing module, a feature extraction module, a channel attention module, a convolution module, a feature segmentation module, a multi-head attention module, a feature refinement module, and a classification module.

[0104] The preprocessing module includes a noise reduction unit and a normalization unit. The noise reduction unit uses an 8-30Hz wideband-pass time-domain filter to reduce the noise of the original EEG signal, removing low-frequency and high-frequency noise. The normalization unit uses the z-score normalization function to process the noise-reduced EEG signal to reduce signal fluctuations and non-stationary characteristics of the EEG signal.

[0105] The feature extraction module uses multiple cascaded CSP spatial domain filters to extract features from the preprocessed EEG signals and obtain the spatial domain features contained therein.

[0106] The channel attention module is used to perform feature analysis and weighting on the spatial features of the input EEG signal to obtain weighted two-dimensional features.

[0107] The convolution module includes channel convolution unit, multi-scale convolution unit and feature concatenation unit. The channel convolution unit is used to perform inter-channel convolution operations on the output of the channel attention model; Figure 3 As shown in the figure, the multi-scale convolution unit is used to perform synchronous convolution processing on the output of the channel convolution unit using convolution kernels of three different scales; the feature splicing unit is used to horizontally splice the output of the multi-scale convolution unit to obtain the corresponding fusion features.

[0108] The feature segmentation module is used to divide the one-dimensional fusion features output by the convolution module into multiple feature slices according to the preset data length using the average pooling layer.

[0109] The multi-head attention module is used to perform feature analysis on the feature slices using a multi-head attention mechanism to extract the global features contained therein.

[0110] The feature refinement module is used to process the output of the self-attention module using the global average pooling module to obtain refined features after global average pooling.

[0111] The classification module is used to normalize the output of the global average pooling module and then connect it to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function to generate the category of the recognized EEG signal.

[0112] The motor imagery recognition system based on multi-scale convolution and Transformer provided in this embodiment mainly relies on the built-in network model to realize the analysis and recognition of EEG signals. The network model is essentially a deep learning algorithm. Specifically, Figure 4 As shown, the design process of the network model in the motor imagery recognition system based on multi-scale convolution and Transformer in this embodiment includes the following steps:

[0113] S01: Design an 8-30 Hz wide-bandpass time domain filter for denoising the original EEG signal, and a normalization filter using the z-score normalization function.

[0114] S02: Design a corresponding feature extraction module according to the number of classification results. The feature extraction module is composed of a cascade of multiple CSP spatial domain filters.

[0115] S03: Design a convolution module that includes a channel convolution unit, a multi-scale convolution unit, and a feature splicing unit; the convolution module is used to generate one-dimensional fusion features from the input two-dimensional features.

[0116] S04: Use a wideband-pass time-domain filter and a normalization filter to form the required pre-processing module.

[0117] S05: Build a network model that includes a preprocessing module, a feature extraction module, a channel attention module, a convolution module, a feature segmentation module, a multi-head attention module, a feature refinement module, and a classification module.

[0118] S06: Obtain a large number of EEG signal samples and add category labels to the samples to form the required data set, and divide the data set into a training set and a test set.

[0119] S07: The network model is trained using the training set and the test set until the preset training target is met and the model parameters are retained to obtain the required network model for analyzing and identifying EEG signals.

[0120] In this embodiment, prediction accuracy is used as the training target, and the loss function for evaluating the training target adopts the following cross entropy loss:

[0121]

[0122] In the above formula, M is the number of EEG samples, N is the number of categories, represents the true label of the mth sample, It represents the probability that the mth sample is predicted to be the nth category.

[0123] Example 3

[0124] This embodiment provides a decoding device for motor imagery EEG signals, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it creates a network model for analyzing and identifying EEG signals; the network model adopts the motor imagery recognition method based on multi-scale convolution and Transformer as in Example 1, and outputs the prediction result of the corresponding motor imagery category based on the input original EEG signal.

[0125] The motor imagery EEG signal decoding device in this embodiment is essentially a computer device. This computer device can be an embedded computing module capable of executing specific data processing tasks according to a pre-programmed program, or a general-purpose computing device such as a smartphone, tablet computer, laptop computer, desktop computer, rack-mounted server, blade server, tower server, or cabinet-mounted server (including a standalone server or a server cluster consisting of multiple servers) that can execute programs. The computer device in this embodiment includes at least, but is not limited to, a memory and a processor that can be interconnected via a system bus.

[0126] In this embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as the hard disk or internal memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store the operating system and various application software installed on the computer device. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0127] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of a computer device.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A motion imagery recognition method based on multi-scale convolution and Transformer, characterized in that: It is used to identify the corresponding motor imagery category online based on the input EEG signal. The motor imagery identification method includes the following steps: S1: Obtain the original EEG signal to be identified, perform bandpass filtering and z-score normalization on it, and obtain the preprocessed EEG signal; S2: Extracting spatial features of the preprocessed EEG signal through a feature extraction module composed of multiple CSP spatial filters; S3: Use the channel attention module in Transformer to build a spatial self-attention mechanism, perform feature analysis on the feature map in the previous step and implement feature channel weighting; S4: Use multiple convolution modules to perform convolution processing on the weighted two-dimensional data. The convolution processing process is as follows: S41: performing inter-channel convolution on the feature map output by the channel attention module. The convolution kernel shape of the inter-channel convolution is (Channels, 1), where Channels represents the number of channels of the original EEG signal, and the convolution stride is 1. After the convolution, the data will be compressed from two-dimensional data to one-dimensional data; S42: Perform one-dimensional convolution on the input data in the time domain using a multi-scale convolution module having three convolution kernels of different scales; the convolution kernel shapes of the multi-scale convolution module are (1, 3), (1, 5), and (1, 7), and the stride is (1, 1); S43: Horizontally concatenate the local features captured by convolution kernels of different scales to obtain fused features; S5: Use the average pooling layer to divide the one-dimensional fusion feature into multiple feature slices according to the preset data length; S6: Use the self-attention module in Transformer to build a multi-head attention mechanism to perform feature analysis on the feature slices in the previous step and extract the global features contained therein; S7: Process the output of the self-attention module using a global average pooling module to obtain refined features after global average pooling; S8: The output of the global average pooling module is normalized and then connected to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function, and then the category of the motor imagery EEG signal is generated and identified.

2. The method for motor imagery recognition based on multi-scale convolution and Transformer according to claim 1, wherein: In step S1, the preprocessing process of the EEG signal is as follows: S11: Design a wide-bandpass time-domain filter of 8-30 Hz based on the characteristic frequency bands of the movement-related components mu / beta rhythm; S12: performing noise reduction processing on the original EEG signal using the broadband bandpass time domain filter to remove low-frequency and high-frequency noise in the original EEG signal; S13: Use the following z-score normalization function to process the denoised EEG signal to alleviate signal fluctuations and the non-stationary characteristics of the EEG signal: Where x is the input signal; μ is the mean value of the input signal; σ is the variance of the input signal; the output signal X obtained after normalization conforms to the normal distribution with a mean of 0 and a standard deviation of 1.

3. The method for motor imagery recognition based on multi-scale convolution and Transformer as claimed in claim 1, characterized in that: Any of the CSP spatial domain filters is used to extract spatial features of a binary classification target, and its expression is as follows: In the above formula, X i represents the preprocessed EEG matrix of the target of the i-th category, C i represents the spatial covariance matrix corresponding to the target of the i-th category; X represents the input preprocessed EEG signal, Z is the result obtained after CSP spatial domain filtering; W represents the corresponding spatial domain coefficient matrix; When multi-classification is required in the recognition task, multiple CSP spatial domain filters are cascaded to form the required feature extraction module.

4. The method for motor imagery recognition based on multi-scale convolution and Transformer according to claim 1, wherein: The expression of the channel attention module in step S3 is as follows: Among them, Attention(Q,K,V) is the weighted feature representation; Q, K, V are the corresponding feature vectors after the original data is mapped to the three vector spaces through linear transformation, d k Indicates the scaling factor.

5. The method for motor imagery recognition based on multi-scale convolution and Transformer according to claim 4, wherein: In step S5, the data of the feature slices after the average pooling layer is segmented is represented as (L, V), where the pooling layer shape is (1,75), the stride is (1,15), L represents the length of the data after segmentation, and V represents the number of segmented segments; the length of the data after segmentation L is adaptively adjusted according to the number of time domain sampling points of the original EEG signal.

6. The method for motor imagery recognition based on multi-scale convolution and Transformer according to claim 4, wherein: The expression of the multi-head attention mechanism constructed in step S6 is as follows: In the above formula, head i represents the feature vector of the i-th feature slice after being processed by the self-attention module; MHA represents the multi-head attention mechanism; [head0; ...; head h-1 ] represents the feature vector head0~head h-1 Perform the connection operation; Q i , K i 、V i They represent the three feature vectors obtained after linear transformation of i feature slices.

7. A motor imagery recognition system based on multi-scale convolution and Transformer, characterized by: It is used to create a network model for analyzing and identifying EEG signals; the network model adopts the motor imagery recognition method based on multi-scale convolution and Transformer as described in any one of claims 1 to 6, and outputs the prediction result of the corresponding motor imagery category based on the input original EEG signal; The motor imagery classification detection system comprises: The preprocessing module includes a noise reduction unit and a normalization unit; the noise reduction unit is used to perform noise reduction processing on the original EEG signal through a wide-bandpass time domain filter of 8-30 Hz to remove low-frequency and high-frequency noise in the original EEG signal; the normalization unit uses a z-score normalization function to process the noise-reduced EEG signal to alleviate signal fluctuations and non-stationary characteristics of the EEG signal; The feature extraction module uses multiple cascaded CSP spatial domain filters to extract features from the preprocessed EEG signal and obtain the spatial domain features contained therein; The channel attention module is used to perform feature analysis and weighting on the spatial features of the input EEG signal to obtain weighted two-dimensional features; The convolution module includes a channel convolution unit, a multi-scale convolution unit, and a feature splicing unit; the channel convolution unit is used to perform inter-channel convolution operations on the output of the channel attention model; the multi-scale convolution unit is used to perform synchronous convolution processing on the output of the channel convolution unit using convolution kernels of three different scales; the feature splicing unit is used to horizontally splice the output of the multi-scale convolution unit to obtain corresponding fusion features; The feature segmentation module is used to divide the one-dimensional fusion features output by the convolution module into multiple feature slices according to the preset data length using the average pooling layer; A multi-head attention module is used to perform feature analysis on the feature slices using a multi-head attention mechanism to extract global features contained therein; A feature refinement module, which is used to process the output of the self-attention module using a global average pooling module to obtain refined features after global average pooling; The classification module is used to normalize the output of the global average pooling module and then connect it to the fully connected layer. Finally, the prediction probability of different categories is obtained through the Softmax function, thereby generating the category of the identified EEG signal.

8. The motor imagery recognition system based on multi-scale convolution and Transformer as claimed in claim 7, characterized in that: The design process of the network model includes the following steps: S01: Design an 8-30 Hz wide-bandpass time domain filter for noise reduction of raw EEG signals, and a normalization filter using the z-score normalization function; S02: Design a corresponding feature extraction module according to the number of classification results, wherein the feature extraction module is formed by cascading multiple CSP spatial domain filters; S03: Design a convolution module including a channel convolution unit, a multi-scale convolution unit, and a feature splicing unit; the convolution module is used to generate a one-dimensional fusion feature from the input two-dimensional features; S04: Using the broadband bandpass time domain filter and the standardization filter to form a required preprocessing module; S05: Build a network model including preprocessing module, feature extraction module, channel attention module, convolution module, feature segmentation module, multi-head attention module, feature refinement module and classification module; S06: Obtain a large number of EEG signal samples and add category labels to the samples to form the required data set, and divide the data set into a training set and a test set; S07: The network model is trained using the training set and the test set until the preset training target is met and the model parameters are retained to obtain the desired network model for analyzing and identifying EEG signals.

9. The motor imagery recognition system based on multi-scale convolution and Transformer according to claim 7, characterized in that: In step S07, the prediction accuracy is used as the training target, and the loss function for evaluating the training target adopts the following cross entropy loss: In the above formula, M is the number of EEG samples, N is the number of categories, represents the true label of the mth sample, It represents the probability that the mth sample is predicted to be the nth category.

10. A device for decoding motor imagery EEG signals, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it creates a network model for analyzing and identifying EEG signals; the network model adopts the motor imagery recognition method based on multi-scale convolution and Transformer as described in any one of claims 1-6, and outputs the prediction result of the corresponding motor imagery category based on the input original EEG signal.

Images