Visual perception and imagination combined semantic decoding method based on electroencephalogram

Abstract

The invention discloses a visual perception and imagination combined semantic decoding method based on an electroencephalogram, and belongs to the technical field of brain-computer interfaces. The invention aims to provide an electroencephalogram-based visual perception and imagination combined semantic decoding method which introduces imagination EEG and uses perception and imagination electroencephalogram signals after group normalization, sample entropy and relative power as input features and data. According to the method, after electroencephalogram signals are collected, basic preprocessing modes such as cutting, marking, artifact removal, filtering, calibration and re-reference are carried out on the collected sensing and imagination EEG, two relatively pure electroencephalogram signals are obtained, group normalization is carried out on the two electroencephalogram signals according to brain region categories, and electroencephalogram data after group normalization are obtained. The invention provides an efficient, robust and interpretable method for the field, and the method has important practical application value and market prospect.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface technology. Background Technology

[0002] Brain-computer interface (BCI) technology is a system that establishes direct information interaction between the human brain and external devices (such as computers, prostheses, and assistive communication devices) without relying on peripheral nerves and muscle tissue. Its core workflow typically includes: acquiring neural activity signals from the human brain through invasive (e.g., intracranial electrodes) or non-invasive (e.g., electroencephalography) sensors; preprocessing the signals (noise reduction, filtering), feature extraction, and pattern recognition (e.g., deep learning, machine learning algorithms) to convert the neural signals into commands that can be recognized by external devices (e.g., controlling wheelchair movement, text input, robotic arm movements); and simultaneously, converting feedback information from external devices into signals perceptible to humans (e.g., directly observing wheelchair movement, text display on a monitor), forming a closed-loop interaction. In today's era of deep integration of artificial intelligence and brain science, the development of BCI-related fields has become a national strategy and a cutting-edge direction of global technological development.

[0003] EEG-based visual perception semantic decoding and EEG-based imaginative semantic decoding are two relatively independent fields of development. Research on decoding visual perception EEG signals has progressed relatively quickly, shifting from traditional machine learning methods to a deep learning-dominated approach, with researchers continuously developing algorithms that offer better pattern recognition and greater robustness. However, decoding imaginative EEG signals is limited by the subjectivity and instability of the imaginary process, resulting in relatively poorer decoding performance compared to visual perception decoding.

[0004] The existing typical decoding technology in the field of visual cognition based on EEG is roughly as follows: (1) Design an experimental paradigm, collect the EEG data of the subjects, and perform basic preprocessing; (2) Divide the preprocessed data into different datasets, use Fourier transform to convert from the time domain to the frequency domain, and extract Riemann space features in the frequency domain data; (3) Use feature selection to splice the Riemann space features of different time periods and perform feature selection; (4) Classify the selected features.

[0005] Existing technologies can decode visual cognitive processes, but there are still some key shortcomings: (1) Existing technologies often only use traditional machine learning methods for feature extraction and classification. Although they are highly interpretable, they are not good at feature extraction and are greatly affected by the types of features selected. This may lead to the inability to deploy the method or poor accuracy in specific scenarios; (2) Existing technologies only use EEG data induced by the visual perception paradigm as input signals, ignoring the commonality between perception and imagination and not taking similar imagination EEG data into consideration. This affects the decoding potential and robustness of the method; (3) Existing technologies do not make full use of the effective information in EEG data. Although EEG is a high temporal resolution signal, there are still unused effective features in its frequency domain and spatial domain (electrode / lead dimension).

[0006] Traditional visual cognitive semantic decoding methods have poor accuracy and robustness. The technical problems are: (1) Existing decoding methods only use traditional machine learning methods or deep learning methods, and there are problems with the interpretability and extraction of the methods; (2) Existing methods ignore the systematic similarity between visual perception and imagination, which affects the decoding potential of the methods; (3) Existing methods do not make sufficient use of the multi-domain effective information in EEG signals, which affects the robustness and decoding effect of the methods. Summary of the Invention

[0007] The purpose of this invention is to introduce the concept of EEG, and to develop a joint semantic decoding method for visual perception and imagination based on EEG, using group-normalized perception and imagination EEG signals, sample entropy and relative power as input features and data.

[0008] The steps of this invention are: S1. Normalize the two relatively pure EEG signals according to brain region categories to obtain group-normalized EEG data: (1) (2) (3) (4) (5) (6) In the formula: x is the input feature map with dimension 1. ; It is the sample size; For channel dimensions; The feature dimension is 1×sampling dimension in this invention; This refers to the number of groups, specifically the brain region dimension. The maximum number of electrodes contained in each brain region. Let be the mean and variance of the g-th feature group; These are the learnable scaling parameter and the learnable offset parameter, respectively; The data is grouped and normalized; Calculated sample entropy: (7) In the formula: m is the embedding dimension of the subsequence; r is the similarity threshold; N is the total number of data points in the time series; Representative and The proportion of similar subsequences It is an m-dimensional vector. For all The average value; The relative power of the k-th frequency band: (8) In the formula: The signal sampling frequency, ; It is the power spectral density at frequency f. ; for The discrete Fourier transform result, , The kth target frequency band, The local power within the k-th frequency band. ; Total power across the entire frequency band ; S2. Construct a deep learning model based on convolution, ReLU activation function, pooling layer, linear layer, and KAN: The convolution calculation process is shown in equation (9). (9) In the formula: the dimension of the input tensor Z is The dimension of the convolution kernel F is h. f ×w f The dimension of the output feature map Y is ; Indicates the number of filters; Represents the output channel index; Identify the time index in the output; h f and w f These represent the spatial dimensions of the filter in the channel direction and the time direction, respectively. Corresponding to the The bias of each filter; The ReLU activation function is shown in equation (10). (10) In the formula: represents the input and output feature values ​​of the nth sample, c channels, and spatial coordinates (i,j), respectively; max(·,·) represents the maximum value function; The pooling layer calculation is shown in equation (11). (11) In the formula: ⌊⋅⌋ represents rounding down; and represents the height and width of the input and output features, respectively; p, k, and s are the kernel size, stride, and padding amount of the pooling operation, respectively. The calculation of the linear layer is shown in equation (12). (12) In the formula: y is the input feature; W is the weight matrix; b is the offset vector; y is the output of the linear layer; The calculation process of the KAN neural network architecture is shown in equations (13) to (17). (13) (14) (15) (16) (17) In the formula: Let represent the function matrix of the m-th layer, defining the transformation within each KAN layer; hin represents the number of input features for that layer. Map the input features zk of KAN to the output; each Both are represented using univariate B-splines; v represents the weight matrix, d j These are the trainable coefficients, C j These are B-spline basis functions; S3. Train a deep learning model using sample entropy, relative power, and group-normalized EEG data from two types of EEG signals; validate the trained deep learning model using a test set to obtain the final output of predicted image semantics.

[0009] The positive effects of this invention are as follows: (1) This method combines sample entropy, relative power, and group-normalized perceptual and imaginative EEG signals, enabling it to possess the interpretability advantage of machine learning while also having the advantage of good semantic decoding performance of deep learning. In addition, this method combines perceptual and imaginative EEG for semantic decoding, allowing it to obtain the semantic information hidden in the cognitive process from two perspectives, systematically improving the decoding accuracy and robustness of the semantic decoding method; (2) This method constructs a novel deep learning network based on convolution, ReLU activation function, pooling, KAN and linear layers to extract and fuse EEG features of different categories, which can more effectively utilize the multi-domain information of EEG and improve the effect of semantic classification. In summary, compared with traditional EEG-based visual semantic decoding methods, this invention provides an efficient, robust, and interpretable method for this field, which has significant practical application value and market prospects. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a visual perception and imagination experiment paradigm of an embodiment of the present invention; Figure 2 This is a schematic diagram of a model structure for a semantic decoding method based on visual perception and imagination provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an experimental paradigm for an embodiment of the present invention. Detailed Implementation

[0011] The EEG-based visual perception and imagination semantic decoding method belongs to the feature recognition and pattern recognition stages of BCI (Brain Correlation Interaction). Furthermore, this method has significant value in extended reality, medical rehabilitation, and neurological disease diagnosis. Research in cognitive neuroscience and related fields has confirmed that there are numerous overlapping neural circuits and similar brain activation patterns between perception and imagination. The semantic information of cognition is invariant in both processes (i.e., the same semantic cognitive information can be expressed in these two different cognitive processes), which is the basis and motivation for this invention to combine the two cognitive processes.

[0012] The present invention will be further described in detail below: Step 1: In a suitable EEG signal acquisition environment, follow the steps as follows Figure 1 The paradigm shown is for acquiring EEG signals. The subject views a monitor displaying images, and the process is as follows: Figure 1As shown, the display first shows "Start" to indicate to the subject that the experiment is about to begin; then "Prompt" is displayed to inform the subject of the next semantic image category; after a short buffer period, the corresponding semantic image is displayed, and the subject's EEG of viewing the image is collected as perceptual EEG data; finally, after another short buffer period, the screen goes black, during which the subject imagines the semantic image they just saw, and the subject's EEG of imagining the image is collected as perceptual EEG.

[0013] Step 2: Perform basic preprocessing on the collected perceptual and imaginative EEG data, including cropping, labeling, artifact removal, filtering, calibration, and rereference, to obtain relatively pure EEG signals (data dimension: 2×N×C×T). Classify the EEG signals according to brain region categories, ensuring that EEG signals from electrodes in the same brain region are in the same dimension (data dimension: 2×N×R×C՚×T). Finally, divide the preprocessed dataset into appropriate training and test sets based on the data volume.

[0014] Step 3: Normalize the two EEG signals according to brain region categories to obtain the normalized EEG data. At the same time, calculate the average sample entropy of each group and the average relative power of each frequency band (5 frequency bands) in the two EEG signals. The frequency bands can be divided as follows: (1) δ wave: frequency range 0.5~4Hz; (2) θ wave: frequency range 4~8Hz; (3) α wave: frequency range 8~13Hz; (4) β wave: frequency range 13~30Hz; (5) γ wave: frequency range 30~100Hz.

[0015] Group normalization is a normalization method in deep learning. It divides features into several groups and normalizes each group of features separately. Its general calculation process is shown in formulas (1)-(6): (1) (2) (3) (4) (5) (6) In formulas (1)-(6), x is the input feature map with dimension 1. ; It is the sample size; For channel dimensions; The feature dimension is 1×sampling dimension in this invention; This refers to the number of groups, specifically the brain region dimension. The maximum number of electrodes contained in each brain region, padded with zeros if necessary; Let be the mean and variance of the g-th feature group; These are the learnable scaling parameter and the learnable offset parameter, respectively; The data is normalized for the group.

[0016] Sample entropy (SampEn) is a nonlinear dynamic index that quantifies the irregularity (complexity) of time series data. It outputs a non-negative real number as a logarithmic ratio by statistically analyzing the similarity probability of sequence patterns under a fixed embedding dimension. This effectively avoids the self-matching problem and is widely used for complexity analysis of physiological signals such as EEG and ECG. Its general calculation process is shown in formula (7): (7) In formula (7), m is the embedding dimension of the subsequence; r is the similarity threshold; N is the total number of data points in the time series; and x is the input time series. ; The set of subsequences has N-m+1 elements. ; Let i be an m-dimensional vector, and let i be the i-th sequence in the sub-sequence set. Chebyshev distance (maximum absolute difference) between two m-dimensional subsequences. ; Representative and The proportion of similar subsequences ( ), ; For all The average value, ; This is the calculated sample entropy.

[0017] Relative power is the percentage of power in a specific frequency band (such as delta wave, theta wave, alpha wave, and beta wave) relative to the total power of the entire frequency band in the frequency domain analysis of signals such as electroencephalograms (EEG). It is used to quantitatively describe the relative strength of EEG activity in different frequency bands and is one of the core frequency domain features for revealing brain functional states (such as wakefulness, sleep, and cognitive tasks). Its general calculation process is shown in formula (8): (8) In formula (8), Represents the time-domain input signal (discrete-time series). N is the total number of signal sampling points; The signal sampling frequency, ; for The discrete Fourier transform result, j is the imaginary unit; It is the power spectral density at frequency f. ; The kth target frequency band (e.g., alpha and theta waves in EEG); The local power within the k-th frequency band. ; Total power across the entire frequency band ; This represents the relative power of the k-th frequency band.

[0018] Step 4: Construct a deep learning model based on convolution, ReLU activation function, pooling layers, linear layers, and KAN. A schematic diagram is shown below. Figure 2 As shown in Table 1.

[0019] Table 1. Network Structure of Deep Learning Models

[0020] In Table 1, layers 1-10 are used simultaneously on the normalized EEG signals, forming a bi-branch structure. Since the branches are the same, only the parameters of a single branch are given. Layer 11 receives the features extracted from the first 10 layers and splices them together with the relative power on the samples to form joint features input to layer 12. Layers 12-14 are used to classify the joint features and output the final predicted semantic category.

[0021] Convolution uses a sliding convolution kernel to perform weighted summation on local regions of the input data, achieving local feature extraction and parameter sharing. It is widely used for feature learning of grid-structured data such as images. Its general calculation process is shown in formula (9): (9) In formula (9), the dimension of the input tensor Z is The dimension of the convolution kernel F is h. f ×w f (Total) (independent filters); the dimension of the output feature map Y is ; Indicates the number of filters; Represents the output channel index; Identify the time index in the output; h f and w f These represent the spatial dimensions of the filter in the channel direction and the time direction, respectively. Corresponding to the The bias of each filter.

[0022] The ReLU activation function solves the gradient vanishing problem and introduces sparsity through a non-linear mapping of "outputting itself when the input is greater than 0, otherwise outputting 0", providing a foundation for efficient training of deep neural networks. Its general calculation process is shown in formula (10): (10) In formula (10), represents the input and output feature values ​​of the nth sample, c channels, and spatial coordinates (i,j), respectively; max(·,·) represents the maximum value function to ensure that the output is non-negative.

[0023] Pooling layers perform max, average, and other aggregation operations on local feature regions, reducing the spatial dimension of the feature map to decrease computational cost while enhancing the model's robustness to local translations. The general calculation process is shown in formula (11): (11) In formula (11), ⌊⋅⌋ represents rounding down; and represents the height and width of the input and output features, respectively; p, k, and s are the kernel size, stride, and padding amount of the pooling operation, respectively.

[0024] Linear layers transform the input features linearly through the weight matrix and bias term of the fully connected layer (i.e., output = input × weight + bias), achieving global integration and dimensionality mapping of features. This is commonly used for classification or regression tasks at the network's end. Its general calculation process is shown in formula (12): (12) In formula (12), y is the input feature; W is the weight matrix; b is the offset vector; y is the output of the linear layer.

[0025] KAN (Kolmogorov-Arnold Network) is a neural network built on Kolmogorov's theorem. It replaces traditional neurons with the superposition of univariate activation functions and achieves more flexible function fitting by adaptively adjusting the activation function parameters. It often exhibits excellent approximation ability in low-data scenarios. Its general calculation process is shown in formulas (13)-(17): (13) (14) (15) (16) (17) In the formula (13)-(17), Let represent the function matrix of the m-th layer, defining the transformation within each KAN layer; hin represents the number of input features for that layer. Map the input features zk of KAN to the output; each Both are represented using univariate B-splines; v represents the weight matrix, d j These are the trainable coefficients, C j It is a B-spline basis function.

[0026] Step 5: Train a deep learning model using the sample entropy, relative power, and group-normalized EEG data of the two types of EEG signals. The model structure is as follows: Figure 2 As shown.

[0027] Step 6: Validate the trained deep learning model using the test set to obtain the final output of the predicted image semantics.

[0028] Implementation example: Step 1: Prepare the experimental environment: In a shielded room equipped with special electromagnetic shielding walls, maintain a temperature of 20℃, relative humidity between 40% and 60%, light intensity of 300-500 lux, and warm white light of 4000K-5000K. Avoid external natural light interference, ensuring uniform, soft, and stable indoor temperature, humidity, and light. Noise levels should be below 40dB. The subject's chair should be comfortable and adjustable, ensuring the subject's head is aligned with the center of the monitor. The laboratory should be far from large motors and transformers, and devices that may generate electromagnetic radiation, such as mobile phones and wireless routers, should be turned off. Select 12 healthy subjects (6 males and 6 females) with consistent educational levels, all right-handed, and without any bad habits. Select two semantic images: flowers and keyboards, with 20 different images prepared for each type. Use a 128-electrode EEG cap, with electrode arrangement conforming to the international 10-20 standard, and a sampling dimension of 2kHz. EEG signals were collected from 12 subjects in rotation. The collection time for each subject should not exceed 60 minutes per day. The experiment should be stopped immediately if the subject felt fatigued during the experiment. The experimental procedure displayed on the monitor is as follows: Figure 3 As shown (taking flower semantics as an example), step 1 of the complete technical solution can also be seen. The total number of sensory and imaginative EEG data collected is (12000, 128, 4000), of which sensory and imaginative data are each half (6000, 128, 4000).

[0029] Step 2: The acquired perceptual and imaginative EEG data undergoes basic preprocessing methods such as cropping, semantic (flower or keyboard) and category (perceptual or imaginative) labeling, artifact removal, filtering, calibration, and rereference to obtain relatively pure EEG signals of two types (data dimension: 2×6000×128×4000). The EEG signals are then categorized by brain region (prefrontal lobe, frontal lobe, parietal lobe, left temporal lobe, right temporal lobe, occipital lobe, central parietal gyrus) to ensure that EEG signals from electrodes in the same brain region are in the same dimension (data dimension: 2×6000×7×20×4000). Finally, the preprocessed dataset is divided into a training set and a test set with a 9:1 ratio based on the data volume.

[0030] Step 3: Normalize the two EEG signals according to brain region categories to obtain the normalized EEG data. At the same time, calculate the sample entropy of each group and the relative power of each frequency band (5 frequency bands) in the two EEG signals. The frequency bands can be divided as follows: (1) δ wave: frequency range 0.5~4Hz; (2) θ wave: frequency range 4~8Hz; (3) α wave: frequency range 8~13Hz; (4) β wave: frequency range 13~30Hz; (5) γ wave: frequency range 30~100Hz. The obtained sample entropy data dimension is (2×6000×128×1), and the obtained relative power data dimension is (2×6000×128×5). The calculation process is shown in formula (7) and formula (8). The obtained average sample entropy and average relative power are (2×6000×1) and (2×6000×5), respectively.

[0031] Step 4: Construct a deep learning model based on convolutional layers, ReLU activation function, pooling layers, linear layers, and KAN. The model is built using the PyTorch architecture, and the structural diagram is shown below. Figure 2 As shown in Table 1, the calculation process can be seen in formulas (9) to (17), the specific parameters are shown in Table 2, and the specific hyperparameters are shown in Table 3.

[0032] Table 2 Specific parameters of the deep learning model

[0033] Table 3 Hyperparameters of Deep Learning Models

[0034] Step 5: Train a deep learning model using the sample entropy, relative power, and group-normalized EEG data of the two types of EEG signals. The model structure is as follows: Figure 2 As shown.

[0035] Step 6: Validate the trained deep learning model using the test set to obtain the final output of the predicted image semantics. See Table 4 for specific results.

[0036] Table 4. Specific experimental results (%)

[0037] Table 4 shows the accuracy performance comparison results (unit: %) of different comparison models, the model of this invention, and various ablation models. The accuracy of the traditional machine learning model Support Vector Machine (SVM) is 78.5%, while the accuracy of the classic deep learning models Long Short-Term Memory Network (LSTM) and Convolutional Neural Network (CNN) reaches 68.1% and 82.2%, respectively, reflecting the basic performance level of existing mainstream models in this task. To verify the necessity of the core components of the model of this invention, we designed three sets of ablation experiments: after removing the sample entropy input module, the model accuracy dropped to 87.9%; after removing the relative power input module, the accuracy declined to 84.5%; and after removing the group normalization input module, the accuracy dropped sharply to 69.0%, only slightly higher than the LSTM model. In comparison, the complete model provided by this invention achieved an accuracy of 92.3%, significantly outperforming traditional and classic models such as SVM, LSTM, and CNN (improving by 13.8, 24.2, and 10.1 percentage points respectively), and also surpassing all ablation models. This fully demonstrates the crucial role of the three core components—sample entropy input, relative power input, and group normalization input—in improving model performance. The absence of any one of these components leads to varying degrees of accuracy decline, especially the group normalization module, which is vital for ensuring model stability and fitting ability. Furthermore, the model of this invention maintains a high accuracy of over 90% in the task scenario, exhibiting superior feature extraction and classification performance compared to existing models, highlighting its application advantages and technological innovation in related research fields.

[0038] The key to this invention lies in: (1) The introduction of imaginative EEG enables the method described in this patent to acquire semantic cognitive information from multiple perspectives; (2) The group-normalized perception and imagination EEG signals, sample entropy and relative power are used as the input features and data of this method; (3) A novel fusion deep learning model is proposed, which is based on convolution, pooling, activation functions, KAN and linear layers.

[0039] The protected points of this invention are: (1) Selected semantic classification features and data: We used group-normalized perception and imagination EEG data based on brain regions, relative power based on 5 frequency bands, and sample entropy as semantic classification features and input data. (2) The proposed novel deep learning model: The deep learning model proposed in this patent, based on convolution, pooling, activation functions, KAN, and linear layers, has the specific structure shown in Table 1. Figure 2 .

Claims

1. A method for joint semantic decoding of visual perception and imagination based on electroencephalography (EEG), characterized in that: The steps are as follows: S1. Normalize the two relatively pure EEG signals according to brain region categories to obtain group-normalized EEG data: （1） （2） （3） （4） （5） （6） In the formula: x is the input feature map with dimension 1. ; It is the sample size; For channel dimensions; The feature dimension is 1×sampling dimension in this invention; This refers to the number of groups, specifically the brain region dimension. The maximum number of electrodes contained in each brain region. Let be the mean and variance of the g-th feature group; These are the learnable scaling parameter and the learnable offset parameter, respectively; The data is grouped and normalized; Calculated sample entropy: （7） In the formula: m is the embedding dimension of the subsequence; r is the similarity threshold; N is the total number of data points in the time series; Representative and The proportion of similar subsequences It is an m-dimensional vector. For all The average value; The relative power of the k-th frequency band: （8） In the formula: The signal sampling frequency, ; It is the power spectral density at frequency f. ; for The discrete Fourier transform result, , The kth target frequency band, The local power within the k-th frequency band. ; Total power across the entire frequency band ; S2. Construct a deep learning model based on convolution, ReLU activation function, pooling layer, linear layer, and KAN: The convolution calculation process is shown in equation (9). （9） In the formula: the dimension of the input tensor Z is The dimension of the convolution kernel F is h. f ×w f The dimension of the output feature map Y is ; Indicates the number of filters; Represents the output channel index; Identify the time index in the output; h f and w f These represent the spatial dimensions of the filter in the channel direction and the time direction, respectively. Corresponding to the The bias of each filter; The ReLU activation function is shown in equation (10). （10） In the formula: These represent the input and output feature values ​​of the nth sample, c channels, and spatial coordinates (i,j), respectively. max(·,·) represents the function that takes the maximum value; The pooling layer calculation is shown in equation (11). （11） In the formula: ⌊⋅⌋ represents rounding down; and represents the height and width of the input and output features, respectively; p, k, and s are the kernel size, stride, and padding amount of the pooling operation, respectively. The calculation of the linear layer is shown in equation (12). （12） In the formula: For input features; W is the weight matrix; b is the offset vector; y is the output of the linear layer; The calculation process of the KAN neural network architecture is shown in equations (13) to (17). （13） （14） （15） （16） （17） In the formula: This represents the function matrix of the m-th layer, defining the transformation within each KAN layer; hin represents the number of input features of this layer. Map the input features zk of KAN to the output; each Both are represented using univariate B-splines; v represents the weight matrix, d j These are the trainable coefficients, C j These are B-spline basis functions; S3. Train a deep learning model using sample entropy, relative power, and group-normalized EEG data from two types of EEG signals; validate the trained deep learning model using a test set to obtain the final output of predicted image semantics.

Images