An evaluation method and system of multi-domain fusion based on electroencephalogram data

Abstract

The application discloses a kind of based on multi-domain fusion of electroencephalogram data's super multi-level attention evaluation method and system, method includes the following steps: using electroencephalogram signal acquisition equipment to collect the attention electroencephalogram signal of test personnel, and carry out filtering processing, obtain the processed electroencephalogram signal, and based on the processed electroencephalogram signal carries out attention state division, obtains state division result;Using fast fourier transform to the processed electroencephalogram signal carries out time-frequency decomposition, obtains weighted time-domain signal;Using Welch method to the processed electroencephalogram signal carries out power spectral density estimation, obtains multi-channel fusion image;Parallel model structure is constructed, and based on weighted time-domain signal, multi-channel fusion image and state division result carries out training to parallel model structure, obtains attention evaluation model;The attention electroencephalogram signal of personnel to be evaluated is collected, and based on attention evaluation model, the attention state of personnel to be evaluated is evaluated, and obtains attention evaluation result.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and attention evaluation technology, specifically relating to a multi-level attention evaluation method and system based on multi-domain fusion of EEG data. Background Technology

[0002] With the development of electroencephalography (EEG) technology, attention assessment based on EEG data has become a research hotspot in fields such as human-computer interaction, cognitive research, and intelligent monitoring. Current technologies mainly focus on extracting features from EEG signals in the time, frequency, and spatial domains, combined with traditional machine learning or deep learning methods, to assess attentional states. However, these technologies still have some significant shortcomings, particularly in multi-level attention assessment and accurate classification.

[0003] Traditional EEG attention assessment methods mostly rely on single-dimensional features (such as power spectral density, frequency band energy, etc.) for state classification. However, this approach often neglects the fusion of multi-dimensional features of EEG signals, making it difficult to accurately assess individual attentional changes under complex tasks. For example, existing methods typically use spectrograms or time-frequency maps to analyze attentional states, but they are relatively weak in handling the spatiotemporal correlation of spectral features and the fine-grained classification of multi-level attentional states.

[0004] Most existing EEG attention assessment methods rely on traditional feature extraction methods, such as features calculated based on entropy and power spectrum. While these features can reflect attentional states to a certain extent, existing methods have limitations in modeling the complexity and nonlinearity of high-dimensional EEG signals. Furthermore, existing deep learning models (such as CNNs and LSTMs) lack the ability to analyze multi-level, time-varying EEG signal features when processing EEG image data, especially in terms of the incomplete modeling of multi-level attentional states during training, thus limiting performance.

[0005] Currently, data annotation in EEG attention assessment typically relies on manual methods, dividing data into several approximate attention state categories. While these methods are simple and effective, due to individual differences and task complexity, existing annotation methods often fail to accurately reflect an individual's dynamic attention state in continuous tasks, especially for the assessment of multi-level attention, where the coarseness of the annotation becomes a bottleneck for accurate modeling. Furthermore, most existing technologies do not consider how to avoid mutual interference between different states in multi-level attention assessment, further affecting the accuracy of the assessment results.

[0006] Therefore, the multi-level attention evaluation method based on EEG data still has many shortcomings. There is an urgent need to develop more accurate and multi-dimensional feature extraction methods and deep learning models to effectively capture the dynamic changes in attention state and achieve efficient multi-level classification. Summary of the Invention

[0007] This invention aims to address the shortcomings of existing technologies and provides the following solutions:

[0008] A multi-domain fusion-based method for evaluating attention based on electroencephalogram (EEG) data includes the following steps:

[0009] S1. Collect the attention EEG signals of the test personnel using an EEG signal acquisition device, and perform filtering processing to obtain the processed EEG signals. Then, classify the attention state based on the processed EEG signals to obtain the state classification results.

[0010] S2. The processed EEG signal is decomposed into time and frequency using Fast Fourier Transform to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and spliced ​​using a frequency domain attention mechanism to obtain a weighted time domain signal.

[0011] S3. The power spectral density of the processed EEG signal is estimated using the Welch method to obtain the EEG power spectral density value. The EEG power spectral density value is mapped to a topographic map matrix. The topographic map matrix of all frequency bands is grayscaled and resized. The adjusted grayscale images are then stitched together to obtain a multi-channel fused image.

[0012] S4. Construct a parallel model structure, and train the parallel model structure based on the weighted time-domain signal, the multi-channel fused image, and the state partitioning result to obtain an attention evaluation model;

[0013] S5. Collect the attention EEG signals of the person to be evaluated, and evaluate the attention state of the person to be evaluated based on the attention evaluation model to obtain the attention evaluation result.

[0014] Preferably, S2 includes:

[0015] The processed EEG signal was decomposed into time and frequency using Fast Fourier Transform to obtain the spectrum:

[0016]

[0017] Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index;

[0018] Define five EEG frequency bands: Delta, Theta, Alpha, Beta, and Gamma. For each EEG frequency band i, i∈{δ,θ,α,β,γ}, construct the corresponding frequency mask function:

[0019]

[0020] Where, m i (f) represents the frequency masking function, where f represents a continuous frequency variable. i,min f represents the lower cutoff frequency of the i-th frequency band. i,max This represents the upper cutoff frequency of the i-th frequency band;

[0021] Using the frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band:

[0022] X i [k] = X[k]·m i (f k )

[0023]

[0024] Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s Indicates the sampling frequency;

[0025] Performing an inverse fast Fourier transform on the frequency domain representation yields the time-varying characteristics of the original EEG signal:

[0026]

[0027] Where, x i [n] represents the time-varying characteristics of the original EEG signal in the i-th frequency band;

[0028] The time-varying features within each frequency band are weighted and concatenated to obtain the weighted time-domain signal:

[0029]

[0030] in, This represents the weighted time-domain signal. This represents the attention feature representation, z i α represents the original weight value of frequency band i. i This represents the learnable weights of frequency band i, and concat represents the concatenation operation.

[0031] Preferably, S3 includes:

[0032] The power spectral density of the processed EEG signal was estimated using the Welch method to obtain the EEG power spectral density value:

[0033]

[0034] Among them, P c (f) represents the brainwave power spectral density value. This represents the signal segment within the k-th window function, where K represents the number of window functions and f represents the frequency.

[0035] Using the standard coordinate mapping function of electrodes on the scalp The EEG power spectral density values ​​P of each electrode c (f) Mapping onto a two-dimensional plane yields the topographic map matrix M corresponding to each frequency band. i ;

[0036] The obtained topographic map matrix M i Convert to grayscale and map to grayscale image I. i and the grayscale image I i Resize the image to a uniform size of H×W;

[0037] The adjusted grayscale images are stitched together along the channel dimension to obtain the multi-channel fused image:

[0038] I = concat(I) δ I θ I α I β I γ )

[0039] Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents the adjusted grayscale image of the Gamma EEG band.

[0040] Preferably, the parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer;

[0041] The video feature extraction branch based on AFC-EEGNET consists of a two-dimensional convolutional layer, a deep convolutional layer, a separable convolutional layer, a flattening layer, and a fully connected layer.

[0042] The ResNet-KAN-based spatial-frequency domain feature extraction branch consists of a ResNet18 residual structure and a KAN network.

[0043] The fusion layer is:

[0044]

[0045] in, This indicates the classification score after fusion. This represents the output score of the video feature extraction branch based on AFC-EEGNET. λ represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN, and λ represents the adjustable fusion coefficient.

[0046] The present invention also provides a multi-domain fusion-based super-multilevel attention evaluation system based on EEG data. The evaluation system applies the evaluation method described above and includes: a signal acquisition module, a temporal feature extraction module, a multi-channel fusion module, a model construction module, and an attention evaluation module.

[0047] The signal acquisition module uses an EEG signal acquisition device to acquire the EEG signals of the test personnel's attention, performs filtering processing to obtain processed EEG signals, and performs attention state classification based on the processed EEG signals to obtain state classification results.

[0048] The time-domain feature extraction module uses Fast Fourier Transform to perform time-frequency decomposition on the processed EEG signal to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and concatenated using a frequency domain attention mechanism to obtain a weighted time-domain signal.

[0049] The multi-channel fusion module uses the Welch method to estimate the power spectral density of the processed EEG signal to obtain the EEG power spectral density value. The EEG power spectral density value is mapped to a topographic map matrix, and the topographic map matrix of all frequency bands is grayscaled and resized. The adjusted grayscale images are then stitched together to obtain a multi-channel fused image.

[0050] The model building module is used to build a parallel model structure and train the parallel model structure based on the weighted time-domain signal, the multi-channel fused image and the state partitioning result to obtain an attention evaluation model;

[0051] The attention evaluation module is used to collect the attention EEG signals of the person to be evaluated, and evaluate the attention state of the person to be evaluated based on the attention evaluation model to obtain the attention evaluation result.

[0052] Preferably, the workflow of the time-domain feature extraction module includes:

[0053] The processed EEG signal was decomposed into time and frequency using Fast Fourier Transform to obtain the spectrum:

[0054]

[0055] Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index;

[0056] Define five EEG frequency bands: Delta, Theta, Alpha, Beta, and Gamma. For each EEG frequency band i, i∈{δ,θ,α,β,γ}, construct the corresponding frequency mask function:

[0057]

[0058] Where, m i (f) represents the frequency masking function, where f represents a continuous frequency variable. i,min f represents the lower cutoff frequency of the i-th frequency band. i,max This represents the upper cutoff frequency of the i-th frequency band;

[0059] Using the frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band:

[0060] X i [k] = X[k]·m i (f k )

[0061] Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s Indicates the sampling frequency;

[0062] Performing an inverse fast Fourier transform on the frequency domain representation yields the time-varying characteristics of the original EEG signal:

[0063]

[0064] Where, x i [n] represents the time-varying characteristics of the original EEG signal in the i-th frequency band;

[0065] The time-varying features within each frequency band are weighted and concatenated to obtain the weighted time-domain signal:

[0066]

[0067] in, This represents the weighted time-domain signal. This represents the attention feature representation, z i α represents the original weight value of frequency band i. i This represents the learnable weights of frequency band i, and concat represents the concatenation operation.

[0068] Preferably, the workflow of the multi-channel fusion module includes:

[0069] The power spectral density of the processed EEG signal was estimated using the Welch method to obtain the EEG power spectral density value:

[0070]

[0071] Among them, P c (f) represents the brainwave power spectral density value. This represents the signal segment within the k-th window function, where K represents the number of window functions and f represents the frequency.

[0072] Using the standard coordinate mapping function of electrodes on the scalp The EEG power spectral density values ​​P of each electrode c (f) Mapping onto a two-dimensional plane yields the topographic map matrix M corresponding to each frequency band. i ;

[0073] The obtained topographic map matrix M i Convert to grayscale and map to grayscale image I. i and the grayscale image I i Resize the image to a uniform size of H×W;

[0074] The adjusted grayscale images are stitched together along the channel dimension to obtain the multi-channel fused image:

[0075] I = concat(I) δ I θ I α I β I γ )

[0076] Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents the adjusted grayscale image of the Gamma EEG band.

[0077] Preferably, the parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer;

[0078] The video feature extraction branch based on AFC-EEGNET consists of a two-dimensional convolutional layer, a deep convolutional layer, a separable convolutional layer, a flattening layer, and a fully connected layer.

[0079] The ResNet-KAN-based spatial-frequency domain feature extraction branch consists of a ResNet18 residual structure and a KAN network.

[0080] The fusion layer is:

[0081]

[0082] in, This indicates the classification score after fusion. This represents the output score of the video feature extraction branch based on AFC-EEGNET. λ represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN, and λ represents the adjustable fusion coefficient.

[0083] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0084] (1) This invention effectively enhances the multidimensional representation ability of EEG signals and significantly optimizes classification performance. The proposed multi-level attention assessment method can maintain high-precision recognition ability with low computational complexity, and is suitable for real-time monitoring and embedded system applications. In addition, this invention performs well in terms of generalization ability and robustness, and can adapt to changes in attention state in different individuals and environments.

[0085] (2) The design of this invention enables the model to maintain low computational complexity and memory consumption during real-time dynamic evaluation, making it suitable for embedded systems and real-time monitoring tasks. At the same time, the method has strong generalization ability and can adapt to changes in attention state in different individuals and environments, exhibiting strong adaptability and robustness. This invention provides a novel solution for EEG attention assessment and has broad application prospects in fields such as intelligent interaction, educational learning monitoring, mental health testing, and security monitoring. It can provide accurate data support for personalized intervention and intelligent system decision-making. Attached Figure Description

[0086] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0088] Figure 2 This is a schematic diagram of the time-frequency feature extraction branch based on AFC-EEGNET in an embodiment of the present invention;

[0089] Figure 3 This is a schematic diagram of the spatial-frequency domain feature extraction branch based on ResNet-KAN in an embodiment of the present invention;

[0090] Figure 4 This is a schematic diagram of the three-class classification results of the experimental data subjected to five-fold cross-validation in an embodiment of the present invention;

[0091] Figure 5 This is a schematic diagram of the six-class classification results of the experimental data subjected to five-fold cross-validation in an embodiment of the present invention;

[0092] Figure 6 This is a schematic diagram of the nine-category classification results obtained by performing five-fold cross-validation on experimental data in an embodiment of the present invention. Detailed Implementation

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0094] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0095] Example 1

[0096] In this embodiment, as Figure 1 As shown, a multi-domain fusion method for evaluating attention based on EEG data includes the following steps:

[0097] S1. Collect the EEG signals of the test subjects' attention using an EEG signal acquisition device, and perform filtering to obtain the processed EEG signals. Then, classify the attention states based on the processed EEG signals to obtain the state classification results.

[0098] In this embodiment, an experimental scheme based on the "Airplane Battle" game is adopted. Firstly, in terms of experimental task design, a virtual battlefield environment is constructed using the "Airplane Battle" game. Within a fixed 60-second game duration, the player's attention-distance value is recorded every second to reflect the player's real-time response to the game situation. Simultaneously, multiple difficulty levels are set in the game, and the task pressure is adjusted by increasing the descent speed of enemy planes, thereby inducing different attention states. For data acquisition, this embodiment uses an 8-channel EEG signal acquisition device with a sampling frequency of 500Hz to ensure high-resolution data capture.

[0099] After acquiring attentional EEG signals, the signals were filtered and downsampled to 128Hz. Artifacts in the 0-60Hz range were then removed to ensure data validity and signal clarity. Next, the EEG data was divided into 2-second windows, and data was saved individually for each participant to obtain state classification results. In the experiment, attentional states were labeled based on player behavior during gameplay (such as attention-distance values, shooting accuracy, number of errors, etc.) combined with changes in EEG signals. The labeling criteria divided players' attentional states into three main states: focused, fatigued, and relaxed. Each state was further subdivided into three levels (such as mild focus, deep focus, etc.) based on the attention-distance value, resulting in nine states in total. This labeling scheme maintains structural consistency with traditional task grading methods but is more suitable for real-time interactive scenarios, offering a higher degree of granularity.

[0100] To address the risks of privacy breaches and misuse of behavioral data during game trials, this embodiment strictly adheres to privacy protection protocols during data collection and processing, ensuring that all collected data is anonymized and encrypted before being used for model training. Furthermore, by designing standardized game task parameters, interference from individual emotional fluctuations or non-attentional factors on data annotation is reduced, thereby ensuring the objectivity, consistency, and repeatability of the annotation process.

[0101] S2. The processed EEG signal is decomposed into time and frequency using Fast Fourier Transform to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and spliced ​​using the frequency domain attention mechanism to obtain the weighted time domain signal.

[0102] S2 includes: performing time-frequency decomposition on the processed EEG signal using Fast Fourier Transform to obtain the spectrum:

[0103]

[0104] Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index, ranging from 0 to N / 2, with each k corresponding to a discrete frequency; five EEG frequency bands are set: Delta (1-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz), and Gamma (30-64Hz). For each EEG frequency band i, i∈{δ,θ,α,β,γ}, the corresponding frequency mask function is constructed:

[0105]

[0106] Where, m i (f) represents the frequency masking function; f represents a continuous frequency variable, with the unit being Hertz (Hz); f i,min f represents the lower cutoff frequency (lowest boundary) of the i-th frequency band; i,max This represents the upper cutoff frequency (highest boundary) of the i-th frequency band; using a frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band:

[0107] X i [k] = X[k]·m i (f k )

[0108] Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s The sampling frequency is represented; the inverse fast Fourier transform is performed on the frequency domain representation to obtain the time-varying characteristics of the original EEG signal:

[0109]

[0110] Where, x i [n] represents the time-varying features of the original EEG signal in the i-th frequency band; to further enhance the extraction of features in each frequency band, a frequency domain attention mechanism is introduced, in which a learnable weight α is assigned to each frequency band i. i The weight parameters are obtained adaptively through training and normalized using the softmax function, and are defined as follows:

[0111]

[0112] Among them, z i α represents the original weight value of frequency band i. i Represents the learnable weights of frequency band i; assign each frequency band signal x i [n] and the corresponding weight α i Multiplying them together yields the weighted time-domain signal:

[0113]

[0114] in, This represents the weighted time-domain signal; finally, the weighted signals are fused, which can be done by concatenation (ultimately done in the code) to construct the overall attention feature representation:

[0115]

[0116] in, This represents the attention feature representation, and concat represents the concatenation operation; thus providing subsequent deep learning models with multi-dimensional inputs that have higher recognition capabilities.

[0117] S3. The power spectral density of the processed EEG signal is estimated using the Welch method to obtain the EEG power spectral density value. The EEG power spectral density value is mapped to a topographic map matrix, and the topographic map matrix of all frequency bands is grayscaled and resized. The adjusted grayscale images are then stitched together to obtain a multi-channel fused image.

[0118] S3 includes: estimating the power spectral density of the processed EEG signal using the Welch method to obtain the EEG power spectral density value.

[0119]

[0120] Among them, P c (f) represents the brainwave power spectral density value. Let f represent the signal segment within the m-th window function, M represent the number of window functions, and f represent the frequency. Next, based on predefined EEG frequency bands (Delta, Theta, Alpha, Beta1, Beta2, Gamma), the PSD values ​​of each electrode in each frequency band are summarized to obtain the energy distribution of each electrode in each frequency band. This is achieved using a standard coordinate mapping function of the electrodes on the scalp. P c (f) — The energy distribution of each frequency band is mapped onto a two-dimensional plane to obtain the topographic map matrix M corresponding to each frequency band. i ; Matrix M i Each element M in i (x,y) represents the energy value of frequency band i at location (x,y). An interpolation method is used to smoothly map the discrete electrode data onto a continuous two-dimensional grid. The resulting topographic map matrix M... i Convert to grayscale and map to grayscale image I. i and for grayscale image I iResize the image to a uniform size of H×W; then stitch the resized grayscale image along the channel dimension to obtain a multi-channel fused image.

[0121] I = concat(I) δ I θ I α I β I γ )

[0122] Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents an adjusted grayscale image of the Gamma EEG band. This frequency domain-topological feature fusion method not only intuitively expresses the energy distribution information of each frequency band in image form, but also enriches the feature representation of EEG signals through the embedding of spatial information, providing more discriminative multi-dimensional inputs for subsequent classification or regression tasks.

[0123] S4. Construct a parallel model structure and train the parallel model structure based on the weighted time-domain signal, multi-channel fused image and state partitioning results to obtain the attention evaluation model.

[0124] The parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer.

[0125] The video feature extraction branch based on AFC-EEGNET consists of two-dimensional convolutional layers, depthwise convolutional layers, separable convolutional layers, flattening layers, and fully connected layers, such as... Figure 2 As shown. In this embodiment, the input preprocessed EEG signal X undergoes a series of convolution operations to extract time-frequency features. First, a two-dimensional convolutional layer (Conv2D) is used to perform preliminary feature extraction on the input signal, the operation of which can be described as follows:

[0126] Y (1) =σ(W (1) *X+b (1) )

[0127] Among them, W (1) b represents the convolution kernel of the Conv2D layer. (1)Let represent the bias term, and σ(·) represent the activation function (e.g., ReLU). Next, a depthwise convolutional layer (DepthwiseConv2D) is used, which performs convolution independently for each input channel, expressed as:

[0128]

[0129] in, This represents the input feature of the c-th channel. This represents the convolution kernel for the corresponding channel. This represents the bias term for the corresponding channel. Subsequently, spatial and channel information are further separated using a separable convolutional layer (SeparableConv2D). This process can be divided into two steps: depthwise convolution and pointwise convolution, and can be represented as follows:

[0130]

[0131] in, This represents a convolution kernel for pointwise convolution, used to achieve channel fusion while maintaining spatial invariance. Y represents the kernel of a depthwise convolution, a kernel used independently for each channel, i.e., one kernel per input channel. (2) b represents the output feature of the previous layer. (3) This represents the trainable bias term. After the above layers, the extracted features are flattened by a flattening layer, and then the classification score is output through a fully connected layer (FC). This process can be represented as:

[0132]

[0133] in, W represents the output score of the video feature extraction branch based on AFC-EEGNET. (4) Y represents the weight matrix of the fully connected layer. (3) b represents the output feature of the previous layer. (4) This represents the bias vector of the fully connected layer.

[0134] The spatial-frequency feature extraction branch based on ResNet-KAN consists of a ResNet18 residual structure and a KAN network, such as Figure 3 As shown. In the parallel second branch, a ResNet-KAN network is used to extract spatial-frequency domain features. This branch is based on the residual structure of ResNet18, where the basic form of each residual module is:

[0135]

[0136] Where y represents the output of the module, which is the element-wise sum of the residual mapping and the input features. This represents the residual mapping after multiple convolutions, normalization, and activation function combinations, where x represents the input features of the module. The difference lies in the sequential replacement of the traditional ResNet with the KAN (Kolmogorov-Arnold Network). The KAN network is based on the Kolmogorov-Arnold representation theorem, which states that any continuous multivariable function can be represented as a combination of a series of univariate functions, mathematically expressed as:

[0137]

[0138] Where z represents the input feature vector, z p Let ψ represent the p-th component. q,p (·) and Φ q (·) denotes a learnable univariate function (typically implemented by a small neural network), d represents the input dimension, and Q represents the number of combinations. This substitution makes the network more flexible in capturing nonlinear EEG signal features, thus outputting a classification score in the space-frequency domain branch:

[0139]

[0140] Among them, Y ResNet This represents the feature map after partial processing by ResNet. This represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN.

[0141] The fusion layer is:

[0142]

[0143] in, The fusion score represents the classification score after fusion, and λ represents the adjustable fusion coefficient. Through the above fusion method, the model combines time-frequency features with spatial-frequency domain features, achieving comprehensive extraction of multi-level, nonlinear features of EEG signals, thereby significantly improving the accuracy and robustness of the final classification results.

[0144] During the model training phase, 100 rounds of training were employed, and five-fold cross-validation was used to ensure the model's stability and generalization performance. Figure 4 , Figure 5 and Figure 6 As shown in the figure, the experimental results for different classification tasks are as follows: for the three-class classification task, the average accuracy reaches 0.9193; for the six-class classification task (refined two-level segmentation), the accuracy is 0.7133; and for the nine-class classification task, the average accuracy is 0.5311. These results indicate that as the classification detail increases, the model faces greater challenges, but it has achieved high recognition accuracy in low to medium-level tasks.

[0145] This output can be used to monitor individual attention changes in real time, thereby providing accurate attention assessments for fields such as intelligent interaction, education, and mental health. At the brain-computer interface (BCI) level, this solution can apply attention state assessment results to control systems, such as automatically adjusting task difficulty during long-term tasks, providing real-time rest suggestions, or controlling assistive devices, thereby achieving personalized attention regulation. By feeding real-time dynamic attention state information back into the system, not only can the user experience be improved, but the risk of fatigue caused by long-term tasks can also be significantly reduced, optimizing overall task performance.

[0146] S5. Collect the attention EEG signals of the person to be evaluated, and evaluate the attention state of the person to be evaluated based on the attention evaluation model to obtain the attention evaluation results.

[0147] Example 2

[0148] In this embodiment, a multi-domain fusion-based ultra-multi-level attention evaluation system based on EEG data includes: a signal acquisition module, a temporal feature extraction module, a multi-channel fusion module, a model construction module, and an attention evaluation module.

[0149] The signal acquisition module uses an EEG signal acquisition device to collect the EEG signals of the test subjects' attention, performs filtering processing to obtain processed EEG signals, and classifies the attention state based on the processed EEG signals to obtain the state classification results.

[0150] The time-domain feature extraction module uses Fast Fourier Transform to perform time-frequency decomposition on the processed EEG signal to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and concatenated using a frequency domain attention mechanism to obtain the weighted time domain signal.

[0151] The workflow of the time-domain feature extraction module includes: performing time-frequency decomposition on the processed EEG signal using Fast Fourier Transform to obtain the spectrum.

[0152]

[0153] Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index; five EEG frequency bands are set: Delta, Theta, Alpha, Beta, and Gamma. For each EEG frequency band i, i∈{δ,θ,α,β,γ}, the corresponding frequency mask function is constructed:

[0154]

[0155] Where, m i (f) represents the frequency masking function, where f represents a continuous frequency variable. i,minf represents the lower cutoff frequency of the i-th frequency band. i,max Let represent the upper cutoff frequency of the i-th frequency band; using a frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band:

[0156] X i [k] = X[k]·m i (f k )

[0157]

[0158] Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s The sampling frequency is represented; the inverse fast Fourier transform is performed on the frequency domain representation to obtain the time-varying characteristics of the original EEG signal:

[0159]

[0160] Where, x i [n] represents the time-varying features of the original EEG signal in the i-th frequency band; the time-varying features in each frequency band are weighted and concatenated to obtain the weighted time-domain signal:

[0161]

[0162]

[0163] in, This represents the weighted time-domain signal. This represents the attention feature representation, z i α represents the original weight value of frequency band i. i The learnable weights of frequency band i are represented by , and concat represents the splicing operation; the attention feature representation is then inversely transformed to obtain the time-domain features of the EEG signal.

[0164] The multi-channel fusion module uses the Welch method to estimate the power spectral density of the processed EEG signal, obtains the EEG power spectral density value, maps the EEG power spectral density value to a topographic map matrix, and grayscales and resizes the topographic map matrices of all frequency bands. Then, the adjusted grayscale images are stitched together to obtain a multi-channel fused image.

[0165] The workflow of the multi-channel fusion module includes: estimating the power spectral density of the processed EEG signal using the Welch method to obtain the EEG power spectral density value.

[0166]

[0167] Among them, P c(f) represents the brainwave power spectral density value. This represents the signal segment within the k-th window function, where K represents the number of window functions and f represents the frequency; the signal is mapped using the standard coordinates of the electrodes on the scalp. The EEG power spectral density values ​​P of each electrode c (f) Mapping onto a two-dimensional plane yields the topographic map matrix M corresponding to each frequency band. i ; for the obtained topographic map matrix M i Convert to grayscale and map to grayscale image I. i and for grayscale image I i Resize the image to a uniform size of H×W; then stitch the resized grayscale image along the channel dimension to obtain a multi-channel fused image.

[0168] I = concat(I) δ I θ I α I β I γ )

[0169] Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents the adjusted grayscale image of the Gamma EEG band.

[0170] The model building module is used to construct a parallel model structure and train the parallel model structure based on the weighted time-domain signal, multi-channel fused image and state partitioning results to obtain the attention evaluation model.

[0171] The parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer; the video feature extraction branch based on AFC-EEGNET consists of 2D convolutional layers, deep convolutional layers, separable convolutional layers, flattening layers, and fully connected layers; the spatial-frequency domain feature extraction branch based on ResNet-KAN consists of a ResNet18 residual structure and a KAN network; the fusion layer is:

[0172]

[0173] in, This indicates the classification score after fusion. This represents the output score of the video feature extraction branch based on AFC-EEGNET. λ represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN, and λ represents the adjustable fusion coefficient.

[0174] The attention evaluation module is used to collect the attention EEG signals of the person being evaluated and to evaluate the attention state of the person being evaluated based on the attention evaluation model, so as to obtain the attention evaluation results.

[0175] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-domain fusion method for evaluating attention based on electroencephalogram (EEG) data, characterized in that, Includes the following steps: S1. Collect the attention EEG signals of the test personnel using an EEG signal acquisition device, and perform filtering processing to obtain the processed EEG signals. Then, classify the attention state based on the processed EEG signals to obtain the state classification results. S2. The processed EEG signal is decomposed into time and frequency using Fast Fourier Transform to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and spliced ​​using a frequency domain attention mechanism to obtain a weighted time domain signal. S3. The power spectral density of the processed EEG signal is estimated using the Welch method to obtain the EEG power spectral density value. The EEG power spectral density value is mapped to a topographic map matrix. The topographic map matrix of all frequency bands is grayscaled and resized. The adjusted grayscale images are then stitched together to obtain a multi-channel fused image. S4. Construct a parallel model structure, and train the parallel model structure based on the weighted time-domain signal, the multi-channel fused image, and the state partitioning result to obtain an attention evaluation model; S5. Collect the attention EEG signals of the person to be evaluated, and evaluate the attention state of the person to be evaluated based on the attention evaluation model to obtain the attention evaluation result.

2. The method for evaluating multi-level attention based on multi-domain fusion of EEG data according to claim 1, characterized in that, S2 includes: The processed EEG signal was decomposed into time and frequency using Fast Fourier Transform to obtain the spectrum: Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index; Define five EEG frequency bands: Delta, Theta, Alpha, Beta, and Gamma. For each EEG frequency band i, i∈{δ,θ,α,β,γ}, construct the corresponding frequency mask function: Where, m i (f) represents the frequency masking function, where f represents a continuous frequency variable. i,min f represents the lower cutoff frequency of the i-th frequency band. i,max This represents the upper cutoff frequency of the i-th frequency band; Using the frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band: X i [k]=X[k]·m i (f k ) Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s Indicates the sampling frequency; Performing an inverse fast Fourier transform on the frequency domain representation yields the time-varying characteristics of the original EEG signal: Where, x i [n] represents the time-varying characteristics of the original EEG signal in the i-th frequency band; The time-varying features within each frequency band are weighted and concatenated to obtain the weighted time-domain signal: in, This represents the weighted time-domain signal. This represents the attention feature representation, z i α represents the original weight value of frequency band i. i This represents the learnable weights of frequency band i, and concat represents the concatenation operation.

3. The method for evaluating multi-level attention based on multi-domain fusion of EEG data according to claim 2, characterized in that, S3 includes: The power spectral density of the processed EEG signal was estimated using the Welch method to obtain the EEG power spectral density value: Among them, P c (f) represents the brainwave power spectral density value. This represents the signal segment within the k-th window function, where K represents the number of window functions and f represents the frequency. Using the standard coordinate mapping function of electrodes on the scalp The EEG power spectral density values ​​P of each electrode c (f) Mapping onto a two-dimensional plane yields the topographic map matrix M corresponding to each frequency band. i ; The obtained topographic map matrix M i Convert to grayscale and map to grayscale image I. i and the grayscale image I i Resize the image to a uniform size of H×W; The adjusted grayscale images are stitched together along the channel dimension to obtain the multi-channel fused image: I=concat(I δ ,I θ ,I α ,I β ,I γ ) Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents the adjusted grayscale image of the Gamma EEG band.

4. The method for evaluating multi-level attention based on multi-domain fusion of EEG data according to claim 1, characterized in that, The parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer; The video feature extraction branch based on AFC-EEGNET consists of a two-dimensional convolutional layer, a deep convolutional layer, a separable convolutional layer, a flattening layer, and a fully connected layer. The ResNet-KAN-based spatial-frequency domain feature extraction branch consists of a ResNet18 residual structure and a KAN network. The fusion layer is: in, This indicates the classification score after fusion. This represents the output score of the video feature extraction branch based on AFC-EEGNET. λ represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN, and λ represents the adjustable fusion coefficient.

5. A multi-domain fusion-based ultra-multilevel attention evaluation system based on electroencephalogram (EEG) data, wherein the evaluation system applies the evaluation method according to any one of claims 1-4, characterized in that, include: The module includes a signal acquisition module, a time-domain feature extraction module, a multi-channel fusion module, a model building module, and an attention evaluation module. The signal acquisition module uses an EEG signal acquisition device to acquire the EEG signals of the test personnel's attention, performs filtering processing to obtain processed EEG signals, and performs attention state classification based on the processed EEG signals to obtain state classification results. The time-domain feature extraction module uses Fast Fourier Transform to perform time-frequency decomposition on the processed EEG signal to obtain the frequency domain representation of the EEG signal. Then, the frequency domain representation is weighted and concatenated using a frequency domain attention mechanism to obtain a weighted time-domain signal. The multi-channel fusion module uses the Welch method to estimate the power spectral density of the processed EEG signal to obtain the EEG power spectral density value. The EEG power spectral density value is mapped to a topographic map matrix, and the topographic map matrix of all frequency bands is grayscaled and resized. The adjusted grayscale images are then stitched together to obtain a multi-channel fused image. The model building module is used to build a parallel model structure and train the parallel model structure based on the weighted time-domain signal, the multi-channel fused image and the state partitioning result to obtain an attention evaluation model; The attention evaluation module is used to collect the attention EEG signals of the person to be evaluated, and evaluate the attention state of the person to be evaluated based on the attention evaluation model to obtain the attention evaluation result.

6. The ultra-multi-level attention evaluation system based on multi-domain fusion of EEG data according to claim 5, characterized in that, The workflow of the temporal feature extraction module includes: The processed EEG signal was decomposed into time and frequency using Fast Fourier Transform to obtain the spectrum: Where X[k] represents the spectrum, N represents the number of sampling points of the EEG signal x[n] on the time axis, and k represents the frequency index; Define five EEG frequency bands: Delta, Theta, Alpha, Beta, and Gamma. For each EEG frequency band i, i∈{δ,θ,α,β,γ}, construct the corresponding frequency mask function: Where, m i (f) represents the frequency masking function, where f represents a continuous frequency variable. i,min f represents the lower cutoff frequency of the i-th frequency band. i,max This represents the upper cutoff frequency of the i-th frequency band; Using the frequency mask function, the spectrum is filtered to obtain the frequency domain representation of each frequency band: X i [k]=X[k]·m i (f k ) Among them, X i [k] represents the frequency domain of each frequency band, f k f represents the actual frequency corresponding to the k-th frequency component. s Indicates the sampling frequency; Performing an inverse fast Fourier transform on the frequency domain representation yields the time-varying characteristics of the original EEG signal: Where, x i [n] represents the time-varying characteristics of the original EEG signal in the i-th frequency band; The time-varying features within each frequency band are weighted and concatenated to obtain the weighted time-domain signal: in, This represents the weighted time-domain signal. This represents the attention feature representation, z i α represents the original weight value of frequency band i. i This represents the learnable weights of frequency band i, and concat represents the concatenation operation.

7. The ultra-multi-level attention evaluation system based on multi-domain fusion of EEG data according to claim 6, characterized in that, The workflow of the multi-channel fusion module includes: The power spectral density of the processed EEG signal was estimated using the Welch method to obtain the EEG power spectral density value: Among them, P c (f) represents the brainwave power spectral density value. This represents the signal segment within the k-th window function, where K represents the number of window functions and f represents the frequency. Using the standard coordinate mapping function of electrodes on the scalp The EEG power spectral density values ​​P of each electrode c (f) Mapping onto a two-dimensional plane yields the topographic map matrix M corresponding to each frequency band. i ; The obtained topographic map matrix M i Convert to grayscale and map to grayscale image I. i and the grayscale image I i Resize the image to a uniform size of H×W; The adjusted grayscale images are stitched together along the channel dimension to obtain the multi-channel fused image: I=concat(I δ ,I θ ,I α ,I β ,I γ ) Where I represents a multi-channel fused image, I δ Represents the adjusted grayscale image of the Delta EEG band, I θ Represents the adjusted grayscale image of the Theta EEG band, I α I represents the adjusted grayscale image of the Alpha EEG band. β I represents the adjusted grayscale image of the Beta EEG band. γ This represents the adjusted grayscale image of the Gamma EEG band.

8. The ultra-multi-level attention evaluation system based on multi-domain fusion of EEG data according to claim 5, characterized in that, The parallel model structure includes: a video feature extraction branch based on AFC-EEGNET, a spatial-frequency domain feature extraction branch based on ResNet-KAN, and a fusion layer; The video feature extraction branch based on AFC-EEGNET consists of a two-dimensional convolutional layer, a deep convolutional layer, a separable convolutional layer, a flattening layer, and a fully connected layer. The ResNet-KAN-based spatial-frequency domain feature extraction branch consists of a ResNet18 residual structure and a KAN network. The fusion layer is: in, This indicates the classification score after fusion. This represents the output score of the video feature extraction branch based on AFC-EEGNET. λ represents the output score of the spatial-frequency domain feature extraction branch based on ResNet-KAN, and λ represents the adjustable fusion coefficient.

Images