Multi-modal neural feedback training method for improving attention
Through the multimodal neural feedback training method, the convolutional neural network is used to identify and adjust the game scene, the problem of insufficient single-channel signal information is solved, high-precision attention recognition and real-time feedback are achieved, and the effect of attention training is improved.
Patent Information
- Application Number
- CN202411363924.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-28
- Publication Date
- 2025-01-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing EEG signal feedback system has shortcomings in accuracy and versatility, especially the lack of single-channel signal information, and the inconvenient wearing of multi-channel devices and poor comfort, which affects patient compliance.
Multimodal neural feedback training method is adopted to obtain the original EEG signal and ECG signals of the subjects during serious games, preprocess and convolutional neural network classification, identify the attention state, and adjust the game scene and the Background music forms a closed-loop feedback system.
High-precision attention recognition and real-time feedback are achieved, the subject's adjustment and control ability of attention state is improved, and the effectiveness and compliance of training are enhanced.
Smart Images
Figure CN119235311A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of attention training, and more specifically, to a multimodal neural feedback training method for improving attention. Background Art
[0002] Attention is crucial to our study and work. Biofeedback training is a medical technology that can improve cognitive function, regulate psychological reactions, and treat various behavioral disorders. It uses professional instruments to feed back imperceptible physiological signals to patients in the form of easy-to-understand numbers, sounds or images, so that patients can learn to actively regulate and control their own physiological processes. Neurofeedback training is a subfield of biofeedback training. It uses brain imaging technologies such as electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy to detect and regulate brain activity to improve various cognitive functions including attention. Among many brain imaging technologies, electroencephalography is widely used due to its advantages such as low cost, non-invasiveness, and high temporal resolution. Many studies have shown that electroencephalogram-based neurofeedback training can not only improve the attention of healthy individuals, but is also widely used in the treatment of attention deficit hyperactivity disorder (ADHD), autism, anxiety, depression and other diseases. Compared with drug treatment, this non-invasive digital therapy has fewer side effects and better patient compliance.
[0003] For EEG signals, they are usually divided into five bands according to the frequency range: delta (0.5-4Hz), theta (4-8Hz), alpha (8-14Hz), beta (14-30Hz) and gamma (30-64Hz). Related studies have found that changes in attention levels can cause changes in the activity of these brain waves. Therefore, in many neurofeedback systems, the power or energy ratio of these brain waves is used as a feedback indicator and a basis for setting feedback thresholds. However, due to individual differences in EEG signals, this method is not accurate and has low versatility. In addition, many neurofeedback systems use multi-channel EEG devices to collect patients' EEG signals, which brings two problems: first, multi-channel EEG devices require cumbersome preparation operations before wearing, which is inconvenient to use; second, too many electrodes will affect the comfort when wearing, which can easily reduce the compliance of patients (especially children). Therefore, single-channel EEG devices are a more appropriate choice. However, compared with multi-channel signals, single-channel signals contain less information, especially lack of information in the spatial dimension. Therefore, it is necessary to use multimodal bioelectric signals to improve recognition accuracy. Summary of the invention
[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a multimodal neural feedback training method for improving attention, so as to solve the above technical problems.
[0005] The above technical objectives of the present invention are achieved through the following technical solutions: A multimodal neurofeedback training method for improving attention, comprising:
[0006] S1. Obtaining the original EEG signals and original ECG signals of the subjects when they play serious games;
[0007] S2, preprocessing the original EEG signal and the original ECG signal to obtain a corresponding EEG signal and an ECG signal with baseline drift removed;
[0008] S3, classifying the brain wave signal and the ECG signal with baseline drift removed by a preset convolutional neural network model to obtain the subject's attention state;
[0009] S4. Adjust the scene of the serious game according to the attention state, and return to step S1.
[0010] Optionally, the obtaining of original EEG signals and original ECG signals of the subject when playing a serious game includes:
[0011] The real-time raw EEG signals and raw ECG signals of the subjects when they play serious games are collected through wearable bioelectric signal acquisition equipment.
[0012] Optionally, the preprocessing of the original EEG signal and the original ECG signal to obtain the corresponding EEG signal and the ECG signal with baseline drift removed includes:
[0013] The collected original EEG signals and original ECG signals are preprocessed using a wavelet packet decomposition algorithm, the EEG signals of the corresponding bands are extracted from the original EEG signals, and the baseline drift in the original ECG signals is removed to obtain ECG signals with baseline drift removed.
[0014] Optionally, extracting the brain wave signal of the corresponding band from the original brain wave signal includes:
[0015] Extract the brain wave signal of the corresponding band according to the following formula:
[0016]
[0017] Among them, S EEG is the input raw EEG signal, wpdec is the wavelet packet decomposition function used to decompose the input raw EEG signal, and wpt EEGis the wavelet packet tree object corresponding to the 9th level decomposition of the input original EEG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal; base line 1 is the baseline signal of node 0 of the 9th layer with a frequency range of 0 to 0.5 Hz when the sampling rate of the signal acquisition device is 512 Hz;
[90] represents node 0 of the 9th layer,
[60] represents node 0 of the 6th layer,
[61] represents node 1 of the 6th layer,
[62] represents node 2 of the 6th layer,
[76] represents node 6 of the 7th layer,
[41] represents node 1 of the 4th layer,
[77] represents node 7 of the 7th layer,
[715] represents node 15 of the 7th layer, and
[31] represents node 1 of the 3rd layer; S delta , S theta , S alpha , S beta and S gamma is the brain wave signal of the corresponding band to be extracted.
[0018] Optionally, removing the baseline drift in the original electrocardiogram signal to obtain the electrocardiogram signal with the baseline drift removed includes:
[0019] The baseline drift in the ECG signal is removed according to the following formula to obtain the ECG signal with the baseline drift removed:
[0020]
[0021] Among them, S ECG is the original ECG signal input, wpdec is the wavelet packet decomposition function used to decompose the original ECG signal input, and wpt EEG is the wavelet packet tree object corresponding to the 9th level decomposition of the input original ECG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal, base line 2 is the baseline drift with a frequency range of 0 to 0.5 Hz, and Y ECG It is the ECG signal with baseline drift removed.
[0022] Optionally, the preset convolutional neural network model includes: three convolutional layers, three maximum pooling layers, two fully connected layers and one Dropout layer; the convolutional layers are used to extract features from input data, the convolution kernel size of all convolutional layers is 3*3, and ReLU is used as the activation function;
[0023] The role of the pooling layer is to reduce parameters while retaining the main features. The size of all maximum pooling layers is 2*2;
[0024] The fully connected layer maps the features obtained from the convolutional layer and the pooling layer to the sample label space, playing the role of a classifier;
[0025] The Dropout layer can randomly select some nodes to participate in model prediction to prevent overfitting, and the Dropout rate is set to 0.2.
[0026] Optionally, the brain wave signal and the ECG signal with baseline drift removed are classified by a preset convolutional neural network model to obtain the attention state of the subject, including:
[0027] The output layer is a fully connected layer, which outputs the probabilities of the subject's low attention state, medium attention state, and high attention state, and takes the attention state with the highest probability as the subject's attention state.
[0028] Optionally, the convolutional neural network model learns and optimizes network parameters through a back propagation algorithm; Adam is used as the optimizer, and the learning rate is set to 0.0001; the loss function is a cross entropy loss function; the batch size is set to 32, and the epoch is set to 100.
[0029] Optionally, the convolutional neural network model uses precision (PR), recall rate (RE), F1-score and accuracy rate (ACC) as evaluation indicators of model performance, which are defined as follows:
[0030]
[0031] Among them, TP represents the number of positive samples correctly identified as positive samples by the model; FP represents the number of negative samples incorrectly identified as positive samples by the model; TN represents the number of negative samples correctly identified as negative samples by the model; and FN represents the number of positive samples incorrectly identified as negative samples by the model.
[0032] Optionally, adjusting the scene of the serious game according to the attention state includes:
[0033] When the attention state is a low attention state, adjusting the scene of the serious game to the first game scene, and adjusting the background music to the first background music;
[0034] When the attention state is a medium attention state, adjusting the scene of the serious game to a second game scene, and adjusting the background music to a second background music;
[0035] When the attention state is a high attention state, the scene of the serious game is adjusted to the third game scene, and the background music is adjusted to the third background music.
[0036] In summary, the present invention has the following beneficial effects:
[0037] A convolutional neural network model was established that can achieve high-precision recognition of attention during neurofeedback training, and output the subject's real-time attention state based on the subject's real-time raw EEG signals and raw ECG signals;
[0038] Output the subject's real-time attention state into the serious game, and prompt the subject to improve or maintain the current attention state by changing the scene and background music of the serious game;
[0039] Through a closed-loop multimodal neurofeedback training system, subjects can actively adjust and control their attention state, thereby achieving the goal of improving attention. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flow chart of the method of the present invention;
[0041] Figure 2 It is a nine-level wavelet packet decomposition structure diagram in the present invention;
[0042] Figure 3 It is a structural diagram of a convolutional neural network model used for attention recognition in the present invention;
[0043] Figure 4 It is a schematic diagram of the closed-loop neural feedback training system in the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.
[0045] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0046] In the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Moreover, the first feature being "above", "above" and "above" the second feature includes the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature includes the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature. The terms "vertical", "horizontal", "left", "right", "above", "below" and similar expressions are for illustrative purposes only, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0047] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.
[0048] The present invention provides a multimodal neural feedback training method for improving attention, such as Figure 1 and Figure 2 As shown, including:
[0049] S1. Obtaining the original EEG signals and original ECG signals of the subjects when they play serious games;
[0050] S2, preprocessing the original EEG signal and the original ECG signal to obtain a corresponding EEG signal and an ECG signal with baseline drift removed;
[0051] S3, classifying the brain wave signal and the ECG signal with baseline drift removed by a preset convolutional neural network model to obtain the subject's attention state;
[0052] S4. Adjust the scene of the serious game according to the attention state, and return to step S1.
[0053] Specifically, an experiment is first designed to collect original single-channel EEG and ECG signals under three attention states. The collected signals will be used to train the attention recognition model. During the experiment, tasks of different difficulty levels are used to induce different attention states of the subjects, including the Stroop color interference test, Schulte table, and mental arithmetic. The subjects collect their EEG and ECG signals synchronously while performing the tasks. After completing each task, the subjects need to use the attention self-rating scale to report the level of attention during the task, and use this as a data label.
[0054] Then, a serious game that can be installed on a smartphone was developed based on the Cocos open source game engine as a feedback tool. The subjects were asked to play the serious game, and the original EEG signals and original ECG signals of the subjects were obtained when they played the serious game. The original EEG signals and original ECG signals were then preprocessed to remove low-frequency noise and baseline drift in the original EEG signals and original ECG signals.
[0055] Subsequently, the brain wave signals and the ECG signals with baseline drift removed are classified through the trained convolutional neural network model. The classification results are sent to the game through the Bluetooth module and displayed in real time in the upper right corner of the screen.
[0056] The serious game adjusts the scene according to the classification results to remind the subject of the current attention level and make the subject improve or maintain the attention level. The real-time changes of the subject's attention level are converted into changes in the serious game scene. The subject constantly adjusts his attention during the training process to ensure the completion of the serious game.
[0057] Since the scenes of serious games change in real time according to the subject's attention state, based on the brain's neuroplasticity and self-regulation mechanism, the subjects can learn to actively adjust and control their attention state through this closed-loop feedback method, thereby achieving the purpose of improving attention.
[0058] Optionally, the obtaining of original EEG signals and original ECG signals of the subject when playing a serious game includes:
[0059] The real-time raw EEG signals and raw ECG signals of the subjects when they play serious games are collected through wearable bioelectric signal acquisition equipment.
[0060] Specifically, wearable bioelectric signal acquisition equipment can easily obtain the original EEG signals and original ECG signals of subjects when they are playing serious games.
[0061] Optionally, the preprocessing of the original EEG signal and the original ECG signal to obtain the corresponding EEG signal and the ECG signal with baseline drift removed includes:
[0062] The collected original EEG signals and original ECG signals are preprocessed using a wavelet packet decomposition algorithm, the EEG signals of the corresponding bands are extracted from the original EEG signals, and the baseline drift in the original ECG signals is removed to obtain ECG signals with baseline drift removed.
[0063] The wavelet packet decomposition algorithm can extract the EEG signal of the corresponding band from the collected original EEG signal, and remove the baseline drift in the original ECG signal to obtain the ECG signal with the baseline drift removed, thereby improving the accuracy of the EEG signal and ECG signal of the corresponding band.
[0064] Optionally, extracting the brain wave signal of the corresponding band from the original brain wave signal includes:
[0065] Extract the brain wave signal of the corresponding band according to the following formula:
[0066]
[0067] Among them, S EEG is the input raw EEG signal, wpdec is the wavelet packet decomposition function used to decompose the input raw EEG signal, and wpt EEG is the wavelet packet tree object corresponding to the 9th level decomposition of the input original EEG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal; base line 1 is the baseline signal of node 0 of the 9th layer with a frequency range of 0 to 0.5 Hz when the sampling rate of the signal acquisition device is 512 Hz;
[90] represents node 0 of the 9th layer,
[60] represents node 0 of the 6th layer,
[61] represents node 1 of the 6th layer,
[62] represents node 2 of the 6th layer,
[76] represents node 6 of the 7th layer,
[41] represents node 1 of the 4th layer,
[77] represents node 7 of the 7th layer,
[715] represents node 15 of the 7th layer, and
[31] represents node 1 of the 3rd layer; S delta , S theta , S alpha , S beta and S gamma is the brain wave signal of the corresponding band to be extracted.
[0068] The purpose of removing the baseline signal is to filter out any low frequency noise that may be present.
[0069] Optionally, removing the baseline drift in the original electrocardiogram signal to obtain the electrocardiogram signal with the baseline drift removed includes:
[0070] The baseline drift in the ECG signal is removed according to the following formula to obtain the ECG signal with the baseline drift removed:
[0071]
[0072] Among them, S ECG is the original ECG signal input, wpdec is the wavelet packet decomposition function used to decompose the original ECG signal input, and wpt EEG is the wavelet packet tree object corresponding to the 9th level decomposition of the input original ECG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal, base line 2 is the baseline drift with a frequency range of 0 to 0.5 Hz, and Y ECG It is the ECG signal with baseline drift removed.
[0073] Optionally, the preset convolutional neural network model includes: three convolutional layers, three maximum pooling layers, two fully connected layers and one Dropout layer; the convolutional layers are used to extract features from input data, the convolution kernel size of all convolutional layers is 3*3, and ReLU is used as the activation function;
[0074] The role of the pooling layer is to reduce parameters while retaining the main features. The size of all maximum pooling layers is 2*2;
[0075] The fully connected layer maps the features obtained from the convolutional layer and the pooling layer to the sample label space, playing the role of a classifier;
[0076] The Dropout layer can randomly select some nodes to participate in model prediction to prevent overfitting, and the Dropout rate is set to 0.2.
[0077] In order to accurately identify the real-time attention state of individuals during neurofeedback training, a lightweight convolutional neural network model was constructed. Multimodal bioelectric signals consisting of five brain waves extracted from the original EEG signal and the ECG signal with baseline drift removed were used to train the model.
[0078] Optionally, the brain wave signal and the ECG signal with baseline drift removed are classified by a preset convolutional neural network model to obtain the attention state of the subject, including:
[0079] The output layer is a fully connected layer, which outputs the probabilities of the subject's low attention state, medium attention state, and high attention state, and takes the attention state with the highest probability as the subject's attention state.
[0080] Optionally, the convolutional neural network model learns and optimizes network parameters through a back propagation algorithm; Adam is used as the optimizer, and the learning rate is set to 0.0001; the loss function is a cross entropy loss function; the batch size is set to 32, and the epoch is set to 100.
[0081] Optionally, the convolutional neural network model uses precision (PR), recall rate (RE), F1-score and accuracy rate (ACC) as evaluation indicators of model performance, which are defined as follows:
[0082]
[0083] Among them, TP represents the number of positive samples correctly identified as positive samples by the model; FP represents the number of negative samples incorrectly identified as positive samples by the model; TN represents the number of negative samples correctly identified as negative samples by the model; and FN represents the number of positive samples incorrectly identified as negative samples by the model.
[0084] The leave-one-out cross-validation method is used to evaluate the recognition ability of the model. The detailed steps of the leave-one-out cross-validation method are: each subject's sample is used as the test set in turn, and the remaining subjects' samples are used as the training set. After verification, the average recognition accuracy of the constructed convolutional neural network model for attention is 95.81%. The detailed evaluation indicators are shown in the following table:
[0085]
[0086] Optionally, adjusting the scene of the serious game according to the attention state includes:
[0087] When the attention state is a low attention state, adjusting the scene of the serious game to the first game scene, and adjusting the background music to the first background music;
[0088] When the attention state is a medium attention state, adjusting the scene of the serious game to a second game scene, and adjusting the background music to a second background music;
[0089] When the attention state is a high attention state, the scene of the serious game is adjusted to the third game scene, and the background music is adjusted to the third background music.
[0090] In order to form a closed-loop neurofeedback training system, a serious game that can be installed on a smartphone was developed based on the Cocos open source game engine as a feedback tool. The virtual plane in the game is controlled by the player's attention level. The higher the attention level, the higher the flight altitude. If the player is always in a low attention state, the virtual plane will gradually fall. At this time, a red warning will appear on the screen, accompanied by fast-paced background music to remind the player to stay focused. As the player's attention level gradually increases, the background color of the screen will gradually change to blue (medium attention state) or green (high attention state), accompanied by slow-paced light music.
[0091] In this way, changes in the player's attention level are translated into changes in the game scene in real time. Players need to constantly adjust their attention during training to ensure that the virtual plane can successfully reach the destination.
[0092] Based on the brain's neuroplasticity and self-regulation mechanism, players can learn to actively adjust and control their attention state through this closed-loop feedback method, thereby achieving the goal of improving concentration.
[0093] In order to verify the effectiveness of the proposed multimodal neurofeedback training system, attention training was conducted on 6 college students (male: 5, female: 1, age: 21.50±0.55). All subjects had no severe cognitive impairment and had not received similar attention training. The training lasted for 3 weeks, 2 times a week, and each time lasted 30 minutes.
[0094] The audio-visual integrated continuous performance test was used to assess the subject's attention level. This test has been widely used in the diagnosis of ADHD and the assessment of attention level. Before the test begins, the subject needs to look at the computer screen and hold the mouse tightly. During the test, the subject needs to click the mouse when he sees or hears "1" (target), and not click the mouse when he sees or hears "2" (interference). The subject's reaction time and the number of incorrect responses during the test will be recorded and converted into a standard quotient with a mean of 100 and a standard deviation of 15. The full-scale attention quotient in the test results is used to assess the subject's attention level (the higher the value, the higher the attention level).
[0095] All subjects took a continuous audio-visual integration test at the beginning and end of the experiment, and the full-scale attention quotient was recorded in the test results. The experimental results showed that after three weeks of attention training, the average full-scale attention quotient of the subjects increased by 14.50.
[0096] The present invention provides a multimodal neurofeedback training method for improving attention. A convolutional neural network model is established that can realize high-precision recognition of attention during neurofeedback training, so as to output the real-time attention state of the subject according to the real-time original EEG signal and original ECG signal of the subject. The real-time attention state of the subject is output in a serious game, and the scene and background music of the serious game are changed to prompt the subject to improve or maintain the current attention state. Through a closed-loop multimodal neurofeedback training system, it is convenient for the subject to actively adjust and control his or her attention state, so as to achieve the purpose of improving attention.
[0097] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A multimodal neurofeedback training method for improving attention, characterized in that: include: S1. Obtaining the original EEG signals and original ECG signals of the subjects when they play serious games; S2, preprocessing the original EEG signal and the original ECG signal to obtain a corresponding EEG signal and an ECG signal with baseline drift removed; S3, classifying the brain wave signal and the ECG signal with baseline drift removed by a preset convolutional neural network model to obtain the subject's attention state; S4. Adjust the scene of the serious game according to the attention state, and return to step S1.
2. A multimodal neurofeedback training method for improving attention according to claim 1, characterized in that: The method of obtaining the original EEG signal and the original ECG signal of the subject when the subject plays the serious game includes: The real-time raw EEG signals and raw ECG signals of the subjects when they play serious games are collected through wearable bioelectric signal acquisition equipment.
3. A multimodal neural feedback training method for improving attention according to claim 2, characterized in that: The preprocessing of the original EEG signal and the original ECG signal to obtain the corresponding EEG signal and the ECG signal without baseline drift includes: The collected original EEG signals and original ECG signals are preprocessed using a wavelet packet decomposition algorithm, the EEG signals of the corresponding bands are extracted from the original EEG signals, and the baseline drift in the original ECG signals is removed to obtain ECG signals with baseline drift removed.
4. A multimodal neurofeedback training method for improving attention according to claim 3, characterized in that: The step of extracting the brain wave signal of the corresponding band from the original brain wave signal comprises: Extract the brain wave signal of the corresponding band according to the following formula: Among them, S EEG is the original EEG signal input, wpdec is the wavelet packet decomposition function used to decompose the original EEG signal input, and wpt EEG is the wavelet packet tree object corresponding to the 9th level decomposition of the input original EEG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal; baseline1 is the baseline signal of node 0 of the 9th layer with a frequency range of 0 to 0.5 Hz when the sampling rate of the signal acquisition device is 512 Hz; [9 0] represents node 0 of the 9th layer, [60] represents node 0 of the 6th layer, [6 1] represents node 1 of the 6th layer, [6 2] represents node 2 of the 6th layer, [7 6] represents node 6 of the 7th layer, [4 1] represents node 1 of the 4th layer, [7 7] represents node 7 of the 7th layer, [7 15] represents node 15 of the 7th layer, and [3 1] represents node 1 of the 3rd layer; S delta , S theta , S alpha , S beta and S gamma is the brain wave signal of the corresponding band to be extracted.
5. A multimodal neural feedback training method for improving attention according to claim 4, characterized in that: The step of removing the baseline drift in the original electrocardiogram signal to obtain the electrocardiogram signal with the baseline drift removed includes: The baseline drift in the ECG signal is removed according to the following formula to obtain the ECG signal with the baseline drift removed: Among them, S ECG is the original ECG signal input, wpdec is the wavelet packet decomposition function used to decompose the original ECG signal input, and wpt EEG is the wavelet packet tree object corresponding to the 9th level decomposition of the input original ECG signal, wprocoef is the wavelet packet reconstruction function used to reconstruct the decomposed signal, baseline2 is the baseline drift with a frequency range of 0 to 0.5 Hz, and Y ECG It is the ECG signal with baseline drift removed.
6. A multimodal neurofeedback training method for improving attention according to claim 1, characterized in that: The preset convolutional neural network model includes: three convolutional layers, three maximum pooling layers, two fully connected layers and one Dropout layer; the convolutional layers are used to extract features from input data, the convolution kernel size of all convolutional layers is 3*3, and ReLU is used as the activation function; The role of the pooling layer is to reduce parameters while retaining the main features. The size of all maximum pooling layers is 2*2; The fully connected layer maps the features obtained from the convolutional layer and the pooling layer to the sample label space, playing the role of a classifier; The Dropout layer can randomly select some nodes to participate in model prediction to prevent overfitting, and the Dropout rate is set to 0.
2.
7. A multimodal neurofeedback training method for improving attention according to claim 6, characterized in that: The brain wave signal and the ECG signal with baseline drift removed are classified by a preset convolutional neural network model to obtain the attention state of the subject, including: The output layer is a fully connected layer, which outputs the probabilities of the subject's low attention state, medium attention state, and high attention state, and takes the attention state with the highest probability as the subject's attention state.
8. A multimodal neurofeedback training method for improving attention according to claim 6, characterized in that: The convolutional neural network model uses the back propagation algorithm to learn and optimize network parameters; Adam is used as the optimizer, and the learning rate is set to 0.0001; the loss function is the cross entropy loss function; the batch size is set to 32, and the epoch is set to 100.
9. A multimodal neurofeedback training method for improving attention according to claim 6, characterized in that: The convolutional neural network model uses precision (PR), recall rate (RE), F1-score and accuracy rate (ACC) as evaluation indicators of model performance, which are defined as follows: Among them, TP represents the number of positive samples correctly identified as positive samples by the model; FP represents the number of negative samples incorrectly identified as positive samples by the model; TN represents the number of negative samples correctly identified as negative samples by the model; and FN represents the number of positive samples incorrectly identified as negative samples by the model.
10. A multimodal neurofeedback training method for improving attention according to claim 1, characterized in that: The adjusting the scene of the serious game according to the attention state comprises: When the attention state is a low attention state, adjusting the scene of the serious game to the first game scene, and adjusting the background music to the first background music; When the attention state is a medium attention state, adjusting the scene of the serious game to a second game scene, and adjusting the background music to a second background music; When the attention state is a high attention state, the scene of the serious game is adjusted to the third game scene, and the background music is adjusted to the third background music.