Real-time training of metacognition based on electroencephalogram data and machine learning model

Abstract

The present application relates to a kind of based on electroencephalogram data and meta-cognition real-time training method of machine learning model, comprising: collecting the current brain cognitive information data of participant and pre-processing;Through fourier transform, the original electroencephalogram data obtained is transferred to the energy value obtained in the preset frequency domain wave band, and the brain cognitive information data is obtained by calculating timing energy value;Brain cognitive information data marking is carried out, brain cognitive information data is labeled according to category and establishes label set, and brain cognitive feature set is established according to preselected feature;Real-time acquisition of brain cognitive information data in training lesson stage, trainer carries out timely voice reminder to participant according to training condition, and the part of training data is marked, and the training set for model training is obtained;The training model obtained in training lesson stage is verified and model correction by verification lesson, until meet training requirement;Real-time guidance information is output by machine learning model in teaching lesson stage, and participant is guided to carry out meta-cognition training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neuroscience, in particular to the field of brain cognitive information processing, and more particularly to a metacognition real-time training method based on electroencephalogram data and a machine learning model. BACKGROUND

[0002] Metacognition is defined as the cognition of cognition, that is, the self-awareness and self-regulation of cognitive activities. It includes people's awareness of their cognitive abilities, their awareness of specific tasks, and their awareness of different strategies. Research shows that "metacognition is essentially a set of explicit and declarative processes that rely on the ability of respondents to report their psychological experiences". Traditional metacognition knowledge interviews and self-report methods in this field cannot be used for real-time guidance.

[0003] Currently, educational institutions mostly use observation methods, which require the participation of teachers throughout the process, and require high manpower investment and high teacher level. Since metacognition is the cognition of people's thinking and inner activities, the results obtained by observation methods are not intuitive. In addition, there are related schemes in the prior art for metacognition assessment through online behavior, which rely on software-recorded student behavior data and provide metacognition participant evaluation reports after class, which cannot provide timely guidance according to the state of students. In addition, there are also shortcomings such as relying on online data, being unable to migrate to offline scenarios, and limited ability to describe "psychological experiences". SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a metacognition real-time training method based on electroencephalogram data and a machine learning model, which provides real-time feedback and precise training process.

[0005] In order to achieve the above purpose, the metacognition real-time training method based on electroencephalogram data and a machine learning model of the present application is as follows:

[0006] The metacognition real-time training method based on electroencephalogram data and a machine learning model, which mainly comprises the following steps:

[0007] (1) The metacognition training course is planned according to training lessons, verification lessons and teaching lessons. First, the current brain cognitive information data of the participant is collected by the electroencephalogram device, and the corresponding electroencephalogram data preprocessing is performed;

[0008] (2) The original electroencephalogram data obtained at present is converted to a preset frequency domain wave band to obtain energy values by Fourier transform, and the brain cognitive information data is obtained by calculating the time sequence energy values;

[0009] (3) brain cognitive information data labeling is carried out, the brain cognitive information data is labeled according to categories to establish a label set, and brain cognitive feature set is established according to preselected features, so as to carry out subsequent model training;

[0010] (4) entering the model training stage, brain cognitive information data obtained by the trainee on the training course is acquired in real time and corresponding data labeling is carried out, and in the training process, the trainer will give timely voice prompts to the trainee according to the training situation, and at the same time, the voice data set of the trainer in the training stage is generated, so as to obtain the training set of the current stage;

[0011] (5) the training model obtained in the training course is verified by the verification course, if not qualified, the data is supplemented for retraining of the model, and the model is corrected until the training requirement is met;

[0012] (6) in the teaching course stage, the training process of the trainee is monitored in real time by the machine learning model, and the voice library data of the trainer is matched synchronously, and real-time voice guidance information is output to guide the trainee to complete the metacognition training.

[0013] Preferably, the step (1) comprises the following steps:

[0014] (1.1) one or more trainees wear an electroencephalogram device, and electroencephalogram data is transmitted to the upper computer software through a Bluetooth module, wherein the sampling rate of each channel is 256 Hz, all electroencephalogram signals are referenced to the mid-frontal point Fpz, and the electroencephalogram signals are processed by a Butterworth filter with a cutoff frequency of 0.5-40 Hz to remove electrooculogram and electromyogram interference;

[0015] (1.2) the length of the moving time window is set as movgT, the moving interval is 1 second, the time point at which the electroencephalogram data collection starts is recorded as 0 second, after movgT length is accumulated, the first energy value one-dimensional array is output, then the time window is moved by 1 second, the second energy value one-dimensional array is output, and so on until the collection is completed.

[0016] Preferably, the step (2) comprises the following steps:

[0017] (2.1) the upper computer software converts the original electroencephalogram data in the time window to 5 frequency bands, including alpha wave, beta wave, gamma wave, delta wave and theta wave, calculates the energy value, and then converts the energy value to a newly created energy data matrix according to the time sequence;

[0018] (2.2) the accelerometer data generated by the accelerometer of the electroencephalogram device is used to determine the motion amplitude of the trainee, and the delta wave energy value in the energy data matrix is used to calculate the quality qlty of the time point for each time windowsec , after the acquisition is completed, obtaining the electroencephalogram data quality dataset qlty;

[0019] (2.3) extracting the energy values in the energy data matrix for calculation, and removing bad data (including poor contact, excessive motion interference, and excessive eye movement) according to the values in the electroencephalogram data quality dataset qlty, obtaining the concentration dataset attCourse with a time resolution of 1 second, the emotional information dataset emoCourse, and the active thinking dataset memCourse.

[0020] Preferably, the step (3) comprises the following steps:

[0021] (3.1) establishing a label set L = {attL, emoL, memL}, wherein attL is a concentration label set, emoL is an emotional information label set, and memL is an active thinking label set;

[0022] (3.2) the system has a built-in brain cognitive feature library, and features with a covariant relationship are grouped into a group, from which the most representative feature is selected to establish a brain cognitive feature set cognFea = {attFea, emoFea, memFea}, wherein attFea is a concentration feature set, emoFea is an emotional information feature set, and memFea is an active thinking feature set.

[0023] Preferably, the step (4) comprises the following steps:

[0024] (4.1) when the training course reaches the preset length of time, if the cumulative total length of valid data is less than the preset length of time, the class time is extended until the valid data length reaches the preset length of time;

[0025] (4.2) according to the real-time displayed brain cognitive information data, the trainer determines the state of the trainee according to the change of one or more items, and when the trainee needs to adjust the metacognition or needs to confirm the current metacognition state of the trainee, the trainer gives the trainee reminder information or feedback information through voice, and forms a voice dataset voice at the end of each class;

[0026] (4.3) According to the preset label, the voice data is divided, the time stamp of the trainer giving the reminder information or feedback information is inserted as a mark insertion point, and the confirmed label forms a label set cnMarker = {aMarker, eMarker, wMarker, keyT} corresponding to the brain cognitive data set cognInd, wherein aMarker is a concentration data label set, eMarker is an emotion information data label set, wMarker is an active thinking data label set, and mkrTRN e represents any one element of any one of the three label sets {mkrTRN e , 1≤e≤fbNums}; keyT is a time stamp set obtained by recording the insertion time point, and tm e represents any one element in keyT {tm e , 1≤e≤fbNums}, and fbNums indicates the number of times of reminding and feedback;

[0027] (4.4) The obtained label set cnMarker j and the data are stacked by rows to obtain a merged set mrkTrain , wherein j is the jth training lesson;

[0028] (4.5) A new empty set cognTrain is created, and according to the time stamp of each label in the merged set mrkTrain, the corresponding time point is found in the brain cognitive information data set {cognInd1, cognInd2, … cognInd j}; The data of s seconds before the time point in cognInd j is stored in cognTrain; From the first to the fbNums-th label in mrkTrain, the label set is traversed to form a brain cognitive data set cognTrain corresponding to the label one by one.

[0029] (4.6) The latest obtained brain cognitive data set cognTrain is proportionally split into a training set and a test set, and cognTrain and mrkTrain are jointly input into a random forest model for further classification processing.

[0030] Preferably, the step (4) further comprises processing the voice data set in the following manner:

[0031] The form of the elements in the voice data set voice is set to {vRec, strT, endT}, wherein vRec is a voice data set, strT is a voice start time set, and endT is a voice end time set;

[0032] Traverse the merged set mrkTrain, and according to the voice start and end time in the data set voice, match the corresponding voice file for each label, the matching rule is: t1 e ≤ tm e ≤ t2 e , wherein t1 e , t2 e are any elements in the set strT, endT, if the voice segment matches tm e corresponding mkrTRN e , then at this time, two of the three elements in the corresponding aMarker, eMarker, wMarker of tm e are empty, and the other is mkrTRN e ; if it cannot be matched, the current label corresponds to an empty file, and the keyT is output to the log file for subsequent checking; when the matching is completed, the set mkrCollect with mkrTRN e or empty as an element is obtained.

[0033] The voice segments classified according to the labels are classified according to the text content, the occurrence frequency of each type of audio is calculated, and the set vProb is obtained, which is used to randomly play the voice according to the corresponding probability in the set vProb when a certain type of label appears when the system automatically guides the training.

[0034] Preferably, the step (5) is specifically:

[0035] The data obtained in the training course stage is verified using the verification course, and the verification set cognTest is obtained, if the verification meets the requirements, the model is put into use, and subsequent training of the participants is carried out by the system without the participation of the trainer; if the verification fails, 50% of the data of each course is selected in the subsequent course and added to the brain cognitive data set cognTrain, and the model is trained and verified again according to the above steps, after a certain number of repeated verification courses, if the verification still fails, 100% of the data of each course is selected in the subsequent course and added to the brain cognitive data set cognTrain, and the model is trained and verified again according to the above steps, until the verification is qualified.

[0036] Preferably, the step (6) is specifically:

[0037] According to the label set L, the concentration, emotional information and active thinking three indexes are respectively judged automatically, and the data with a preset time length in the concentration data set attCourse, the emotional information data set emoCourse and the active thinking data set memCourse are extracted, if the current data set meets one or more characteristics of the label set L set to meet the instant feedback, the machine learning model processes according to the instant feedback; if the current data set meets one or more characteristics of the label set L set to meet the instant reminder, the machine learning model processes according to the instant reminder; if none of them meets, the next time point data is skipped; in this way, the system will make corresponding voice prompts according to real-time judgment, so as to guide the trainees to carry out corresponding metacognition training.

[0038] The metacognition real-time training method based on the EEG data and the machine learning model has the advantages that the mature electroencephalogram technology in the field of neuroscience is used to obtain brain cognitive information, the implicit cognition of the trainee can be obtained without affecting the completion of the task, and the method is more accurate than the observation method. Through the machine learning model trained in advance, the system gives reminders and feedback in the subsequent training task, and the whole process does not need to be guided by experienced trainers. For the trainee, the method can help the trainee master the cognitive regulation method and achieve the purpose of improving the metacognition ability. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The figure is a flow chart of the metacognition real-time training method based on the EEG data and the machine learning model.

[0040] Figure 2 The figure is a framework diagram of the metacognition real-time training method based on the EEG data and the machine learning model. DETAILED DESCRIPTION

[0041] In order to more clearly describe the technical content of the present application, the following further describes in combination with specific embodiments.

[0042] Before the embodiments according to the present application are described in detail, it should be noted that in the following, the terms “include”, “contain” or any other variant are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes these elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0043] The meta-cognition real-time training method based on the electroencephalogram data and the machine learning model can be used in scenes such as life, study and work. It can be divided into two parts: 1, a plurality of portable electroencephalogram acquisition devices and a matching host computer software, responsible for collecting original electroencephalogram data and calculating brain cognitive information; 2, a cloud system, aiming at the brain cognitive information of each person, training a machine learning model, and providing real-time feedback and other functions in the meta-cognition training task.

[0044] Please refer to Figure 1 and 2 , the specific processing process of the meta-cognition real-time training method based on the electroencephalogram data and the machine learning model is as follows:

[0045] I. Obtain brain cognitive information data set from electroencephalogram data

[0046] 1. One or more participants wear electroencephalogram equipment, and the electroencephalogram data is transmitted through a Bluetooth module and received by the host computer software. The sampling rate of each channel is 256 Hz, all electroencephalogram signals are taken as reference with Fpz, and the electroencephalogram signals are processed by a Butterworth filter with a cutoff frequency of 0.5-40 Hz to remove electro-oculogram and electromyogram interference.

[0047] The moving time window length is movgT, and the moving interval is 1 second; the time point when the electroencephalogram data collection starts is recorded as 0 second, and after movgT is accumulated, the software outputs the first energy value one-dimensional array; then the time window moves 1 second backward, and the second energy value one-dimensional array is output; in this way, until the collection is completed.

[0048] 2. The host computer software converts the original electroencephalogram data in the time window to five frequency bands (alpha wave, beta wave, gamma wave, delta wave and theta wave) by Fourier transform, and calculates the energy values; then these energy values are converted into a newly created energy data matrix in chronological order.

[0049] The electroencephalogram equipment is equipped with an accelerometer, and the accelerometer data is used to determine whether the participant's movement amplitude is too large, and then the delta wave energy value in the above-mentioned matrix is used to calculate the quality qlty sec of the time point for each time window; after the collection is completed, the electroencephalogram data quality (quality) data set qlty;qlty sec is obtained. sec elc .

[0050] Each element in qlty sec corresponds to an electrode and the signal quality at the current time point; the subscript elc represents the number of electrodes of the electroencephalogram equipment used; q elc has the following value range and meaning: ​

[0051] • 1, Good signal

[0052] • -1, Partial data affected by eye movement, but still usable

[0053] • -2, Electrode is touching the skin, but connection is not normal

[0054] • -3, Severely affected by eye movement, data is not usable

[0055] • -4, Data is affected by large head movement

[0056] • -5, Signal is severely interfered

[0057] • -6, Corresponding electrode is not touching the skin

[0058] The software extracts the energy values in the energy data matrix to perform calculations, and according to the values in qlty, bad data (including: poor contact, excessive motion interference, excessive eye movement, and qlty less than or equal to -2, i.e. bad data) is removed. The data set attCourse, emoCourse, and memCourse with a time resolution of 1 second are obtained, which correspond to the brain cognitive information data of attention (Attention), emotion information (Emotion), and active thinking (Thinking), respectively. The electroencephalogram data is collected in the classroom, and the data of any course cl j The corresponding data set is represented as cognInd j . Hereinafter, it is referred to as cognInd, and is represented as:

[0059] cognInd = {attCourse, emoCourse, memCourse}

[0060] The length of the course cl j is h minutes; the length of the three items contained in cognInd is len seconds, , in seconds.

[0061] In this embodiment, movgT = 10 seconds; h = 30 minutes.

[0062] In actual applications, the extraction of the above bad data is as follows:

[0063] Example a: The data of the three-axis accelerometer is obtained from the electroencephalogram device, the motion amplitude is calculated, and if it exceeds the threshold value, it is marked as excessive motion interference.

[0064] Example b: The electrooculogram data is extracted from the forehead electroencephalogram electrode data, the eye movement amplitude is calculated, and if it exceeds the threshold value, it is marked as excessive eye movement interference.

[0065] When the quality is less than or equal to -2, the condition is true, and the data is marked as bad data; there are many methods for rejection. For example, the average of the data of the previous 5 time points (5 seconds in this example) of the marked bad data region is m1, and the average of the data of the next 5 time points is m2; example a: use the average of m1 and m2 to replace all data points in the bad data region; example b: use the line (straight line) connecting m1 and m2 to replace the bad data region.

[0066] II. Labeling of brain cognitive information data

[0067] 1. Label set

[0068] The label set is represented as L, and the label sets of attention, emotional information, and active thinking are represented as attL, emoL, and memL, respectively.

[0069] L = {attL, emoL, memL}, attL = {aUp, aDown}, emoL = {eUp, eDown}, memL = {wUp, wDown}, and any one of attL, emoL, and memL is represented as cognL.

[0070] The first element in cognL corresponds to "immediate feedback"; it represents one or more of the following states of the brain cognitive index: rising, maintaining a high level, rising above the threshold; the system sends "feedback".

[0071] The second element in cognL corresponds to "immediate reminder"; it represents one or more of the following states of the brain cognitive index: decreasing, maintaining a low level, decreasing below the threshold; the system sends "reminder".

[0072] 2. Feature set

[0073] The system has a built-in brain cognitive feature library, and after screening, features with a common relationship are grouped into a group, and the most representative one is selected, where "common relationship" refers to the coordinated change between variables or objects, which means that multiple features have the same trend. In practical applications, common methods for determining "common relationship" include:

[0074] Method a: use X and Y to represent two feature data sets, and their covariance (Covariance) is written as cov.

[0075] If Cov(X, Y) > 0, X and Y are positively correlated; if Cov(X, Y) < 0, X and Y are negatively correlated; if Cov(X, Y) = 0, X and Y may be independent.

[0076] If the absolute value of cov is greater than the threshold value, it can be detected whether the multiple sets of feature values have a covariant relationship.

[0077] Method B: Draw a time series graph of multiple sets of feature values, and visually determine which features have a covariant relationship.

[0078] And "most representative" usually refers to the most sensitive indicator when analyzing data, from which a "most representative" feature is selected for subsequent calculation, and other features are discarded. Common methods for determining "most representative" include selecting based on cov value, experience, or trial and error.

[0079] Accordingly, the brain cognitive feature set cognFea is obtained, which is represented as:

[0080] cognFea = {attFea, emoFea, memFea};

[0081] attFea = {aF1, aF2, aF3, aF4, aF5, aF6, aF7, aF8};

[0082] emoFea = {eF1, eF2, eF3, eF4, eF5, eF6, eF7, eF8, eF9, eF10};

[0083] memFea = {wF1, wF2, wF3, wF4, wF5, wF6, wF7, wF8};

[0084] Wherein, aF1, eF1, wF1 represent the average value of amplitude; aF2, eF2, wF2 represent the variance; aF3, eF3, wF3 represent the similarity; aF4, eF4, wF4 represent the minimum value; aF5, eF5, wF5 represent the slope; aF6, eF6, wF6 represent the waveform factor; aF7, eF7, wF7 represent the pulse factor; aF8, eF8, wF8 represent the center of gravity frequency; eF9 represents the zero-crossing rate; eF10 represents the positive value ratio;

[0085] For a time series signal with a time length of s and a time resolution of 1 second (i.e. ep e in the third step 2), represented as: cng = {cng1, cng2, cng3…, cng s};

[0086] Mean:

[0087] ;

[0088] Variance:

[0089] ;

[0090] Similarity: freDist = {fre1, fre2, fre3…, fre x}, x < m, means not to calculate the similarity for each label l m .

[0091] fre x is the similarity between a piece of data of length s in cognCourse and the feature waveform marked by a label in cognL.

[0092] Min: is the minimum value in cng.

[0093] Slope: is obtained by fitting cng with least square method or the like. The slope greater than zero represents an upward trend; the slope less than zero represents a downward trend; the slope approximately equal to zero represents a basically unchanged trend.

[0094] Waveform factor: the ratio of effective value and mean value.

[0095] Effective value, i.e. root mean square value RMS:

[0096] ;

[0097] Pulse factor: the ratio of peak value and mean value of absolute value.

[0098] Gravity frequency:

[0099] ;

[0100] Wherein, P(k) is the corresponding power spectrum value, f k is the frequency amplitude of the corresponding point, and Num is the frequency quantity.

[0101] Zero-crossing rate: represents the number of times that the signal cng passes through zero.

[0102] Positive value proportion: represents the ratio of the number of data points greater than zero in the signal cng to the total amount of data points. Its minimum value is 0, and the maximum value is 1.

[0103] III. Training of the model

[0104] In a specific embodiment, the method is designed as 40 lessons, represented as {cl1, cl2, cl3,…,cl j}, 1≤j≤40. In addition to the 40 lessons, there is a verification lesson. In actual application, lessons 1-5 are training lessons, after the training lessons, the model is trained, and the participants take the verification lesson. After the verification lesson, the data obtained by the verification lesson is used to verify the model. If the verification is passed, the model training starts automatically from lesson 6. If the verification is not passed, the data of lessons 5-10 is supplemented to the cognTrain training set to train the model again, and the verification lesson data is used to verify again until the training is passed. Lessons 11-40 are automatically carried out according to the participant's own situation.

[0105] 1. For training courses and data:

[0106] Lesson 1, Stage 1: Learn as usual; Stage 2: Learn with metacognitive awareness.

[0107] Lesson 2, Stage 1: Learn related tasks; Stage 2: Non-learning tasks.

[0108] Lesson 3, Stage 1: Interesting things; Stage 2: Uninteresting things.

[0109] Lesson 4, Stage 1: Tasks that are good at; Stage 2: Tasks that are not good at.

[0110] Lesson 5, Stage 1: Tasks that do not feel nervous; Stage 2: Tasks that feel stressed.

[0111] For the course of model training stage (i.e. training lessons), when the course reaches the preset duration, if the cumulative total duration of valid data is less than h, the class (task) time can be extended until the valid data duration reaches h.

[0112] According to the data of the same course of multiple people stored in the system norm library, the threshold set mkTH = {aH, aL, eH, eL, wH, wL} is calculated, wherein aH and aL are high and low thresholds of concentration; eH and eL are high and low thresholds of emotional information; wH and wL are high and low thresholds of active thinking.

[0113] When the participant's brain cognitive information exceeds 1.15 times of the corresponding high threshold and maintains for prd seconds, the system pops up a blue notice window; when the participant's brain cognitive information is less than 1.15 times of the corresponding low threshold and maintains for prd seconds, the system pops up a red notice window;

[0114] The name of the brain cognitive information is displayed in the window; the participant should immediately adjust the brain cognitive information; the trainer observes the real-time display data to determine whether the participant can effectively adjust; if the participant's adjustment is effective, the trainer does not send a prompt message; if the adjustment is not effective, the trainer sends a prompt message;

[0115] Definition of effective adjustment: for blue notice window, the participant can maintain the item of brain cognitive information above the threshold level within r x n Z seconds after the pop-up; for red notice window, the participant can adjust the item of brain cognitive information back to above the threshold level within r x n Z seconds after the pop-up and keep it;

[0116] Take concentration as an example: when the concentration of the participant is lower than If the participant fails to adjust the concentration by using metacognition within prd seconds, the system pops up a red notice window with the word “concentration”; the participant is busy with the course task and ignores the notice window and fails to adjust the concentration by using metacognition; the trainer observes the real-time data and finds that the participant fails to effectively adjust, and immediately gives a prompt information (reminder).

[0117] In this example, prd = 5 seconds and r x n Z = 30 seconds

[0118] When giving voice reminders or feedback, the trainer follows the principle of not giving two voices in the minimum time interval; in this example, the minimum time interval is 50 seconds. Each time, only one of the concentration, emotional information, and active thinking is given.

[0119] After 1 to 5 lessons, the system starts model training.

[0120] 2. Labeling of training phase data

[0121] According to the real-time displayed brain cognitive information data, the trainer gives the participant “prompt information” or “feedback information” when necessary. The above two types of information are given by the trainer through voice, recorded by the microphone connected to the system, and the voice start and end time stamps are recorded; at the end of each lesson, the voice data set voice j is formed, j represents the jth lesson, “prompt information” is given when the concentration, emotional information, and active thinking of the participant drop significantly, or drop and keep, and reach the judgment standard of the trainer; “feedback information” is given when the concentration, emotional information, and active thinking of the participant rise and keep, and reach the judgment standard of the trainer; the trainer gives the above two types of information (a total of six types) at the same time, presses the corresponding keys on the keyboard, and the system records the key press time stamp as the marking insertion time point, and records the key press to distinguish which type of marking; after each lesson, the label set cnMarker j is obtained.

[0122] The trainer observes and records the state of the participants on site. For example: stuck in the problem-solving process, the person is also in a free state, which can be judged as the participant is in a non-active thinking state, and the usual concentration level is also at a low level. The above is only an example, and there are many similar judgment methods.

[0123] After the course is completed, the trainer communicates with the participants and reviews the whole class; the participants review the brain cognitive state of the key period in the course and give their own description and evaluation of the cognitive process.

[0124] For the mispressing situation: the trainer confirms the accuracy of the real-time key pressing again after the class.

[0125] Combined with the above three aspects, re-evaluate the accuracy of each label; and can modify or delete the existing label, then update the label set cnMarker j . At the same time, the label modification will be recorded in the system.

[0126] Suppose course C j has fbNums times of reminders and feedback; the confirmed labels (Labels) form the brain cognitive data set cognInd j The corresponding label set is:

[0127] cnMarker j = {aMarker, eMarker, wMarker, keyT}

[0128] aMarker, eMarker, wMarker, respectively represent the label set for concentration, emotional information, and active thinking data; use mkrTRN e to represent any element in any of the three label sets {mkrTRN e ,1≤e≤fbNums}.

[0129] keyT is a timestamp set obtained by recording the key pressing time; use tm e to represent any element in keyT {tm e ,1≤e≤fbNums}.

[0130] Take concentration as an example:

[0131] ;

[0132] The elements mkrTRN e in eMarker or wMarker are the same as those in aMarker.

[0133] cnMarker j will be used for the next model training.

[0134] 3. Generate training set

[0135] For the label set obtained in the first to fifth lessons, i.e., the labels in the model training stage, merge for training. As follows, do stacking processing by row:

[0136] ;

[0137] Newly create an empty set cognTrain, if the set already exists, empty it.

[0138] According to the timestamp of each label in mrkTrain, find the corresponding time point in the brain cognitive information data set {cognInd1, cognInd2, … cognInd5}; take the data of s seconds from the time point in cognInd j , and store the segment of data in cognTrain; from the first to the fbNums labels in mrkTrain, traverse the label set to form a brain cognitive data set cognTrain corresponding to each label.

[0139] In this example, s = 30 seconds.

[0140] 4. Train the model

[0141] The construction process of the decision tree involves the division of nodes: the algorithm uses Gini index minimization as the division criterion, i.e., selecting the feature and the split point that make the Gini index after division minimum.

[0142] ;

[0143] wherein, and are the left and right subsets divided by feature A according to the value .

[0144] When making predictions, each tree votes for a class, and the final prediction result is the class that obtains the most votes;

[0145] Classification:

[0146] ;

[0147] wherein, N is the number of trees in the forest, is the prediction result of the i-th tree.

[0148] The training adopts the 10-fold cross-validation method. cognTrain is proportionally split into a training set and a test set.

[0149] 5. Training process

[0150] CognTrain and mrkTrain are inputted into a random forest (RF) model together.

[0151] Subsets of data generated by backfill sampling are done on the training set.

[0152] For the training of each decision tree, a subset of features is randomly selected from all features for node splitting.

[0153] Each decision tree is trained using the subset of features and the subset of data generated by sampling.

[0154] The training process of a decision tree includes selecting the best feature for node splitting until the maximum depth of the tree is reached or the number of samples contained in a node is less than a certain threshold.

[0155] The RF model determines the final classification result by majority voting mechanism.

[0156] OOB (Out-of-Bag) error estimation: the number of samples incorrectly classified by the model among the samples not sampled divided by the total number of samples not sampled.

[0157] Each decision tree is evaluated using samples not used in training (OOB samples).

[0158] The hyperparameters of the RF model, such as the number of trees, the maximum depth of each tree, etc., are adjusted to optimize the performance of the model.

[0159] Take focus as an example: from the first to the fbNums data segment, extract the feature data vector attFea e ; the RF model classifies according to the features.

[0160] Four, processing of voice set

[0161] The form of elements in the voice dataset voice is {vRec, strT, endT}, where vRec is a set of voice data, strT is a set of voice start time, and endT is a set of voice end time; rec e , t1 e , t2 e are any elements in vRec, strT, and endT, respectively.

[0162] Iterate through mrkTrain and match each label to the corresponding voice file according to the voice start and end times.

[0163] Matching rule: when t1 e ≤ tm e ≤ t2e , consider the segment as matching tm e e , consider the segment as matching tm e , consider the segment as matching tm e If no match, the label is empty, output keyT to log file, for later check. After matching, get a set mkrCollect with elements as mkrTRN e or empty.

[0164] vocBank = {keyT, mkrCollect, vRec}, the elements in vocBank are in the form (tm e , mkrTRN e , rec e ).

[0165] classify vRec in vocBank according to mkrTRN e , totally 6 classes, same as the elements in label set L.

[0166] classify rec e according to label, then convert to text by speech recognition module, classify the rec e according to the text content, calculate the frequency of each class, get a set vProb, when a class (mkrTRN e ) appears, play a rec e according to the probability in vProb.

[0167] V. Verification and model correction

[0168] 1. Course and data

[0169] Verification course, task type covers all the types in phase 2 of lesson 1-5.

[0170] After the verification course, use the same method in step 3 to label the data of the verification course as cognTest. If the verification meets the requirements, the model is put into use, and from lesson 6, the system automatically sends guidance information to the participants.

[0171] If the verification course shows that the verification indicators do not meet the standards, the data of lessons 6-10 need to be added to cognTrain, and then the model needs to be trained again until it is qualified.

[0172] Lesson 6 corresponds to lesson 1; task: meta-cognitive observation.

[0173] Lesson 7, corresponding to Lesson 2; Task: Can learning be made enjoyable?

[0174] Lesson 8, corresponding to Lesson 3; Task: Can one become interested in oneself?

[0175] Lesson 9, corresponding to Lesson 4; Task: Find the feeling of being at ease in the task.

[0176] Lesson 10, corresponding to Lesson 5; Task: Can one improve one's state of tension?

[0177] 2. Verification index

[0178] Parameters used in verification:

[0179] Sensitivity (sensitivity), denoted as sensitivity:

[0180] ;

[0181] False Positive Rate (False Positive Rate), denoted as FDR up or FDR down , abbreviated as FDR.

[0182] ;

[0183] h is the course length; subscript up corresponds to "immediate feedback"; subscript down corresponds to "immediate reminder"; for concentration, its FDR is denoted as attFDR up and attFDR down ; emotional information is denoted as emoFDR up and emoFDR down ; active thinking wkFDR up and wkFDR down .

[0184] sensitivity, FDR and the preset threshold are compared to evaluate the accuracy of the model.

[0185] 3. Verification and correction

[0186] Data verification is arranged after the completion of the verification lesson, and the data of the verification lesson is taken as the verification set cognTest.

[0187] After verification, the model is put into use, and from Lesson 6, the system automatically conducts training for participants without the need for trainers to participate in the course.

[0188] If the model fails the verification, it needs to be trained again; from the 6th to the 10th lesson, 50% of the data from each completed lesson is added to congTrain; at this time, the newly added data in the training set and test set needs to be labeled according to the method in the third step; then the model is trained and verified again; if it is qualified, the trainer does not need to participate from the next lesson; if it is not qualified, continue to train and verify after each lesson is completed until the 10th lesson.

[0189] After the 10th lesson, if the model has not passed the verification, 100% of the data from each lesson needs to be added to congTrain; at this time, all the content from the 6th to the 10th lesson needs to be labeled according to the method in the third step; then the model is trained and verified again. Before the model passes the verification, the trainer should participate in each lesson and give the trainee "reminders" or "feedback".

[0190] Six, machine learning model guided training

[0191] 1. Course and meta-cognitive teaching process

[0192] The 10th to 40th lessons (i.e. teaching lessons), where the dataset cognInd j , 10≤j≤40. The task is mainly to learn the content, supplemented by projects of interest to the trainee, built-in cognitive training tasks in the system, mindfulness, etc.

[0193] Each lesson is 30 minutes long. After the start of the course, brain cognitive data is delivered to the model, and the system helps the trainee to carry out meta-cognitive training through fully automatic "real-time guidance information".

[0194] 2. Output real-time guidance information

[0195] According to the label set L in the second step, the system automatically determines the three indicators of concentration, emotional information, and active thinking; taking concentration as an example, extract the data (epoch) with a length of s seconds from attCourse to make a judgment, if it meets the characteristics of aUp, the system processes it as "immediate feedback"; if it meets the characteristics of aDown, the system processes it as "immediate reminder"; if it does not meet the above two characteristics, it will skip to the next epoch; after the system makes a judgment, it will play the voice according to the method in the fourth step; at the same time, the system pops up a text prompt window.

[0196] The course starts, cognIndj is empty, which is the time 0 of the class; when s seconds of data have been stored in the data set, the system carries out identification on the data (epoch 0); after win seconds, the system takes the data from win seconds to win+s seconds (epoch 1) to carry out identification; that is, s is used as a time window, win is used as a translation, and a sliding time window method is used to extract data from cognIndj for system analysis; in this example, win=10 seconds.

[0197] In order to avoid frequent prompt information or misjudgment, it is necessary to accumulate epNums consecutive epochs consistent with the determination to reach the trigger condition; in this example, epNums=3.

[0198] The "instant feedback" corresponds to the increase of the three brain cognitive information indicators, and when at least one of them reaches the standard, the system can inform the participants through the information box and the rece voice in the same category vRec.

[0199] The "instant reminder" corresponds to the decrease of the three brain cognitive information indicators, and when at least one of them reaches the standard, the system can inform the participants through the information box and the rece voice in the same category vRec.

[0200] Both of them are used to let the user instantly find out the improvement (feedback) and decrease (reminder) of his state during the task process, so that the participants can adjust the concentration, emotional information and active thinking in time.

[0201] Before the course starts, the students can choose to give guidance information to multiple or a certain item among concentration, emotional information and workload during the course; the students can choose the frequency of the appearance of the guidance information in high, medium and low three grades; or choose no guidance information in the class.

[0202] The technical solution has the following technical effects in practical application:

[0203] (1) Save manpower

[0204] The technical solution makes a judgment according to the brain cognitive information of the participants by the machine learning model, and the system automatically gives a reminder and feedback, which has the characteristics of high efficiency and full automation. In the metacognition teaching process, the trainer without metacognition teaching experience does not need to accompany all the time, which can save a lot of manpower.

[0205] (2) Can be used for group training in actual scene

[0206] The technical solution can collect the brain electrical data of multiple people at the same time, and is suitable for multiple people communication application scene; it is suitable for embedding in actual life, learning and work scenes to carry out measurement and training.

[0207] In addition, in the training process of meta-cognition, a tutor cannot usually take care of multiple students, and the technical solution reduces the threshold of real-time meta-cognition training of multiple people.

[0208] (3) Fast feedback speed

[0209] The technical solution automatically gives guidance according to the brain cognitive information index of the participant by the machine learning model based on the electroencephalogram data. The electroencephalogram data has high time resolution, and can timely reflect the cognitive activity of the participant in the system; the calculation speed of the random forest model is relatively faster than that of the machine learning model; the combination of the two can issue guidance information to the participant within an acceptable delay.

[0210] Any process or method descriptions, or any other descriptions herein, can be understood as representing embodiments of the application comprising an executable code of one or more steps for implementing a particular logic function or process, and the scope of preferred embodiments of the application includes additional implementations that can not be expressly shown or discussed below, where the functions involved can be performed in different order from that shown or discussed, including essentially simultaneously or in reverse order, as will be understood by those having ordinary skill in the art to which embodiments of the application pertain.

[0211] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0212] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, and the programs include one of the steps of the method embodiments or a combination thereof when executed.

[0213] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0214] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

[0215] The meta-cognition real-time training method based on the electroencephalogram data and the machine learning model has the advantages that brain cognitive information can be obtained by means of mature electroencephalogram technology in the field of neurology, implicit cognition of a trainee can be obtained without affecting the trainee to complete a task, and the method is more accurate than an observation method. Through a machine learning model trained in advance, a reminder and feedback are given by the system in subsequent training tasks, and the whole process does not need to be guided by an experienced trainer. The method can help the trainee to master a cognitive regulation method and achieve the purpose of improving meta-cognition ability.

[0216] In this specification, the application has been described with reference to specific embodiments thereof. It is apparent, however, that various modifications and changes can be made thereto without departing from the spirit and scope of the application. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A real-time metacognitive training method based on EEG data and machine learning models, characterized in that, The method includes the following steps: (1) The metacognitive training course is planned in the form of training course, verification course and teaching course. First, the brain cognitive information data of the trainees is collected by EEG device and the corresponding EEG data preprocessing is performed. (2) By using Fourier transform, the raw EEG data currently acquired is transferred to a preset frequency domain band to obtain energy values, and brain cognitive information data is obtained by calculating the time-series energy values. (3) Mark the brain cognitive information data, establish a label set according to the category of the brain cognitive information data, and establish a brain cognitive feature set according to the preset selected features for subsequent model training; (4) Enter the model training stage, obtain the brain cognitive information data obtained by the trainees in the training class in real time and mark the data accordingly. During the training process, the trainer will give timely voice reminders to the trainees according to the training situation, and generate the voice dataset of the trainer in the training stage to obtain the training set of the current stage. (5) The training model obtained in the training phase is verified through the verification course. If it is not qualified, data is added to train the model again and the model is corrected until it meets the training requirements. (6) During the teaching phase, the machine learning model monitors the training process of the trainees in real time and matches the trainer's voice database data in real time to output real-time voice guidance information to guide the trainees to complete metacognitive training. Step (1) includes the following steps: (1.1) One or more trainees wear EEG devices, and the EEG data is transmitted to the host computer software via Bluetooth module. The sampling rate of each channel is 256 Hz. All EEG signals are referenced with the frontal pole midpoint Fpz. Ocular and electromyographic interferences are removed, and the data is processed by a Butterworth filter with a cutoff frequency of 0.5 to 40 Hz. (1.2) Set the length of the moving time window to movgT, the moving interval to 1 second, and the start time of EEG data acquisition to 0 seconds. After accumulating the acquisition of movgT length, output the first one-dimensional array of energy values, then move the time window forward by 1 second and output the second one-dimensional array of energy values, and so on, until the acquisition ends. Step (2) includes the following steps: (2.1) The host computer software converts the raw EEG data within the time window into five frequency bands, including alpha, beta, gamma, delta and theta waves, through Fourier transform, calculates the energy values, and then stores the calculated energy values ​​in a newly created energy data matrix in chronological order. (2.2) The amplitude of the trainee's movement is determined by the accelerometer data generated by the accelerometer built into the EEG device, and the mass qlty at each time point is calculated using the delta wave energy value in the energy data matrix. sec After the data collection is completed, the EEG data quality dataset qlty is obtained; (2.3) Extract the energy values ​​from the energy data matrix and perform calculations. Based on the values ​​in the EEG data quality dataset qlty, remove bad data to obtain the attention dataset attCourse, the emotion information dataset emoCourse, and the active thinking dataset memCourse with a time resolution of 1 second.

2. The metacognitive real-time training method based on EEG data and machine learning models according to claim 1, characterized in that, Step (3) includes the following steps: (3.1) Establish a label set L = {attL, emoL, memL}, where attL is the focus label set, emoL is the emotional information label set, and memL is the active thinking label set; (3.2) The system has a built-in brain cognitive feature library. Features with covariation relationships are grouped together, and the most representative feature is selected to establish a brain cognitive feature set cognFea = {attFea, emoFea, memFea}, where attFea is the focus feature set, emoFea is the emotional information feature set, and memFea is the active thinking feature set.

3. The real-time metacognitive training method based on EEG data and machine learning models according to claim 1, characterized in that, Step (4) includes the following steps: (4.1) When the training course reaches the preset duration, if the total accumulated duration of effective data is less than the preset duration, the class time will be extended until the effective data duration reaches the preset duration. (4.2) Based on the real-time displayed brain cognitive information data, the trainer determines the trainee's state based on one or more of the changes. When the trainee needs to adjust metacognition or when the trainee's current metacognition state needs to be affirmed, the trainer provides the trainee with targeted reminders or feedback information through voice, and forms a voice dataset at the end of each class. (4.3) Divide the voice data according to the preset labels, and use the timestamp of the reminder or feedback information given by the trainer as the insertion point of the label. Form a label set cnMarker = {aMarker, eMarker, wMarker, keyT} corresponding to the brain cognition dataset cognInd, where aMarker is the set of attention data labels, eMarker is the set of emotion information data labels, and wMarker is the set of active thinking data labels. Use mkrTRN e This represents any element {mkrTRN} in any of the three tag sets. e ,1≤e≤fbNums};keyT is the set of timestamps obtained from the time of record insertion, denoted by tm e Represents any element {tm} in keyT e ,1≤e≤fbNums}, where fbNums refers to the number of reminders and feedback provided; (4.4) Obtain the tag set cnMarker j Data is stacked row by row to obtain a merged set. Where j is the j-th training session; (4.5) Create a new empty set cognTrain, and according to the timestamp of each label in the merged set mrkTrain, in the brain cognitive information dataset {cognInd1, cognInd2, ... cognInd... j Find the corresponding time point in}; store this time point in cognInd j Data from the previous s seconds is stored in cognTrain; starting from the 1st to the fbNumsth label in mrkTrain, the label set is traversed to form a brain cognition dataset cognTrain that corresponds one-to-one with each label; (4.6) The newly obtained brain cognition dataset cognTrain is split into training set and test set according to the proportion, and cognTrain and mrkTrain are input into the random forest model for further classification processing.

4. The metacognitive real-time training method based on EEG data and machine learning models according to claim 3, characterized in that, Step (4) further includes processing the speech dataset in the following manner: Set the elements in the voice dataset to the form {vRec, strT, endT}, where vRec is the set of voice data, strT is the set of voice start times, and endT is the set of voice end times; Iterate through the merged set mrkTrain and, based on the start and end times of speech in the voice dataset, match the corresponding speech file for each label. The matching rule is: t1 e ≤tm e ≤t2 e , where t1 e t2 e Given any element in sets strT and endT, if the audio segment is related to tm e The corresponding mkrTRN e If it matches, then it is now with tm e Of the three elements in the corresponding aMarker, eMarker, and wMarker, two are empty, and the other is mkrTRN. e If no match is found, the current tag corresponds to an empty file, and keyT is output to a log file for later verification; when a match is found, mkrTRN is obtained. e or an empty collection of elements, mkrCollect; The audio segments categorized by labels are classified according to their text content. The frequency of each audio category is calculated and compiled into a set vProb. During automatic training guided by the system, when a certain label appears, audio is played randomly according to the corresponding probability in the vProb set.

5. The real-time metacognitive training method based on EEG data and machine learning models according to claim 4, characterized in that, The specific steps (5) are as follows: The data obtained in the training phase is validated using the validation course to obtain the validation set cognTest. If the validation meets the requirements, the model is put into use, and the system will automatically train the participants without the need for trainers to participate in the course. If the validation fails, 50% of the data from each subsequent course needs to be added to the brain cognition dataset cognTrain, and the model training and validation are repeated according to the above steps. After a set number of repeated validation courses, if the validation still fails, 100% of the data from each subsequent course needs to be added to the brain cognition dataset cognTrain, and the model training and validation are repeated according to the above steps until the validation is successful.

6. The metacognitive real-time training method based on EEG data and machine learning models according to claim 4, characterized in that, The specific steps (6) are as follows: Based on the aforementioned label set L, the three indicators of focus, emotional information, and active thinking are automatically judged. Data of a preset time length is extracted from the focus dataset attCourse, the emotional information dataset emoCourse, and the active thinking dataset memCourse. If the current dataset meets one or more features that satisfy the instant feedback set by label set L, the machine learning model processes it according to the instant feedback. If the current dataset contains one or more features that satisfy the instant reminder as defined by the label set L, then the machine learning model will process the data according to the instant reminder. If none of these conditions are met, skip processing the data at the next time point. Similarly, the system will make targeted voice prompts based on real-time judgments and match the trainer's voice database data to guide trainees in carrying out corresponding metacognitive training.

Images