A method and apparatus for error-aware prediction

Abstract

The present disclosure relates to an error awareness prediction method and device, electronic equipment and storage medium. The method comprises: showing a preset task to a testee, and collecting electroencephalogram signals, corrugator muscle electromyogram signals and eye movement information of the testee when completing the preset task; after preprocessing the electroencephalogram signals, electromyogram signals and eye movement information, performing data analysis and feature extraction to generate electroencephalogram signal amplitude and latency characteristics, corrugator muscle electromyogram characteristics, integrated electromyogram values, Lempel-Ziv complexity and pupil diameter characteristics; using the above characteristics as input, determining based on a preset error awareness prediction model based on an XGBoost classifier to complete error awareness prediction. The present disclosure has high prediction accuracy and can reduce the probability of human error occurrence while minimizing the impact on operation continuity.

Description

Technical Field

[0001] This disclosure relates to the fields of human factors engineering and safety management, and more specifically, to a method, apparatus, electronic device, and computer-readable storage medium for error awareness prediction. Background Technology

[0002] In aerospace, nuclear power, and petrochemical industries, human error accounts for over 60% of accidents. Taking effective measures to reduce and prevent human error is a key focus of human-machine system reliability research. Recognizing that errors occur is a prerequisite for behavioral adjustment and is crucial for early warning and correction of human error. Therefore, identifying the relevant physiological activity patterns of different error awareness states and proposing related error prediction / early warning methods to reduce the probability of human error has significant theoretical and engineering application value for improving system reliability and preventing catastrophic accidents in a timely manner.

[0003] Previous research on predicting human error has primarily focused on whether brain activity changes before an error occur differ from those before a correct action. For example, EEG power spectroscopy studies have found an increase in alpha power in the occipital-parietal region before an error, suggesting that alpha power is related to decreased attention. Other studies, using correct response-related negative waves (CRNs) in event-related potentials, have found that the average CRN amplitude in responses before a correct response is higher than that in responses before an error. Studies using pre-error positive waves (EPPs) have found that the amplitude of EPPs before an error is larger than that before a correct response. Still other studies have compared the differences in electrical signals before and after an error using other peripheral nerve signals, such as corrugating eyebrow electromyography (CEMG).

[0004] In summary, current research on human error prediction mainly compares the differences in physiological signals before correct and incorrect responses to characterize whether an error will occur in the subsequent response. In fact, errors can be further categorized according to the level of consciousness. The brain's error processing is related to behavioral monitoring. During reactions and operations, the brain constantly evaluates the operation and makes flexible adaptive adjustments. Timely detection and correction of behavioral errors constitute error processing, a crucial characteristic of adaptive behavior in the dynamic and ever-changing daily environment. Typically, human error correction is so rapid that the error trend is corrected as soon as it appears, or even before an error result occurs. For example, when we are about to press the wrong phone key, our hand automatically stops moving before the action is executed. Therefore, for conscious errors, the brain can correct them promptly using its own error processing mechanisms, thus avoiding serious consequences. Only for unconscious errors do we need to provide early warnings and interventions to prevent subsequent serious consequences. Conversely, if we disregard error consciousness and provide warnings and interventions for all predicted errors, it will not only waste the brain's error processing resources but also exacerbate operational inconsistencies and reduce work efficiency.

[0005] Therefore, one or more methods are needed to solve the above problems.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this disclosure is to provide a method, apparatus, electronic device, and computer-readable storage medium for predicting error awareness, thereby overcoming, at least to some extent, one or more problems caused by limitations and defects in related technologies.

[0008] According to one aspect of this disclosure, a method for predicting error awareness is provided, comprising:

[0009] A preset task is presented to the test subject, and the electroencephalogram (EEG) signal and electromyogram (EMG) signal of the corrugator supercilii muscle are collected based on the EEG cap when the test subject completes the preset task. The eye movement information of the test subject is collected based on the eye tracker when the test subject completes the preset task.

[0010] After preprocessing the EEG signals, EMG signals, and eye movement information, data analysis and feature extraction are performed to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0011] Using the amplitude and latency features of the EEG signal, the brow furrow electromyography features, the integrated electromyography value, the Lempel-Ziv complexity, and the pupil diameter features as inputs, the system makes a judgment based on a preset error consciousness prediction model based on the XGBoost classifier, and completes the error consciousness prediction.

[0012] In one exemplary embodiment of this disclosure, the preset task in the method includes:

[0013] Preset stimulus patterns are displayed on the computer screen in the form of a preset program;

[0014] Each stimulation pattern is a single colored Chinese character, and the color of each stimulation is one of green, red, yellow, blue, purple, or white.

[0015] When the font color of the Chinese character is inconsistent with the meaning of the Chinese character, the tester should press the D key on the keyboard with his left hand. When the same Chinese character appears twice in a row or the meaning of the Chinese character is consistent with the font color of the Chinese character, the tester should press the F key on the keyboard with his left index finger.

[0016] After the button press is completed, the screen will display a consciousness state evaluation interface with three preset expressions: a smiling face, a calm face, and a crying face. The test subject will subjectively evaluate the accuracy of the left-hand button press. If the test subject thinks the reaction is correct, the right ring finger will press the L key to correspond to the smiling face; if the test subject thinks the reaction is incorrect, the right index finger will press the J key to correspond to the crying face; if the test subject is unsure whether the reaction is correct or not, the right middle finger will press the K key to correspond to the calm face.

[0017] In one exemplary embodiment of this disclosure, the preset stimulus pattern in the method includes:

[0018] A fixed white crosshair gaze point, lasting 500ms;

[0019] After the cross gaze point disappears, a colored Chinese character stimulus is displayed and lasts for 800ms;

[0020] After the tester pressed the button with their left hand, the stimulation interface disappeared and a 1000ms black screen appeared.

[0021] The consciousness state evaluation interface is displayed within 1000ms;

[0022] After the test subject presses the button with their right hand, the consciousness state evaluation interface disappears and a 600ms black screen appears.

[0023] In one exemplary embodiment of this disclosure, the method further includes an error awareness prediction model training step:

[0024] Show the preset task to the test subject and receive the key press information of the test subject after completing the preset task;

[0025] The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task, and the eye movement information of the test subject was collected based on the eye tracker when the test subject completed the preset task.

[0026] After preprocessing the EEG signals, EMG signals, and eye movement information, data analysis and feature extraction are performed to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0027] A mistaken awareness prediction model based on the XGBoost classifier was established, and the key information, EEG signal amplitude and latency features, frowning electromyography features, integral electromyography value, Lempel-Ziv complexity, and pupil diameter features were used as training data to train the mistaken awareness prediction model based on the XGBoost classifier.

[0028] In one exemplary embodiment of this disclosure, the method further includes:

[0029] The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task. After collecting the eye movement information of the test subject when the test subject completed the preset task based on the eye tracker, the EEG, EMG, pupil diameter and behavioral data of the test subject were synchronized based on time tags.

[0030] In one exemplary embodiment of this disclosure, the method further includes preprocessing the electroencephalogram (EEG) signal:

[0031] The EEG signal was referenced to the average value of the bilateral mastoid processes and bandpass filtered at 0.1 to 35 Hz to remove artifacts caused by high-frequency electromyography and slow voltage drift. The EEG signal was divided into time windows of 400 ms to 1000 ms with the left hand key press time as the zero point and then baseline correction was performed to generate segmented EEG signals.

[0032] The segmented EEG signals were visually inspected to remove segments with artifact content greater than a preset value, and spherical splines were used to interpolate the channels of segments with noise greater than a preset number.

[0033] Independent component analysis was performed on the segmented EEG signals to identify and remove components containing blinking / eye movement artifacts;

[0034] Extract the CRN and EPP event-related potential components from each segment of the segmented EEG signal data. CRN is the average amplitude within a window of 10-40ms after the left hand key press, and EPP is the average amplitude within a time window of 120-150ms after the left hand key press.

[0035] In one exemplary embodiment of this disclosure, the method further includes preprocessing the electromyographic signal:

[0036] The electromyography signal was filtered using a 28-500Hz bandpass filter and a 50Hz notch filter, and then the signal was smoothed by moving the signal with a window width of 20 data points.

[0037] Based on the left-hand key press response time, the electromyographic signal is divided into 1000ms time windows and then baseline correction is performed to generate segmented electromyographic signals.

[0038] The segmented electromyographic signals are visually inspected to remove segments with amplitudes greater than a preset value or artifact content greater than a preset value.

[0039] The segmented electromyographic signal was further divided into 10 time intervals, each lasting 100 milliseconds.

[0040] In one exemplary embodiment of this disclosure, the method further includes preprocessing the eye-tracking information as follows:

[0041] The missing data caused by blinking in the eye movement information is supplemented using linear interpolation.

[0042] Using the left-hand button press time as zero point, the pupil diameter data in the eye movement information is divided into 1000ms long segmented eye movement information.

[0043] The segmented eye-tracking information was further divided into 10 time periods, each lasting 100 milliseconds.

[0044] In one exemplary embodiment of this disclosure, the method further includes:

[0045] Data analysis and feature extraction were performed based on preprocessed EEG signals, 10 segmented EMG signals with a duration of 100 ms, and 10 eye movement data with a duration of 100 ms. This generated features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0046] In one aspect of this disclosure, a device for predicting error awareness is provided, comprising:

[0047] The information acquisition module is used to display a preset task to the test subject and collect the electroencephalogram (EEG) signal and the electromyogram (EMG) signal of the corrugator supercilii muscle when the test subject completes the preset task based on the EEG cap, and collect the eye movement information of the test subject when completing the preset task based on the eye tracker.

[0048] The feature extraction module is used to preprocess the EEG signals, EMG signals, and eye movement information, and then perform data analysis and feature extraction to generate features such as EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0049] The error consciousness prediction module is used to make a judgment based on the amplitude and latency features of the EEG signal, the eyebrow-raising electromyography features, the integrated electromyography value, the Lempel-Ziv complexity, and the pupil diameter features as inputs, and to complete the error consciousness prediction based on a preset error consciousness prediction model based on the XGBoost classifier.

[0050] In one aspect of this disclosure, an electronic device is provided, comprising:

[0051] Processor; and

[0052] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.

[0053] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to any one of the preceding claims.

[0054] An exemplary embodiment of this disclosure provides a method for predicting error awareness. The method includes: presenting a preset task to a test subject and collecting electroencephalogram (EEG) signals, electromyographic (EMG) signals of the corrugator supercilii muscle, and eye movement information of the test subject while completing the preset task; preprocessing the EEG signals, EMG signals, and eye movement information, performing data analysis and feature extraction to generate features of EEG signal amplitude and latency, EMG features of the corrugator supercilii muscle, integrated EMG value, Lempel-Ziv complexity, and pupil diameter; using the above features as input, performing a judgment based on a preset error awareness prediction model based on an XGBoost classifier to complete the error awareness prediction. This disclosure offers high prediction accuracy and can reduce the probability of human error while minimizing the impact on operational continuity.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0057] Figure 1 A flowchart of an error awareness prediction method according to an exemplary embodiment of the present disclosure is shown;

[0058] Figure 2 A schematic diagram of the hardware connection for synchronous acquisition of multiple physiological signals in an exemplary embodiment of the error awareness prediction method according to the present disclosure is shown.

[0059] Figure 3 A comparison chart of the effects of four error awareness prediction models according to an exemplary embodiment of the present disclosure is shown;

[0060] Figure 4 A schematic block diagram of an error awareness prediction device according to an exemplary embodiment of the present disclosure is shown;

[0061] Figure 5 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown; and

[0062] Figure 6 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0063] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0064] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0065] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0066] In this example embodiment, a method for predicting error awareness is first provided; refer to Figure 1 As shown, this error awareness prediction method may include the following steps:

[0067] Step S110: Show the preset task to the test subject, and collect the EEG signal and the electromyography signal of the corrugator supercilii muscle of the test subject when completing the preset task based on the EEG cap, and collect the eye movement information of the test subject when completing the preset task based on the eye tracker.

[0068] Step S120: After preprocessing the EEG signal, EMG signal, and eye movement information, data analysis and feature extraction are performed to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG value, Lempel-Ziv complexity, and pupil diameter features.

[0069] Step S130: Using the EEG signal amplitude and latency features, frowning electromyography features, integrated electromyography value, Lempel-Ziv complexity, and pupil diameter features as inputs, the system makes a judgment based on a preset error consciousness prediction model based on the XGBoost classifier to complete the error consciousness prediction.

[0070] An exemplary embodiment of this disclosure provides a method for predicting error awareness. The method includes: presenting a preset task to a test subject and collecting electroencephalogram (EEG) signals, electromyographic (EMG) signals of the corrugator supercilii muscle, and eye movement information of the test subject while completing the preset task; preprocessing the EEG signals, EMG signals, and eye movement information, performing data analysis and feature extraction to generate features of EEG signal amplitude and latency, EMG features of the corrugator supercilii muscle, integrated EMG value, Lempel-Ziv complexity, and pupil diameter; using the above features as input, performing a judgment based on a preset error awareness prediction model based on an XGBoost classifier to complete the error awareness prediction. This disclosure offers high prediction accuracy and can reduce the probability of human error while minimizing the impact on operational continuity.

[0071] The following will further explain a method for predicting error awareness in this example embodiment.

[0072] Example 1:

[0073] In step S110, a preset task can be presented to the test subject, and the electroencephalogram (EEG) signal and electromyogram (EMG) signal of the corrugator supercilii muscle can be collected based on the EEG cap when the test subject completes the preset task, and the eye movement information of the test subject when completing the preset task can be collected based on the eye tracker.

[0074] In this example embodiment, the preset task in the method includes:

[0075] Preset stimulus patterns are displayed on the computer screen in the form of a preset program;

[0076] Each stimulation pattern is a single colored Chinese character, and the color of each stimulation is one of green, red, yellow, blue, purple, or white.

[0077] When the font color of the Chinese character is inconsistent with the meaning of the Chinese character, the tester should press the D key on the keyboard with his left hand. When the same Chinese character appears twice in a row or the meaning of the Chinese character is consistent with the font color of the Chinese character, the tester should press the F key on the keyboard with his left index finger.

[0078] After the button press is completed, the screen will display a consciousness state evaluation interface with three preset expressions: a smiling face, a calm face, and a crying face. The test subject will subjectively evaluate the accuracy of the left-hand button press. If the test subject thinks the reaction is correct, the right ring finger will press the L key to correspond to the smiling face; if the test subject thinks the reaction is incorrect, the right index finger will press the J key to correspond to the crying face; if the test subject is unsure whether the reaction is correct or not, the right middle finger will press the K key to correspond to the calm face.

[0079] In this example embodiment, the preset stimulus pattern in the method includes:

[0080] A fixed white crosshair gaze point, lasting 500ms;

[0081] After the cross gaze point disappears, a colored Chinese character stimulus is displayed and lasts for 800ms;

[0082] After the tester pressed the button with their left hand, the stimulation interface disappeared and a 1000ms black screen appeared.

[0083] The consciousness state evaluation interface is displayed within 1000ms;

[0084] After the test subject presses the button with their right hand, the consciousness state evaluation interface disappears and a 600ms black screen appears.

[0085] In this example embodiment, the method further includes:

[0086] The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task. After collecting the eye movement information of the test subject when the test subject completed the preset task based on the eye tracker, the EEG, EMG, pupil diameter and behavioral data of the test subject were synchronized based on time tags.

[0087] In step S120, the electroencephalogram (EEG) signal, electromyogram (EMG) signal, and eye movement information can be preprocessed, followed by data analysis and feature extraction to generate features such as EEG signal amplitude and latency, frowning EMG features, integrated EMG value, Lempel-Ziv complexity, and pupil diameter features.

[0088] In this example embodiment, the method further includes preprocessing the EEG signal:

[0089] The EEG signal was referenced to the average value of the bilateral mastoid processes and bandpass filtered at 0.1 to 35 Hz to remove artifacts caused by high-frequency electromyography and slow voltage drift. The EEG signal was divided into time windows of 400 ms to 1000 ms with the left hand key press time as the zero point and then baseline correction was performed to generate segmented EEG signals.

[0090] The segmented EEG signals were visually inspected to remove segments with artifact content greater than a preset value, and spherical splines were used to interpolate the channels of segments with noise greater than a preset number.

[0091] Independent component analysis was performed on the segmented EEG signals to identify and remove components containing blinking / eye movement artifacts;

[0092] Extract the CRN and EPP event-related potential components from each segment of the segmented EEG signal data. CRN is the average amplitude within a window of 10-40ms after the left hand key press, and EPP is the average amplitude within a time window of 120-150ms after the left hand key press.

[0093] In this example embodiment, the method further includes preprocessing the electromyographic signal:

[0094] The electromyography signal was filtered using a 28-500Hz bandpass filter and a 50Hz notch filter, and then the signal was smoothed by moving the signal with a window width of 20 data points.

[0095] Based on the left-hand key press response time, the electromyographic signal is divided into 1000ms time windows and then baseline correction is performed to generate segmented electromyographic signals.

[0096] The segmented electromyographic signals are visually inspected to remove segments with amplitudes greater than a preset value or artifact content greater than a preset value.

[0097] The segmented electromyographic signal was further divided into 10 time intervals, each lasting 100 milliseconds.

[0098] In this example embodiment, the method further includes preprocessing the eye-tracking information as follows:

[0099] The missing data caused by blinking in the eye movement information is supplemented using linear interpolation.

[0100] Using the left-hand button press time as zero point, the pupil diameter data in the eye movement information is divided into 1000ms long segmented eye movement information.

[0101] The segmented eye-tracking information was further divided into 10 time periods, each lasting 100 milliseconds.

[0102] In this example embodiment, the method further includes:

[0103] Data analysis and feature extraction were performed based on preprocessed EEG signals, 10 segmented EMG signals with a duration of 100 ms, and 10 eye movement data with a duration of 100 ms. This generated features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0104] In step S130, the amplitude and latency features of the EEG signal, the eyebrow electromyography features, the integrated electromyography value, the Lempel-Ziv complexity, and the pupil diameter features can be used as inputs to make a judgment based on a preset error consciousness prediction model based on the XGBoost classifier, thereby completing the error consciousness prediction.

[0105] In this example embodiment, the method further includes an error awareness prediction model training step:

[0106] Show the preset task to the test subject and receive the key press information of the test subject after completing the preset task;

[0107] The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task, and the eye movement information of the test subject was collected based on the eye tracker when the test subject completed the preset task.

[0108] After preprocessing the EEG signals, EMG signals, and eye movement information, data analysis and feature extraction are performed to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

[0109] A mistaken awareness prediction model based on the XGBoost classifier was established, and the key information, EEG signal amplitude and latency features, frowning electromyography features, integral electromyography value, Lempel-Ziv complexity, and pupil diameter features were used as training data to train the mistaken awareness prediction model based on the XGBoost classifier.

[0110] Example 2:

[0111] In this example embodiment, the task design step uses a modified version of the error awareness task, where the task is presented as a program on a computer screen. Each stimulus is a single colored Chinese character, and the color of each stimulus is one of green, red, yellow, blue, purple, or white. Subjects are instructed to respond to each stimulus by pressing the "D" key on the keyboard as quickly and correctly as possible with their left middle finger when the font color and meaning of the Chinese character are inconsistent (i.e., the standard stimulus), and to respond by pressing the "F" key on the keyboard with their left index finger in the following two situations (i.e., the biased stimulus): the first situation is when the same Chinese character appears in two consecutive trials (repeated biased stimulus); the second situation is when the meaning of the Chinese character is consistent with its font color (consistent biased stimulus). After each stimulus and button response, three expressions (smiling, calm, and crying) will appear on the screen. At this time, the subject is asked to subjectively evaluate the accuracy of the response (i.e., error awareness evaluation). If the subject thinks the response is correct, the right ring finger presses the "L" key (corresponding to the smiling face); if the subject thinks the response is incorrect, the right index finger presses the "J" key (corresponding to the crying face); if the subject is unsure whether the response is correct or not, the right middle finger presses the "K" key (corresponding to the calm face).

[0112] Each Chinese character stimulus begins with a fixed white crosshair fixation point, lasting 500ms. Immediately after the crosshair fixation point disappears, a colored Chinese character stimulus appears and lasts for 800ms. Participants are instructed to respond to the stimulus by pressing designated keys on a keyboard as quickly and accurately as possible. After the response, the stimulus interface disappears, followed by a 1000ms black screen. A consciousness evaluation interface then appears for 1000ms, during which participants are asked to assess the accuracy of their response. After pressing a key, the interface disappears, followed by a 600ms black screen. The next trial then begins.

[0113] In this example embodiment, the physiological signal acquisition step involves measuring continuous electroencephalographic (EEG) activity of the cortex using 64 Ag / AgCl electrodes embedded in elastic Lycra caps. The signal is grounded to the forehead and referenced online to the FCz electrode. The sampling rate is set to 1000 Hz, and the resistance of the EEG signal is maintained below 5 kΩ.

[0114] Continuous electromyography (EMG) activity is recorded using one electrode on the EEG cap. The reference and ground electrodes are the same as those used for EEG acquisition. Participants are asked to furrow their brows; the recording electrode is placed at the point where the belly of the left corrugator supercilii muscle is most prominent during movement. Conductive gel is applied to the recording electrode before placement, and it is secured with medical tape.

[0115] Eye trackers were used to collect pupil diameter data, and pupil data for both eyes were recorded at a sampling rate of 150Hz. Eye tracking calibration was performed before the experiment.

[0116] During the mission, to simultaneously acquire EEG, brow muscle electromyography (EMG), and pupil diameter data, an event marker signal is sent from the same mission operating computer, and different acquisition devices receive the corresponding signals to ensure the temporal synchronization of different physiological signals. The marker signal is sent from the parallel port of the mission operating computer and simultaneously transmitted to different devices via a 1-to-2 parallel port. The hardware connection diagram is shown below. Figure 2 As shown. In terms of software, the Eprime extension for Tobii plugin is used and embedded into the E-prime software to achieve synchronization of EEG, brow twitching electromyography, pupil diameter, and behavioral data at the trial level using time tags.

[0117] In this example embodiment, the physiological data preprocessing step categorizes all response trials into six types based on response type: conscious error, uncertain error, unconscious error, conscious correctness, uncertain correctness, and unconscious correctness. Since correct responses do not affect task performance, conscious correctness, uncertain correctness, and unconscious correctness are combined and collectively referred to as "correct." Ultimately, four trial types are included in the analysis: conscious error, uncertain error, unconscious error, and correctness. To predict error awareness, physiological data from the trial preceding each of these response types is processed and analyzed.

[0118] EEG signals acquired from the scalp typically contain various noises and artifacts, necessitating preprocessing of the raw signals. The EEG signals were rereferenced to the average of bilateral mastoids and bandpass filtered from 0.1 to 35 Hz to remove artifacts caused by high-frequency electromyography (EMG) and slow voltage drift. Using the time between the incorrect and correct responses in each trial as zero, continuous EEG data was segmented into time windows from -400 ms to 1000 ms. Baseline correction was performed by subtracting the average amplitude from the 200 to 100 ms time window preceding the start of the response from each channel in each trial. After segmentation, segments with numerous artifacts were visually removed, and then spherical splines were used to interpolate channels with poor or high noise levels. Independent component analysis was then performed on the clean, segmented data to identify and remove components containing blink / eye movement artifacts or other artifacts. The CRN and EPP event-related potential components were extracted from each data segment. CRN was defined as the average amplitude within a window of 10-40 ms after the start of the reaction, and EPP was defined as the average amplitude within a time window of 120-150 ms after the start of the reaction.

[0119] For corrugator supercilii muscle data, the signal was filtered using a 28-500Hz bandpass filter and a 50Hz notch filter, and then smoothed using a window of 20 data points. Based on the time of the previous trial response before error / correctness, the continuous corrugator supercilii EMG signal was segmented into segments from the start of the response to 1000ms after the response. Baseline correction was performed on each segment by subtracting the average voltage value from 200ms to 100ms before the response. Since artifacts caused by eye movements or blinking are low-frequency waves, they are removed after bandpass filtering, so there is no need to perform artifact removal for eye movements / blinking. After segmentation, segments with excessively high amplitude or many artifacts were visually removed. Each 1000ms EMG signal segment was further divided into 10 time intervals of 100ms each.

[0120] The analysis of pupil diameter data first requires checking for data loss primarily caused by blinking, as the eye tracker temporarily loses visual contact with the pupil when the eyelids close. Data is then supplemented using linear interpolation. Subsequently, consecutive pupil diameter data are segmented into 1000ms segments, ranging from the start of the previous trial response before the error / correct response to 1000ms after the response. Next, the pupil diameters of both eyes are averaged, and each segment is further divided into ten 100ms intervals. Within each interval, the pupil diameter data for different response types are averaged.

[0121] In this example embodiment, the data analysis and feature extraction step extracts the ERP features, frowning electromyography (EMG) features, and pupil diameter features of the previous trial through single-trial analysis. The ERP features are the amplitude and latency of the CRN and EPP components at 61 electrode points (61×2×2=244). The frowning EMG features are the amplitude (average value every 100ms, 10), the integrated EMG value (average value every 100ms, 10), and the Lempel-Ziv complexity (average value every 100ms, 10). The pupil feature is the pupil diameter (average value every 100ms, 10), for a total of 284 features.

[0122] In this example embodiment, the error awareness prediction model construction steps involve performing principal component analysis on the aforementioned features to reduce dimensionality, remove redundant features, and prevent overfitting. Using reaction type as the label, the dimensionality-reduced features are trained and tested using Support Vector Machine (SVM), K-Nearest Neighbors, Decision Tree, and XGBoost classifiers, respectively, and the model is validated using five-fold cross-validation. The optimal classification performance of the above four methods is compared, such as... Figure 3As shown, the results indicate that XGBoost achieves the highest classification evaluation metrics, with an accuracy of 0.928, precision of 0.919, recall of 0.933, and F1 score of 0.925. This demonstrates that the proposed method exhibits excellent predictive ability for error awareness and possesses high application value.

[0123] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0124] Furthermore, in this example embodiment, an error awareness prediction device is also provided. (Refer to...) Figure 4 As shown, the error awareness prediction device 400 may include: an information acquisition module 410, a feature extraction module 420, and an error awareness prediction module 430. Wherein:

[0125] The information acquisition module 410 is used to display a preset task to the test subject and collect the electroencephalogram (EEG) signal and the electromyogram (EMG) signal of the corrugator supercilii muscle of the test subject when completing the preset task based on the EEG cap, and collect the eye movement information of the test subject when completing the preset task based on the eye tracker.

[0126] The feature extraction module 420 is used to preprocess the electroencephalogram (EEG) signal, electromyogram (EMG) signal, and eye movement information, and then perform data analysis and feature extraction to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG value, Lempel-Ziv complexity, and pupil diameter.

[0127] The error consciousness prediction module 430 is used to make a judgment based on the EEG signal amplitude and latency features, frowning electromyography features, integrated electromyography value, Lempel-Ziv complexity, and pupil diameter features as inputs, and to complete the error consciousness prediction based on a preset error consciousness prediction model based on XGBoost classifier.

[0128] The specific details of each of the above-mentioned error awareness prediction device modules have been described in detail in the corresponding error awareness prediction method, so they will not be repeated here.

[0129] It should be noted that although several modules or units of a fault awareness prediction device 400 have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0130] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0131] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”

[0132] The following reference Figure 5 To describe an electronic device 500 according to such an embodiment of the present invention. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0133] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.

[0134] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform actions such as... Figure 1 Steps S110 to S130 are shown in the diagram.

[0135] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.

[0136] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5203, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0137] Bus 550 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0138] Electronic device 500 can also communicate with one or more external devices 570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 550. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0139] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0140] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0141] refer to Figure 6 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0145] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0147] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0148] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for predicting error awareness, characterized in that, The method includes: The test subject is presented with a preset task, which is an error awareness task. The task presents colored Chinese character stimuli sequentially on the computer screen in the form of a preset program. Each stimulus pattern is a colored Chinese character. According to preset rules, trials in which the font color of the Chinese character is inconsistent with its meaning are set as standard stimuli. Trials in which the same meaning appears in two consecutive trials and / or trials in which the meaning of the Chinese character is consistent with the font color are set as deviation stimuli. The test subject is guided to respond using different keys of the left hand under the standard stimuli and deviation stimuli respectively. In each trial, after the test subject completes the left-hand key press response, a consciousness state evaluation interface is presented on the screen. The consciousness state evaluation interface includes at least three preset emoticon icons that respectively represent "thinking the response is correct", "uncertain" and "thinking the response is wrong". It receives the test subject's subjective evaluation of the accuracy of the response made just now by pressing the right-hand key, so as to obtain the error consciousness evaluation result of the corresponding trial. The test subjects' electroencephalogram (EEG) signals and corrugator supercilii (CSU) signals were collected using an EEG cap while completing the preset task. Eye movement information of the test subjects while completing the preset task was collected using an eye tracker. The eye movement information included pupil diameter. The test subjects' EEG, CSU, pupil diameter and behavioral data were synchronized based on time tags. The EEG signals, EMG signals, and eye movement information are preprocessed. The EEG signals, EMG signals, and eye movement information of the previous trial before the incorrect response and / or correct response are segmented and artifacts are removed according to a predetermined time window. Data analysis and feature extraction are then performed to generate multimodal features, including EEG signal amplitude and latency features, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features. The EEG signal amplitude and latency features include at least the amplitude features of CRN and EPP event-related potential components. Based on the objective response results of each trial and the error awareness evaluation results, each trial is initially divided into six response types: conscious error, uncertain error, unconscious error, conscious correctness, uncertain correctness, and unconscious correctness. The conscious correctness, uncertain correctness, and unconscious correctness are then combined into the correct response type to obtain four target response types: conscious error, uncertain error, unconscious error, and correctness. For each target trial, the amplitude and latency of the CRN and EPP event-related potential components from the previous trial at 61 electrode points were extracted to form 244 EEG features. The brow furrow electromyography (EMG) signal from the start of the previous trial to 1000ms after the reaction was divided into 10 100ms time intervals, and the EMG amplitude, integrated EMG value, and Lempel-Ziv complexity of each time interval were extracted to form 30 brow furrow EMG features. The pupil diameter data of both eyes from the start of the previous trial to 1000ms after the reaction were averaged and divided into 10 100ms time intervals, and the average pupil diameter of each time interval was extracted to form 10 pupil diameter features. The 244 EEG features, 30 brow furrow EMG features, and 10 pupil diameter features constitute a 284-dimensional multimodal feature. Principal component analysis was performed to reduce the dimensionality of the 284-dimensional multimodal features. The four target response types were used as labels, and the dimensionality-reduced multimodal features were used as input to train an error awareness prediction model based on the XGBoost classifier. The multimodal features of the previous trial of the target trial to be predicted are input into the trained error awareness prediction model, and the prediction result is outputting that the target trial to be predicted belongs to one of the following: a conscious error, an uncertain error, an unconscious error, or a correct error.

2. The method as described in claim 1, characterized in that, The preset tasks in the method include: Preset stimulus patterns are displayed on the computer screen in the form of a preset program; Each stimulation pattern is a single colored Chinese character, and the color of each stimulation is one of green, red, yellow, blue, purple, or white. When the font color of the Chinese character is inconsistent with the meaning of the Chinese character, the tester should press the D key on the keyboard with his left hand. When the same Chinese character appears twice in a row or the meaning of the Chinese character is consistent with the font color of the Chinese character, the tester should press the F key on the keyboard with his left index finger. After the button press is completed, the screen will display a consciousness state evaluation interface with three preset expressions: a smiling face, a calm face, and a crying face. The test subject will subjectively evaluate the accuracy of the left-hand button press. If the test subject thinks the reaction is correct, the right ring finger will press the L key to correspond to the smiling face; if the test subject thinks the reaction is wrong, the right index finger will press the J key to correspond to the crying face; if the test subject is unsure whether the reaction is correct or not, the right middle finger will press the K key to correspond to the calm face.

3. The method as described in claim 2, characterized in that, The preset stimulus patterns in the method include: A fixed white crosshair gaze point, lasting 500 ms; After the cross gaze point disappears, a colored Chinese character stimulus is displayed and lasts for 800 ms; After the tester pressed the button with their left hand, the stimulation interface disappeared and a 1000ms black screen appeared; The consciousness state evaluation interface is displayed for 1000 ms; After the test subject presses the button with their right hand, the consciousness state evaluation interface disappears and a 600ms black screen appears.

4. The method as described in claim 3, characterized in that, The method also includes a training step for an error awareness prediction model: Show the preset task to the test subject and receive the key press information of the test subject after completing the preset task; The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task, and the eye movement information of the test subject was collected based on the eye tracker when the test subject completed the preset task. After preprocessing the EEG signals, EMG signals, and eye movement information, data analysis and feature extraction are performed to generate features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features. A mistaken awareness prediction model based on the XGBoost classifier was established, and the key information, EEG signal amplitude and latency features, frowning electromyography features, integral electromyography value, Lempel-Ziv complexity, and pupil diameter features were used as training data to train the mistaken awareness prediction model based on the XGBoost classifier.

5. The method as described in claim 1, characterized in that, The method further includes: The EEG signals and EMG signals of the corrugator supercilii muscle of the test subject were collected based on the EEG cap when the test subject completed the preset task. After collecting the eye movement information of the test subject when the test subject completed the preset task based on the eye tracker, the EEG, EMG, pupil diameter and behavioral data of the test subject were synchronized based on time tags.

6. The method as described in claim 1, characterized in that, The method further includes preprocessing the electroencephalogram (EEG) signal as follows: The EEG signal was referenced to the average value of the bilateral mastoid processes and bandpass filtered at 0.1~35Hz to remove artifacts caused by high-frequency electromyography and slow voltage drift. The EEG signal was divided into time windows of 400ms to 1000ms with the left hand key press time as the zero point and then baseline correction was performed to generate segmented EEG signals. The segmented EEG signals were visually inspected to remove segments with artifact content greater than a preset value, and spherical splines were used to interpolate the channels of segments with noise greater than a preset number. Independent component analysis was performed on the segmented EEG signals to identify and remove components containing blinking / eye movement artifacts; Extract the CRN and EPP event-related potential components from each segment of the segmented EEG signal data. CRN is the average amplitude within a window of 10-40ms after the left hand key press, and EPP is the average amplitude within a time window of 120-150ms after the left hand key press.

7. The method as described in claim 6, characterized in that, The method further includes preprocessing the electromyographic signal as follows: The electromyographic signal was filtered using a 28-500 Hz bandpass filter and a 50 Hz notch filter, and then the signal was smoothed by moving the filter with a window width of 20 data points. Based on the left-hand key press response time, the electromyographic signal is divided into 1000ms time windows and then baseline correction is performed to generate segmented electromyographic signals. The segmented electromyographic signals are visually inspected to remove segments with amplitudes greater than a preset value or artifact content greater than a preset value. The segmented electromyographic signal was further divided into 10 time intervals, each lasting 100 milliseconds.

8. The method as described in claim 7, characterized in that, The method further includes preprocessing the eye-tracking information as follows: The missing data caused by blinking in the eye movement information is supplemented using linear interpolation. Using the left-hand button press time as zero point, the pupil diameter data in the eye movement information is divided into 1000ms long segmented eye movement information. The segmented eye-tracking information was further divided into 10 time periods, each lasting 100 milliseconds.

9. The method as described in claim 8, characterized in that, The method further includes: Data analysis and feature extraction were performed based on preprocessed EEG signals, 10 segmented EMG signals with a duration of 100 ms, and 10 eye movement data with a duration of 100 ms. This generated features of EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features.

10. A device for predicting error awareness, characterized in that, Based on the method according to any one of claims 1-9, the apparatus comprises: The information acquisition module is used to display a preset task to the test subject and collect the electroencephalogram (EEG) signal and the electromyogram (EMG) signal of the corrugator supercilii muscle when the test subject completes the preset task based on the EEG cap, and collect the eye movement information of the test subject when completing the preset task based on the eye tracker. The feature extraction module is used to preprocess the EEG signals, EMG signals, and eye movement information, and then perform data analysis and feature extraction to generate features such as EEG signal amplitude and latency, frowning EMG features, integrated EMG values, Lempel-Ziv complexity, and pupil diameter features. The error consciousness prediction module is used to make a judgment based on the amplitude and latency features of the EEG signal, the eyebrow-raising electromyography features, the integrated electromyography value, the Lempel-Ziv complexity, and the pupil diameter features as inputs, and to complete the error consciousness prediction based on a preset error consciousness prediction model based on the XGBoost classifier.

11. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images