Error and error awareness warning method and device based on multiple physiological characteristics

Abstract

The present invention discloses a method and device for early warning of errors and error awareness based on multiple physiological characteristics, wherein the method comprises the following steps: synchronously collecting EEG data, frowning electromyography data, eye movement data and facial micro-expression data of operators; performing data processing on the synchronously collected data, extracting EEG features, frowning electromyography features, eye movement features and facial expression features, and fusing the extracted EEG features, frowning electromyography features, eye movement features and facial expression features to generate a physiological behavior feature set; based on the multi-stage and multi-type prediction of the physiological behavior feature set, obtaining the error stage and error awareness type of the operator, and matching the error alarm level and alarm action corresponding to the current error and error awareness based on the error stage and error type, so as to perform error early warning. The method can effectively improve the accuracy of early warning of errors and error awareness through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

Description

Technical Field

[0001] The present invention relates to the field of life science technology, and in particular to an error and error awareness early warning method and device based on multiple physiological characteristics. Background Art

[0002] Behavior monitoring, which functions to evaluate actions and allow for flexible adjustment of behavior, is a fundamental mechanism of cognitive control. Behavior monitoring is associated with activity in the medial frontal cortex, particularly the anterior cingulate cortex. This mechanism includes not only the identification of behavioral errors or conflicts, but also the subsequent adjustment of behavior to correct any such identified problems and make the behavior more consistent with intention.

[0003] Therefore, the engagement and efficiency of behavioral monitoring are crucial to reducing the likelihood of error repetition. Brief disengagement or inefficiency in behavioral monitoring has been identified as one of the factors leading to reduced accuracy of subsequent responses. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, one purpose of the present invention is to propose an error and error awareness warning method based on multiple physiological characteristics, which can effectively improve the accuracy of error and error awareness warnings through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

[0006] Another object of the present invention is to provide an error and error awareness warning device based on multiple physiological characteristics.

[0007] To achieve the above-mentioned purpose, an embodiment of the present invention proposes a method for warning of errors and error awareness based on multiple physiological characteristics, including the following steps: synchronously collecting EEG data, frowning EMG data, eye movement data and facial micro-expression data of the operator; processing the synchronously collected EEG data, frowning EMG data, eye movement data and facial micro-expression data to extract EEG features, frowning EMG features, eye movement features and expression features, and fusing the extracted EEG features, frowning EMG features, eye movement features and expression features to generate a physiological behavior feature set; based on the physiological behavior feature set, multi-stage and multi-type predictions are made to obtain the error stage and error awareness type of the operator, and based on the error stage and the error type, the error alarm level and alarm action corresponding to the current error and error awareness are matched to perform an error warning.

[0008] The error and error awareness warning method based on multiple physiological characteristics in the embodiment of the present invention can predict errors and error awareness in the behavior of operators based on the processing and analysis results of synchronously collected EEG data, frown electromyography data, eye movement data and facial micro-expression data, and can realize error warning, so that the accuracy of error and error awareness warning can be effectively improved through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

[0009] In addition, the error and error awareness warning method based on multiple physiological characteristics according to the above embodiment of the present invention may also have the following additional technical features:

[0010] Furthermore, the multi-stage and multi-type prediction based on the physiological behavior feature set to obtain the error stage and error consciousness type of the operator also includes: when the time corresponding to the physiological behavior feature set is in a first interval, determining the error stage as an error trial stage; when the time corresponding to the physiological behavior feature set is in a second interval, determining the error stage as a pre-error trial stage; when the time corresponding to the physiological behavior feature set is in a third interval, determining the error stage as a pre-error trial stage, wherein the minimum value of the first interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the third interval.

[0011] Furthermore, the operator's error consciousness type is obtained based on the multi-stage and multi-type prediction of the physiological behavior feature set, including: when the feature value of the behavior feature set is greater than a first threshold, the error consciousness type is determined to be an aware of error type; when the feature value of the behavior feature set is less than a second threshold, the error consciousness type is determined to be an unaware of error type, wherein the first threshold is greater than the second threshold; when the feature value of the behavior feature set is less than or equal to the first threshold and greater than or equal to the second threshold, the error consciousness type is determined to be an uncertain type.

[0012] Furthermore, it also includes: obtaining the physiological behavioral characteristics of the trials before the errors of different work operation results in the test data as a basic threshold; obtaining the physiological behavioral characteristics of the operator after the work operation and the physiological behavioral characteristics of the previous trial, and performing AI training based on the physiological behavioral characteristics after the work operation, the physiological behavioral characteristics of the previous trial and the basic threshold to adjust the physiological behavioral characteristic threshold.

[0013] Furthermore, before obtaining the error stage and error awareness type of the operator based on the multi-stage and multi-type prediction of the physiological and behavioral feature set, it also includes: predicting the next trial error and error state based on a machine learning method based on a single trial analysis; extracting the previous trial physiological and behavioral features corresponding to different error intention states in a single trial, and using multiple prediction models to train the previous trial physiological and behavioral features one by one to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type prediction.

[0014] To achieve the above-mentioned purpose, on the other hand, an embodiment of the present invention proposes an error and error awareness warning device based on multiple physiological characteristics, including: an acquisition module, used to synchronously collect the EEG data, frowning EMG data, eye movement data and facial micro-expression data of the operator; a processing module, used to process the synchronously collected EEG data, frowning EMG data, eye movement data and facial micro-expression data, extract EEG features, frowning EMG features, eye movement features and expression features, and fuse the extracted EEG features, frowning EMG features, eye movement features and expression features to generate a physiological behavior feature set; a prediction module, used to obtain the error stage and error awareness type of the operator based on multi-stage and multi-type predictions of the physiological behavior feature set; an alarm module, used to match the error alarm level and alarm action corresponding to the current error and error awareness based on the error stage and the error type, so as to provide an error warning.

[0015] The error and error awareness warning device based on multiple physiological characteristics in the embodiment of the present invention can predict errors and error awareness in the behavior of operators based on the processing and analysis results of synchronously collected EEG data, frown electromyography data, eye movement data and facial micro-expression data, and can realize error warning, so that the accuracy of error and error awareness warning can be effectively improved through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

[0016] In addition, the error and error awareness warning device based on multiple physiological characteristics according to the above embodiment of the present invention may also have the following additional technical features:

[0017] Furthermore, the prediction module also includes: a first-stage prediction unit, used to determine that the error stage is an error trial stage when the time corresponding to the physiological behavior feature set is in a first interval; a second-stage prediction unit, used to determine that the error stage is a pre-error trial stage when the time corresponding to the physiological behavior feature set is in a second interval; a third-stage prediction unit, used to determine that the error stage is a pre-error trial stage when the time corresponding to the physiological behavior feature set is in a third interval, wherein the minimum value of the first interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the third interval.

[0018] Furthermore, the prediction module also includes: a first type prediction unit, used to determine the error awareness type as a conscious error type when the characteristic value of the behavior feature set is greater than a first threshold; a second type prediction unit, used to determine the error awareness type as an unaware error type when the characteristic value of the behavior feature set is less than a second threshold, wherein the first threshold is greater than the second threshold; a third type prediction unit, used to determine the error awareness type as an uncertain type when the characteristic value of the behavior feature set is less than or equal to the first threshold and greater than or equal to the second threshold.

[0019] Furthermore, it also includes: a threshold adjustment module, which is used to obtain the physiological behavior characteristics of the trials before the errors of different work operation results in the test data as a basic threshold; obtain the physiological behavior characteristics of the operator after the work operation and the physiological behavior characteristics of the previous trial, and perform AI training based on the physiological behavior characteristics after the work operation, the physiological behavior characteristics of the previous trial and the basic threshold to adjust the physiological behavior characteristic threshold.

[0020] Furthermore, it also includes: a modeling module, which is used to predict the error and error state of the next trial based on a machine learning method of single trial analysis, extract the physiological and behavioral characteristics of the previous trial corresponding to different error intention states in a single trial, and use multiple prediction models to train the physiological and behavioral characteristics of the previous trial one by one to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type predictions.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 is a flow chart of a method for early warning of errors and error awareness based on multiple physiological characteristics according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of hardware connection for synchronously collecting multiple physiological signals according to an embodiment of the present invention;

[0025] Figure 3 A schematic diagram showing comparison of optimal classification effects of four classifiers according to an embodiment of the present invention;

[0026] Figure 4Schematic diagram of a block diagram of an error and error awareness warning device based on multiple physiological characteristics according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0028] The present invention is based on the inventor's understanding and discovery of the following problems:

[0029] Human behavior may be determined by ongoing dynamic activity in the brain. Although it is conceivable that the brain's behavioral monitoring patterns before the execution of a behavior are causally related to the behavioral outcome, most previous studies on behavioral monitoring have focused on brain activity triggered by behavioral errors and the subsequent adaptive compensation mechanisms. A few studies have also explored whether brain activity before an error predicts subsequent errors. The extent to which changes in brain activity before an error can predict errors is uncertain, but it has important theoretical and practical significance. Some event-related potential studies have shown that the efficiency of response monitoring before an error is lower than that before a correct response. The amplitude modulation of the two components of the response-evoked event-related potential, CRN (Correct response-related negativity) and EPP (Error-preceding positivity), is thought to reflect fluctuations in the efficiency of the behavior monitoring system.

[0030] First, the CRN is a negative deflection that occurs immediately after a correct response (approximately 50 ms). The CRN has similar morphology and scalp topography to the ERN, so it can be argued that the two components may reflect the activity of the same response monitoring system and that the CRN can serve as an indicator of response monitoring activity in the ACC during correct trials. Under the assumption that the CRN monitors behavior, instantaneous changes in the efficiency of the behavior monitoring system should be reflected in changes in the amplitude of the CRN and in changes in accuracy on subsequent trials. In fact, several studies have reported that the average CRN amplitude is higher in trials preceding correct responses than in trials preceding errors. In the related art, pre-error trials were associated with an increase in positive waves in the response-locked ERP within a similar time window as the ERN. Subsequent studies in the related art have replicated this pattern of findings, reporting CRNs that are not very negative in amplitude (positive values) in trials preceding error responses, and proposed that this pre-error positive wave reflects a decrease in the CRN.

[0031] These studies suggest that modulations in CRN amplitude may reflect the engagement of the behavioral monitoring system and may be associated with subsequent response accuracy. A second ERP component that has been found to be associated with trials preceding an error response is the EPP, a positive deflection generated by the pericingulate anterior cingulate cortex. This positive deflection peaks between 50 and 300 ms after the response preceding the error and is observed over midline electrodes. However, the EPP has been poorly investigated to date, and the few studies that have addressed this component have produced conflicting results. The EPP has been interpreted as reflecting a momentary disengagement of the behavioral monitoring system to detect subthreshold conflicts or erroneous response tendencies. Several studies have shown that the EPP amplitude is larger in trials preceding errors than in trials preceding correct responses. However, other studies have reported contrasting results, suggesting that EPP amplitude modulation may be task-specific and particularly sensitive to conflict.

[0032] Although there are some studies that predict subsequent errors by brain activity before an error, few studies have explored whether the effect of brain activity before an error on subsequent errors is related to the state of consciousness of the subsequent error involved. Some researchers believe that in repetitive and monotonous manual response inhibition tasks, action errors may arise from failures in response inhibition or failures in sustained attention. These two possible mechanisms have very different relationships with consciousness: unaware errors are pure failures in sustained attention, while aware errors are mainly due to failure to inhibit erroneous responses, although aware errors may also involve at least part of the failure of sustained attention. Related technical studies believe that errors in simple repetitive forced-choice tasks may have two main non-exclusive causes: first, they are caused by transient fluctuations in neural activity during stimulus processing or response selection in some trials. Second, a certain proportion of errors may be caused by systematic disorders in cognitive control systems, which develop more slowly over time, and there may be a certain trend of trial-to-trial changes before the erroneous response. At the trial level, as for the possible causes of these two errors, if the error is caused by instantaneous fluctuations in neural activity, then this fluctuation is most likely to have affected the trial before the error. If it affected the previous trials, the instantaneous fluctuations in neural activity would have been adjusted long ago and no error would have occurred at all. If the error is caused by a systematic disorder in the cognitive control system and the changes are slow and there is a certain trend of deterioration, then the trial before the error should be the trial with the most serious systematic disorder, and the effect of predicting subsequent errors will be most obvious at this time.

[0033] In this context, the main goal of the current study was to explore whether realized errors, uncertain errors, unrealized errors, and correct responses could be distinguished based on the behavioral monitoring mechanisms activated in the trial preceding an error response. More specifically, it was to investigate whether changes in brain activity in the trial preceding an error were associated with only a certain type of error or with multiple errors (general errors).

[0034] Most of the understanding of temporal aspects of human visual processing comes from ERP studies, which examine brain function by manipulating stimulus properties and task demands. ERPs are direct electrophysiological indicators of neuronal activity with high temporal resolution. ERPs are extracted by time-locking and averaging EEG signals, which are small relative to background EEG activity, which is typically removed as noise in traditional analysis. The amplitude of phase-locked ERPs is typically only a few microvolts, while the amplitude of the ongoing background EEG is about 100 microvolts. This in turn means that a large part of the variation in the measured EEG signal cannot be explained by the stimulus. This is one possible reason why visual perception can be highly variable given a set of constant parameters of the stimulus. The results in the previous chapter of this paper demonstrate that there are differences in the processing of constant parameters of the stimulus, which in turn induces differences in false consciousness states. Since the visual stimulus itself is constant, this variability in consciousness is likely related to changes in the background EEG of the brain.

[0035] Trial-to-trial background EEG variations are often considered noise and are therefore often largely ignored. In recent years, such changes in neuronal responses have attracted increasing attention because the information they carry can provide new insights into brain function. A classic view of brain function is that the brain is an input-output system driven by external stimuli, and this view can be tested by studying how the physical properties of the stimulus and the task demands are reflected in the neural responses elicited. However, the anatomical connections between visual areas are reciprocal, and information not only propagates upward in a bottom-up manner, but also higher-order areas also exert top-down influences on lower-level areas. The relative timing or causal relationship between top-down and bottom-up processes remains a matter of debate. In addition to visual areas along the dorsal and ventral sides, non-visual brain areas in the parietal and frontal cortices also contribute to the understanding of visual input. However, functional hierarchy does not necessarily imply temporal hierarchy. For example, when stimulus identification becomes difficult, higher-level orbitofrontal areas can function before lower-level visual areas. Some experiments have detected signals related to conscious perception even before the presentation of visual stimuli. For example, alpha power before stimulus onset is associated with increased visual detection effects, and another example is the phase of EEG slow waves associated with increased cortical excitability. Similarly, functional magnetic resonance imaging (fMRI) studies have documented sustained activity in the medial dorsal frontal-parietal intracortical network before stimulus onset.

[0036] It can be seen that the brain activity that produces conscious perception signals before the stimulus appears is largely related to the EEG activity before the stimulus. It can also be said that the preparation before the stimulus and the upcoming stimulus together trigger the activation of the brain and thus produce conscious perception. Since the corrugator muscle activity and pupil diameter can indirectly reflect brain activity, there may also be differences in the corrugator muscle activity and pupil diameter before the stimulus in different consciousness states.

[0037] At present, there are no relevant research and patents on error types, error awareness warning technology and systems at home and abroad. The present invention proposes a method and device for warning errors and error awareness based on multiple physiological characteristics, including: error awareness classification technology, error and error awareness prediction technology, and error and error awareness warning technology. Among them, the error awareness classification technology divides the operation results into four types of error consciousness and operation based on the degree of awareness of the personnel on the operation results, namely, conscious errors, uncertain pre-errors, unaware errors and correct errors. The error and error awareness prediction technology is based on the analysis and research of the EEG, corrugator muscle and eye movement data of the pre-error trials corresponding to the four types of error consciousness and operation, extracts specific indicators, and combines single extraction with machine algorithms to realize the prediction of errors and error consciousness. The error and error awareness warning technology is based on the actual application needs, divides the four types of error consciousness and operation into different (three) warning levels, and uses different warning methods (sound, vision, touch) to distribute information to relevant personnel.

[0038] The following will describe the error and error awareness warning method and device based on multiple physiological characteristics proposed in an embodiment of the present invention with reference to the accompanying drawings. First, the error and error awareness warning method based on multiple physiological characteristics proposed in an embodiment of the present invention will be described with reference to the accompanying drawings.

[0039] Figure 1 It is a flow chart of an error and error awareness warning method based on multiple physiological characteristics according to an embodiment of the present invention.

[0040] like Figure 1 As shown, the error and error awareness warning method based on multiple physiological characteristics includes the following steps:

[0041] In step S101, the operator's EEG data, frown EMG data, eye movement data and facial micro-expression data are synchronously collected.

[0042] It is understandable that the embodiments of the present invention can collect physiological behavior signals such as EEG, corrugator muscle, eye movement information, facial micro-expressions, etc. when the operator is performing a task.

[0043] Specifically, 1.1 EEG data acquisition

[0044] Continuous EEG activity of the cortex can be measured using 64 Ag / AgCl electrodes embedded in an elastic Lycra cap. The EEG signal was amplified using a BrainAmp amplifier, and the signal was grounded to the forehead (GND) and referenced online to the FCz electrode. The continuous EEG signal was recorded using software, and the sampling rate was set to 1000Hz, and the resistance of the EEG signal was kept below 5kΩ.

[0045] 1.2 Frowning electromyography data collection

[0046] Use an Ag / AgCl electrode on the EEG cap to record continuous EMG activity. The reference electrode and ground electrode are consistent with EEG acquisition. Ask the participants to frown, and the most protruding position of the left corrugator muscle belly during muscle movement is the recording electrode placement position. Before placing the recording electrode, apply conductive paste and fix it with used medical tape.

[0047] 1.3 Eye Movement Data Collection

[0048] Eye tracking was performed using an eye tracker to collect eye movement data, and the acquisition software was used for acquisition. During acquisition, pupil data of both eyes can be recorded at a sampling rate of 150 Hz, and eye movement calibration was performed before the experiment. The marker information of task events and continuous eye movement data information were exported to xlsx format for further processing in Matlab.

[0049] 1.4 Facial micro-expression collection

[0050] A high-definition camera is used to collect the micro-expressions of operators during the entire task. A synchronous interface module has been developed for this camera.

[0051] 1.5 Synchronous collection of EEG, frown EMG, eye movement and facial micro-expression data

[0052] When synchronously collecting EEG, frown electromyography, eye movement and facial micro-expression data in a task, the same task operation computer sends out event marker signals, and different acquisition devices receive corresponding signals, thus ensuring the time synchronization of different physiological signals. In terms of hardware, a one-to-two parallel port cable is required. The marker signal is sent from the parallel port of the task operation computer and sent to different devices simultaneously through the one-to-two parallel ports. The hardware connection diagram is as follows: Figure 2 As shown. In terms of software, you can use the Eprime extention for Tobii plug-in developed by Tobii and PST, embed it into the E-prime software, and use time tags to synchronize the EEG, frown electromyography, eye movement data, and behavioral data (reactions and reaction times) at the trial level.

[0053] During the signal collection process, the participants sat about 70 cm away from the eye tracker and the stimulus display screen. In order to reduce the influence of head movement artifacts and prevent the eye position from exceeding the tracking range of the eye tracker, the participants were required to keep their heads as still as possible during the experiment.

[0054] In step S102, the synchronously collected EEG data, frowning EMG data, eye movement data and facial micro-expression data are processed to extract EEG features, frowning EMG features, eye movement features and expression features, and the extracted EEG features, frowning EMG features, eye movement features and expression features are fused to generate a physiological behavior feature set.

[0055] It can be understood that the embodiments of the present invention can analyze the previous characteristics of the four operation results based on different classifications of operation results, and extract EEG power spectrum, EEG ERP, time-frequency features, frown EMG amplitudes in multiple time periods, IEMG features (the time periods can be set to: 0-100ms, 100-200ms, 200-300ms, 300-400ms, 400-500ms, 500-600ms, 600-700ms, 700-800ms, 800-900ms, 900-1000ms), eye movement data in multiple time periods, facial micro-expression features, etc.

[0056] Specifically, 2.1 EEG data processing and analysis

[0057] ERP analysis: EEG signals were analyzed offline using Matlab 2013b software and the eeglab toolbox. EEG signals were re-referenced to the average of the bilateral mastoids (TP9, TP10). Bandpass filtering was performed with a FIR filter at 0.1-35 Hz (12 dB / oct) to remove artifacts caused by high-frequency electromyography and slow voltage drift. Continuous EEG data were segmented into time windows from -400 ms to 1000 ms based on the time of the response before the error. For each participant, the response-locked ERP (CRN and EPP) was averaged across conscious errors, uncertain errors, unrealized errors, and correct errors. Baseline correction was performed by subtracting the average voltage in the time window from 200 to 100 ms before the onset of the response on each channel in each trial. The segmented signals were visually inspected to remove artifacts, and spherical splines were then used to interpolate the channels with poor or noisy behavior. The clean segmented data were then used for independent component analysis (ICA). 61 ICA components were identified for each participant's EEG signal. The morphology, time course, and spectral characteristics of the ICA scalp were visually inspected to identify and remove components containing blink / oculomotor artifacts or other artifacts. The CRN was defined as the average amplitude in the window within 10-40ms after the onset of the response. The EPP was defined as the average amplitude in the time window 120-150ms after the onset of the response. The CRN amplitude was extracted from the three obvious frontal-central sites (Fz, FCz, Cz). For the EPP, the amplitudes of the three most obvious frontal-central electrode positions (FCz, Cz, CPz) were extracted.

[0058] For the CRN and EPP, the two within-subject factors were response type (realized pre-error, uncertain pre-error, unrealized pre-error, and correct pre-error) and electrode (Fz, FCz, Cz or FCz, Cz, CPz). When the main effects or their interactions were significant, P values ​​were corrected for post hoc comparisons using the Greenhouse-Geisser method. For all statistical calculations, partial Eta squared (η p 2) The effect size is expressed, and simple effect analysis is used when the interaction is significant.

[0059] The average number of available trials for realized errors, uncertain errors, unrealized errors, and correct responses was 176, 16, 41, and 220, respectively. Two-way repeated measures ANOVA (response type × individual) was used to compare the inter-individual differences in the number of trials for each response type.

[0060] Time-frequency analysis: For each trial, time-frequency analysis was performed in the time domain from -400 to 1000 ms and in the frequency domain from 1 to 35 Hz (with a step of 1 Hz) using MATLAB R2013b. The time-frequency decomposition of each segment was performed by window Fourier transform using a Hanning window with a fixed window width of 200 ms. Such a window width allows a good balance between time resolution and frequency resolution in the frequency range explored. In order to avoid the influence of edge artifacts, baseline correction was performed at each frequency using the power spectral density from -300 to -100 ms before the response. The method used for baseline correction was the subtraction method, and its formula is:

[0061] P s (t,f)=P(t,f)-R(f),

[0062] Where R(f) is the average power spectral density of the signal during the baseline period before the response, and P(t,f) is the power spectral density at each time-frequency point. The subtractive baseline correction method has been shown to be more effective in avoiding positive bias than the percentage method.

[0063] When the original power spectral density is converted into ERSP in time-frequency representation, an exploratory data-driven approach is used to identify spatial regions of interest (spatial regions of interest) and time-frequency regions of interest (time-frequency regions of interest). The exploratory data-driven analysis process is as follows:

[0064] First, by calculating the differences in the time-frequency topographic maps corresponding to different types of errors (corrections) on all electrodes, several time-frequency regions of interest related to error perception processing (with large differences between groups) were roughly identified.

[0065] Second, based on the defined time-frequency region of interest, the time-frequency plane averages of different types of responses in the specific time-frequency region of interest were calculated, and the results corresponding to all electrodes were plotted into a scalp topography map. Based on the difference map of scalp topography, the frontocentral region [(Fz+FCz+FC1+FC2) / 4] was determined as the spatial region of interest.

[0066] Third, based on the defined spatial regions of interest, ERSP differences were calculated for the time-frequency representations of the different response types. In this case, the theta / alpha (4-13 Hz, 0-380 ms) and alpha (8-13 Hz, 550-950 ms) frequency bands were selected as the time-frequency regions of interest. Based on the defined time-frequency regions of interest, the mean oscillation amplitudes within the specific time-frequency regions of interest for the four types of responses were analyzed by variance analysis, and post hoc comparisons were performed using the LSD method.

[0067] Power spectrum analysis: The time period within 500ms before the onset of stimulation was selected, and the absolute power spectrum of the EEG before stimulation corresponding to the different consciousness states was calculated using the Welch method in Matlab2013b software. A non-overlapping Hamming window with a window length of 1 second was used. The power spectrum density was calculated in the δ (1-4Hz), θ (4-8Hz), α (8-13Hz), β (13-30Hz), and γ (30-48Hz) frequency bands. The power spectrum density value was normalized using decibel (dB) conversion. The scalp electrodes were divided into four regions of interest, including the frontal area (F1, Fz, F2), the central area (C1, Cz, C2), the parietal area (P1, Pz, P2) and the occipital area (O1, Oz, O2). The power spectrum density in each region of interest was averaged between electrodes.

[0068] Two-way repeated measures ANOVA (region of interest × response type) was performed within each frequency band. When main effects or their interactions were significant, P values ​​were corrected for post hoc comparisons using the Greenhouse-Geisser method. For all statistical calculations, partial Eta squared (η p 2 ) represents the effect size, and simple effect analysis was used when the interaction was significant.

[0069] 2.2 Frown EMG data processing and analysis

[0070] Since the spectrum of frown EMG is mainly distributed between 28-500Hz. According to the Nyquist sampling theorem, the sampling rate must be greater than twice the highest frequency component of interest of the measured signal. Therefore, the frown EMG signal is first upsampled to 1024Hz, and then the frown EMG signal is filtered offline using a FIR filter with a 28-500Hz bandpass and a 50Hz notch filter. Then, the signal is moved and smoothed using the smoothdata function in Matlab with a window width of 20 data points. Based on the reaction time, the continuous frown EMG signal is divided into segments from the beginning of the reaction to 1000ms after the reaction. The baseline correction is performed on each segment by subtracting the average voltage value from 200ms to 100ms before the reaction. Since the artifacts caused by eye movement or blinking are low-frequency waves, the relevant artifacts will be removed after bandpass filtering, so there is no need to remove the artifacts caused by eye movement / blinking. The segmented signal is visually inspected to delete segments with too high amplitude or more artifacts. The processed segmented corrugator electromyographic signals were averaged within each response type in each subject. According to the processing methods of previous similar related studies, the post-response electromyographic signals of different response types on the corrugator supercilii were divided into time periods of 100 milliseconds. The data of different time periods were first Z-standardized and then the average amplitude of each time period was calculated.

[0071] In addition, since the electromyographic signal can be approximately regarded as a random signal with a mean of zero, if the characteristics of the electromyographic signal are extracted by directly calculating the mean of the signal, the mean is approximately zero, which is obviously not enough to characterize the difference between the signals. If the absolute value of the electromyographic signal is taken, the average value of the electromyographic signal is always greater than zero, so the integral electromyography (IEMG) can be used to extract the characteristics of the electromyographic signal. The calculation formula is:

[0072]

[0073] Among them, N 1 is the starting point of integration, N2 is the end point of integration, χ(t) is the electromyographic curve, and dt is the sampling time interval. IEMG is the integral of the area under the electromyographic change curve on the time axis, which can reflect the strength of electromyography within a certain period of time. Therefore, the embodiment of the present invention also calculates the average IEMG of each 100ms time period after different reactions. In addition, since the surface electromyographic signal is considered to be a chaotic signal, the collected frown electromyography can be analyzed using nonlinear characteristics, and the Lempel-Ziv complexity index is used to compare the differences in each 100ms time period after different reactions.

[0074] Two-way repeated measures ANOVA was used to analyze the frown EMG amplitude, IEMG, and Lempel-Ziv complexity with response type (realized error, uncertain error, unaware error, and correct) and time period (0-100ms, 100-200ms, 200-300ms, 300-400ms, 400-500ms, 500-600ms, 600-700ms, 700-800ms, 800-900ms, 900-1000ms) as within-subject factors. For all statistical calculations, partial Eta square (η p 2 ) represents the effect size. To reduce the incidence of type I errors in multiple comparisons, the Greenhouse-Geisser method was used to correct for the spherical shape assumption.

[0075] 2.3 Eye Movement Data Processing and Analysis

[0076] The analysis of eye movement data first checks whether there is data loss mainly caused by blinking, because the eye tracker temporarily loses visual contact with the pupil when the eyelids close. The data is completed by linear interpolation. Subsequently, the continuous eye movement data is divided into 1000ms long segments, ranging from the beginning of the reaction to 1000ms after the reaction. Next, the eye movement data of both eyes are averaged, and then each segment is divided into 10 100ms time periods, and the eye movement data of different reaction types are averaged in each time period.

[0077] The eye movement data were analyzed using a two-way repeated measures ANOVA with response type (realized error, uncertain error, unrealized error, and correct) and time period (0-100ms, 100-200ms, 200-300ms, 300-400ms, 400-500ms, 500-600ms, 600-700ms, 700-800ms, 800-900ms, 900-1000ms) as within-subject factors. For all statistical calculations, the partial Eta square (η p 2 ) represents the effect size. To reduce the incidence of type I errors in multiple comparisons, the Greenhouse-Geisser method was used to correct for the spherical shape assumption.

[0078] 2.4 Expression features include frowning, twitching of the corners of the mouth, tense facial muscles, anger, nervousness and other expressions.

[0079] 2.5 MLR-based single-trial EEG detection

[0080] The cross-trial variability is objective and contains information about fluctuations. The latency and amplitude of the CRN and EPP components evoked before errors showed particularly high cross-trial variability, which is likely due to fluctuations in cognitive factors (e.g., alertness and attention). This study used a method that has been proven to be reliable for obtaining single-trial parameters, multiple linear regression (MLR).

[0081] The MLR-based single-shot detection method was first proposed by Mayhew et al. to automatically estimate the latency and amplitude of single-trial event-related potentials. This MLR method is often used to analyze functional magnetic resonance imaging data. In addition to the standard hemodynamic response function (similar to the average ERP), its time derivative (used to explain the variability of hemodynamic response time) is also used to fit the brain response of the single-trial ERP. Therefore, including the time derivative in the regression variable can not only obtain the amplitude of the single-trial ERP response, but also the latency of the single-trial ERP response.

[0082] Taking the CRN wave and EPP wave of the trial before the error / correct response as an example, the difference in latency and amplitude of these two components can be described as follows:

[0083] f(t)=k C y C (t+a C )+k E y E (t+a E ),

[0084] where f(t) represents the ERP waveform of a single trial that changes over time. f(t) can be expressed as the CRN wave k of different trials with different C y C (t+a C ) and EPP wave k E y E (t+a E ) to simulate. k C and k E is the weighting constant of the CRN wave and the EPP wave, a C and a E Represents the latency difference of CRN wave and EPP wave respectively. It is worth noting that CRN and EPP waves are modeled separately because the CRN and EPP components of the previous trial response before the error / correct response reflect the activities of different neural generators. Using Taylor expansion, the MLR model can be described in detail as:

[0085] f(t)≈k C y C (t)+a Ck C y' C (t)+k E y E (t+a E )+a E k E y' E (t),

[0086] Among them, y C (t) and y E (t) represents the average value of CRN and EPP waves. C '(t) and y E '(t) represent the time derivatives of CRN and EPP waves, respectively. In other words, the single-trial ERP waveform is approximated as the sum of the weighted averages of CRN and EPP waves and their respective time derivatives.

[0087] Based on the fitted waveforms, the latency and amplitude of the ERP response in each single trial were estimated in the average ERP waveform of each subject by calculating the following within a predetermined time window centered on the CRN and EPP latencies: For the CRN wave, if k C >0 (positive fit), it is the largest negative peak; if k C <0 (negative fit), it is the maximum positive peak. For EPP wave, if k E >0 (positive fit), it is the largest positive peak; if k E <0 (negative fitting), it is the maximum negative peak. Estimate the latency of a single trial based on the latency of the corresponding amplitude.

[0088] 2.6 Feature Set Construction

[0089] CRN and EPP differed in the previous trial of errors and correctness in different consciousness states, so the amplitude and latency of CRN and EPP on 61 electrodes in the whole brain were extracted as EEG features (61×2×2=244) using MatlabR2013b software and STEP1 toolbox. The features of corrugator supercilii were amplitude (average every 100ms, 10), IEMG (average every 100ms, 10), LZC (average every 100ms, 10), and the eye movement data features were the eye movement data in the previous section (average every 100ms, 10), with a total of 284 features used as classification features. The number of single trials included in the analysis was 22,678.

[0090] In step S103, the error stage and error awareness type of the operator are obtained based on multi-stage and multi-type predictions of the physiological behavior feature set, and the error alarm level and alarm action corresponding to the current error and error awareness are matched based on the error stage and error type to perform error warning.

[0091] It is understandable that the embodiment of the present invention divides the operation results of people into four types of operation: conscious errors, uncertain errors, unaware errors, and correct operations through behavioral confirmation (there are three levels of error awareness, or three types of errors). By analyzing and studying the EEG, corrugator supercilii, and eye movement data before the errors corresponding to the four types of operations, specific indicators are extracted to distinguish the four types of operations. Statistical analysis is applied to effectively identify the differences in the above characteristics of conscious errors, unaware errors, uncertain errors, and correct operations, and physiological behavioral characteristics are used to effectively distinguish the four operation results, and multi-stage multi-physiological behavioral characteristics are used for distinction.

[0092] For example, the embodiment of the present invention can design an experimental task, and determine four error types by having the tester confirm the operation result by pressing the key. Specifically: a modified version of the error awareness task (EAT) task is used for testing. The task is implemented using software programming. Each task stimulus is presented in a specific font on a black background in a certain viewing angle, brightness, and contrast. The stimulus is 6 colored Chinese characters or different types of pictures. The subject is required to respond to each stimulus by pressing a key on the keyboard as quickly and correctly as possible with a finger when the font color of the word and its meaning, or the picture content is inconsistent with the meaning of the Chinese character (i.e., standard stimulus), and to respond by pressing another key on the keyboard with the left index finger when the following two situations (i.e., deviation stimulus) occur. After each stimulus is triggered and a key is pressed, three expressions (smiling face, calm face, crying face) will appear on the screen. At this time, the subject is required to make a subjective evaluation of the accuracy of the response just made (i.e., error awareness evaluation). If the response is considered correct, the ring finger of the right hand presses a key (corresponding to the smiling face); if the response is considered wrong, the index finger of the right hand presses another key (corresponding to the crying face); if it is not sure whether the response is correct or not, the middle finger of the right hand presses another key (corresponding to the calm face). Four operation results are determined in this way. This is the first time that three error awareness levels and four operation results have been classified through experimental design. The embodiment of the present invention can construct a specific indicator system by analyzing the characteristics of the forehead EEG, corrugator supercilii, and eye movement information of the four error types.

[0093] In an embodiment of the present invention, before obtaining the error stage and error awareness type of the operator based on the multi-stage and multi-type prediction of the physiological and behavioral feature set, it also includes: predicting the next trial error and error state based on a machine learning method based on a single trial analysis; extracting the previous trial physiological and behavioral features corresponding to different error intention states in a single trial, and using multiple prediction models to train the previous trial physiological and behavioral features one by one to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type prediction.

[0094] It is understandable that the embodiments of the present invention can extract the EEG features, corrugator muscle EMG features and eye movement data before the reaction through single trial analysis, and combine the XGBoost classifier algorithm to construct the prediction of error consciousness based on EEG, corrugator muscle and eye movement data. The steps of this part include: EEG single trial detection based on MLR (multiple linear regression), feature set construction, and error classification based on machine learning. The feature of this part is that it constructs multiple physiological feature indicators covering EEG, corrugator muscle, eye movement information, etc., and after analyzing the effects of multiple classifier algorithms, the most effective classifier algorithm is determined to ensure the accuracy and timeliness of the prediction.

[0095] Specifically, XGBoost is a type of ensemble learning algorithm. Ensemble learning is a supervised learning algorithm, and as the name suggests, ensemble learning integrates many different algorithms to give the model better predictive performance. The general idea is to improve the overall performance by combining decisions received from different multiple models. It is based on the concept of diversity, where more different models are considered to obtain results for the same problem than a single model. This gives a set of hypotheses that can be combined to get better performance. All the single models are called base learners, and when the combination is called an ensemble, the ensemble is mostly better than the base learners that make up the ensemble.

[0096] XGBoost implements a general Tree Boosting algorithm, one of the representatives of this algorithm is the Gradient Boosting Decision Tree (GBDT). The principle of GBDT is to first train a tree using the training set and the true value of the sample (i.e. the standard answer), and then use the tree to predict the training set to obtain the predicted value of each sample. Since there is a deviation between the predicted value and the true value, the "residual" can be obtained by subtracting the two. Next, train the second tree. Instead of using the true value, use the residual as the standard answer. After training the two trees, we can get the residual for each sample again, and then further train the third tree, and so on. The total number of trees can be manually specified, or certain indicators (such as errors on the validation set) can be monitored to stop training. When predicting a new sample, each tree has an output value, and the final predicted value of the sample is obtained by adding these output values.

[0097] XGBoost combines many previous works on gradient boosting algorithms and makes a lot of optimizations in engineering implementation. It is one of the best machine learning algorithms at present. The reason for its excellent learning effect is that the essence of machine learning is the fitting of models to data. For a set of data, using an overly complex model to fit often leads to overfitting. At this time, it is necessary to introduce regularization terms to limit the complexity of the model. However, the selection of regularization terms and the setting of regularization coefficients are relatively arbitrary and difficult to achieve the best. If a too simple model is used, it is difficult to grasp the rules contained in the data due to the limited model capabilities, resulting in poor results. The boosting algorithm (XGBoost is one of the boosting algorithms) is more clever. First, a simple model is used to fit the data to obtain a relatively general result, and then simple models (in most cases, decision trees with shallow layers) are continuously added to the model. As the number of trees increases, the complexity of the entire boosting model gradually increases until it approaches the complexity of the data itself. At this time, the training reaches the best level.

[0098] The embodiment of the present invention selects SVM, K nearest neighbor, decision tree and XGBoost four classification methods to compare the optimal classification effects. The comparison results are as follows: Figure 3 ,As shown in Table 1, XGBoost has the highest classification evaluation index.

[0099] Table 1

[0100]

[0101] In an embodiment of the present invention, the error stage and error consciousness type of the operator are obtained based on multi-stage and multi-type predictions of the physiological behavior feature set, and also include: when the time corresponding to the physiological behavior feature set is in a first interval, the error stage is determined to be an error trial stage; when the time corresponding to the physiological behavior feature set is in a second interval, the error stage is determined to be a pre-error trial stage; when the time corresponding to the physiological behavior feature set is in a third interval, the error stage is determined to be a pre-error trial stage, wherein the minimum value of the third interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the first interval.

[0102] Among them, the first to third intervals can be set according to actual conditions without specific limitations. For example, the first interval can be (-500—-1000ms), the second interval can be (-1000—-1800ms), and the third interval can be (-3—-4s). (-500—-1000ms), (-1000—-1800ms) and (-3—-4s) are respectively the first to third intervals. In order to, the stages of the embodiment of the present invention are divided from the perspective of time, including error trials (-500—-1000ms), before error trials (-1000—-1800ms), and before error trials (-3—-4s).

[0103] In an embodiment of the present invention, it also includes: obtaining the physiological behavioral characteristics of the trials before the errors of different work operation results in the test data as the basic threshold; obtaining the physiological behavioral characteristics of the operator after the work operation and the physiological behavioral characteristics of the previous trial, and performing AI training based on the physiological behavioral characteristics after the work operation, the physiological behavioral characteristics of the previous trial and the basic threshold to adjust the physiological behavioral characteristic threshold.

[0104] It is understandable that the embodiments of the present invention can set a time window for the continuous characteristics of the continuous task, based on the physiological and behavioral characteristics before the error, combined with the behavioral response after the operation, apply AI learning training, adaptively adjust the physiological and behavioral characteristics threshold and prediction strategy, and construct a continuous monitoring task error and error awareness warning. The main steps are as follows. (1) Determine the physiological and behavioral characteristics of the trials before the error of different operation results through basic research as the threshold basis; (2) The operator collects information such as EEG, corrugator muscle, eye movement information, facial micro-expression, etc. in real time during the execution of the task, and records the behavioral result data after the operation. The post-operation behavior and its corresponding physiological and behavioral characteristics of the previous trial are added to the training as AI training data; (3) Based on the training results, the physiological and behavioral characteristics threshold, prediction time and prediction strategy are adaptively adjusted; (4) The physiological and behavioral data are analyzed and distinguished in real time based on the threshold of accurate classification, and a certain threshold is met, which is a certain prediction interval.

[0105] Specifically, the embodiment of the present invention can use the threshold method to perform real-time prediction, continuously monitor the errors and error awareness of the task, wherein the adaptive adjustment of the physiological behavior feature threshold and the prediction strategy is realized in combination with AI; the method of real-time error prediction includes:

[0106] (1) Initial threshold: Based on basic research, the physiological and behavioral characteristics of the trials before the error of different operation results are determined as the basis of the threshold.

[0107] (2) Threshold adjustment: The post-operation behavior and its corresponding physiological and behavioral characteristics of the previous trial are added to the training as AI training data.

[0108] (3) AI training method: convolutional deep learning algorithm is used for training.

[0109] (4) Prediction strategy: multi-stage and multi-type prediction. The first level is error tendency prediction, which is the prediction of error tendency, that is, the probability of error is greater than 50% through the analysis of physiological and behavioral characteristic thresholds; the second level is error prediction, that is, the probability of error is greater than 60% through the analysis of physiological and behavioral characteristic thresholds; the third level is error prediction, that is, the probability of error is greater than 80% through the analysis of physiological and behavioral characteristic thresholds; the fourth level is error type prediction, that is, the level of consciousness is predicted through the analysis of physiological and behavioral characteristic thresholds: aware, unaware, uncertain.

[0110] (5) Adaptively adjust the physiological behavior feature threshold, prediction time, and prediction strategy based on the training results.

[0111] (6) Implementation scenario: During the experiment, the operator's EEG, frown EMG, eye movement, facial expression and other data are collected, and the EEG power spectrum, frown EMG amplitude, IEMG, pupil diameter, blinking frequency, facial micro-expressions and other features are calculated in real time. According to the threshold conditions and types, the operation results are predicted accordingly.

[0112] In an embodiment of the present invention, the type of error consciousness of the operator is obtained based on multi-stage and multi-type predictions of physiological and behavioral feature sets, including: when the characteristic value of the behavioral feature set is greater than a first threshold, the error consciousness type is determined to be an conscious error type; when the characteristic value of the behavioral feature set is less than a second threshold, the error consciousness type is determined to be an unaware error type; when the characteristic value of the behavioral feature set is less than or equal to the first threshold and greater than or equal to the second threshold, the error consciousness type is determined to be an uncertain type.

[0113] Among them, the first threshold is greater than the second threshold, which can be specifically calibrated without specific limitation.

[0114] Specifically, 1. Error trials

[0115] Pe amplitude: realized error > uncertain error > unrealized error;

[0116] Frown electromyography: realized error > uncertain error > unaware error;

[0117] Pupil diameter: realized error > uncertain error > unaware error;

[0118] ①N1 amplitude: realized errors > uncertain errors > unaware errors;

[0119] ②P2 amplitude: realized error < uncertain error < unaware error;

[0120] ③P3 amplitude: realized errors > uncertain errors > unaware errors;

[0121] Pupil diameter: Realized error ≈ uncertain error ≈ unaware error.

[0122] 2. Before the error trial

[0123] α-ERD: Realized errors ≈ uncertain errors > unaware errors;

[0124] Frown electromyography: conscious error ≈ uncertain error ≈ unaware error, the difference is not significant;

[0125] Pupil diameter: realized errors > uncertain errors > unaware errors, the difference was not significant.

[0126] 3. Response to the previous trial

[0127] EPP magnitude: realized errors > uncertain errors > unrealized errors.

[0128] 4. Time-frequency analysis:

[0129] ①θ / α-ERD: Realized error ≈ uncertain error ≈ unaware error;

[0130] ②α-ERS: Realized error ≈ uncertain error ≈ unaware error;

[0131] Frown electromyography: conscious error ≈ uncertain error ≈ unaware error, the difference is not significant;

[0132] Pupil diameter minimum uncertain error > unaware error ≈ aware.

[0133] Furthermore, the warning part of the embodiment of the present invention includes: (1) setting the warning level and method; (2) determining the warning information distribution object and strategy; wherein,

[0134] The warning levels are divided into: general, medium, serious, urgent, and very urgent; the warning methods are: sound: audio of different frequencies; vision: flashing lights, etc.; touch: vibration.

[0135] Information distribution targets may include: Task supervision platform and personnel: All warning results are fed back to the task supervision platform for record, and supervisors are reminded when an error warning occurs; Task assistants: All warning results are fed back to the task supervision platform for record, and assistants are reminded when an error warning occurs; Task executors: When an uncertain or unaware error warning occurs, the executors are reminded, and no warning reminders are given in other situations

[0136] In summary, the embodiments of the present invention can collect the operator's EEG, corrugator muscle, eye movement information, facial micro-expressions and other physiological behavior data while the operator is performing a continuous monitoring task, and record the operation results. The online prediction system can predict the type of error and error consciousness based on the threshold conditions through real-time analysis of the physiological behavior data. Combined with the warning classification and distribution objects and strategies, different warning methods are used to distribute the relevant warning information to the relevant objects using corresponding strategies. The method of the embodiment of the present invention can be used for the monitoring and prediction, warning and prevention of the operating behavior, operating status and human errors of personnel in continuous monitoring tasks in special industries such as aviation, aerospace, and nuclear power. And it has the following beneficial effects:

[0137] (1) Realize online prediction and early warning of error types and error awareness.

[0138] (2) The operation results are divided into three categories: errors and correctness. The classification of error awareness is divided from the traditional two categories to three categories, making the classification results more accurate. Separating "uncertain errors" can, on the one hand, prevent low-confidence errors from contaminating unaware errors, and on the other hand, obtain an error awareness state between conscious errors and unaware errors, which is more conducive to studying the neural mechanism of error awareness. At the same time, in order to circumvent the shortcoming of the active error reporting method used in previous related studies, which makes participants more inclined not to report errors, this study modified the error reporting stage to conduct a subjective evaluation of error awareness after each trial response, and then accurately obtain the error awareness state of each response through the consistency between the objective response accuracy and the subjective evaluation.

[0139] (3) The neurophysiological characteristics of error awareness before an error are studied, and it is found that the brain activity state before the reaction can characterize the subsequent error awareness reaction state, providing a method and indicator basis for subsequent error prediction / warning.

[0140] (4) Using multiple physiological indicators to study the physiological characteristics of error consciousness with a wider coverage. Errors and error consciousness in a task may be represented by different neural correlates. Unlike previous related studies that mostly use a single method or indicator, this study simultaneously collects different physiological signals such as EEG, frown electromyography and eye movement data to analyze the neural correlates of error consciousness. The data complement and confirm each other, indicating that brain activities related to different error consciousness reactions can be detected from multiple aspects. At the same time, these measurement indicators at least partially reflect different underlying mechanisms and therefore cannot replace each other.

[0141] (5) The prediction of errors and error consciousness states based on machine learning methods of single trial analysis of multiple physiological features was realized. This paper integrates multiple physiological indicators such as EEG, frown electromyography and eye movement data, and predicts the error and error consciousness state of the next trial based on machine learning methods of single trial analysis. The signals of the previous trial with different error consciousness states are extracted in a single trial, and the error and error consciousness classification model is established using the XGboost method in machine learning. It can predict error reactions and error consciousness states about 4 seconds in advance. The results show that it has high accuracy and certain application value.

[0142] (6) Multi-level prediction: the first level is error tendency prediction (there is an error tendency prediction, that is, the probability of error is greater than 50% through the analysis of the physiological behavior characteristic threshold), the second level is error prediction (that is, the probability of error is greater than 60% through the analysis of the physiological behavior characteristic threshold), the third level is error prediction (that is, the probability of error is greater than 80% through the analysis of the physiological behavior characteristic threshold), and the fourth level is error type prediction (that is, the level of consciousness is predicted through the analysis of the physiological behavior characteristic threshold: aware, unaware, uncertain)

[0143] (7) The adaptive adjustment of the error prediction warning threshold is realized, the threshold method is used for warning, and AI training is applied to adaptively adjust the threshold.

[0144] (8) Accurate synchronization of multiple physiological data is achieved.

[0145] According to the error and error awareness warning method based on multiple physiological characteristics proposed in an embodiment of the present invention, errors and error awareness of the behavior of operators can be predicted based on the processing and analysis results of synchronously collected EEG data, frown electromyography data, eye movement data and facial micro-expression data, and error warning can be achieved. Therefore, the accuracy of error and error awareness warning can be effectively improved through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

[0146] Next, the error and error awareness warning device based on multiple physiological characteristics proposed in an embodiment of the present invention will be described with reference to the accompanying drawings.

[0147] Figure 4 It is a block diagram of an error and error awareness warning device based on multiple physiological characteristics according to an embodiment of the present invention.

[0148] like Figure 4 As shown, the error and error awareness warning device 10 based on multiple physiological characteristics includes: a collection module 100, a processing module 200, a prediction module 300 and an alarm module 400.

[0149] Among them, the acquisition module 100 is used to synchronously collect the EEG data, frowning EMG data, eye movement data and facial micro-expression data of the operator; the processing module 200 is used to process the synchronously collected EEG data, frowning EMG data, eye movement data and facial micro-expression data, extract EEG features, frowning EMG features, eye movement features and expression features, and fuse the extracted EEG features, frowning EMG features, eye movement features and expression features to generate a physiological behavior feature set; the prediction module 300 is used to obtain the error stage and error consciousness type of the operator based on multi-stage and multi-type predictions of the physiological behavior feature set; the alarm module 400 is used to match the error alarm level and alarm action corresponding to the current error and error consciousness based on the error stage and error type, so as to provide error warning.

[0150] Furthermore, the prediction module 300 also includes: a first-stage prediction unit, a second-stage prediction unit, and a third-stage prediction unit. The first-stage prediction unit is used to determine that the error stage is the error trial stage when the time corresponding to the physiological behavior feature set is in the first interval; the second-stage prediction unit is used to determine that the error stage is the pre-error trial stage when the time corresponding to the physiological behavior feature set is in the second interval; the third-stage prediction unit is used to determine that the error stage is the pre-error trial stage when the time corresponding to the physiological behavior feature set is in the third interval, wherein the minimum value of the first interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the third interval.

[0151] Furthermore, the prediction module also includes: a first type prediction unit, a second type prediction unit and a third type prediction unit. The first type prediction unit is used to determine the error consciousness type as the aware error type when the characteristic value of the behavior feature set is greater than the first threshold; the second type prediction unit is used to determine the error consciousness type as the unaware error type when the characteristic value of the behavior feature set is less than the second threshold, wherein the first threshold is greater than the second threshold; the third type prediction unit is used to determine the error consciousness type as the uncertain type when the characteristic value of the behavior feature set is less than or equal to the first threshold and greater than or equal to the second threshold.

[0152] Furthermore, the device 10 of the embodiment of the present invention further includes: a threshold adjustment module. The threshold adjustment module is used to obtain the physiological behavior characteristics of the trials before the errors of different operation results in the test data as the basic threshold; obtain the physiological behavior characteristics of the operator after the operation and the physiological behavior characteristics of the previous trial, and perform AI training based on the physiological behavior characteristics after the operation, the physiological behavior characteristics of the previous trial and the basic threshold to adjust the physiological behavior characteristic threshold.

[0153] Furthermore, the device 10 of the embodiment of the present invention further includes: a modeling module. The modeling module is used to predict the next trial error and error state based on a machine learning method of single trial analysis, extract the physiological and behavioral characteristics of the previous trial corresponding to different error states in a single trial, and use multiple prediction models to train the physiological and behavioral characteristics of the previous trial one by one to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type prediction.

[0154] It should be noted that the aforementioned explanation of the embodiment of the error and error awareness warning method based on multiple physiological characteristics is also applicable to the error and error awareness warning device based on multiple physiological characteristics of this embodiment, and will not be repeated here.

[0155] The error and error awareness warning device based on multiple physiological characteristics proposed in the embodiment of the present invention can predict errors and error awareness of the behavior of operators based on the processing and analysis results of synchronously collected EEG data, frown electromyography data, eye movement data and facial micro-expression data, and can realize error warning, so that the accuracy of error and error awareness warning can be effectively improved through multiple physiological characteristics, so as to effectively monitor the behavior of operators and greatly reduce the possibility of repeated errors.

[0156] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0158] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for early warning of errors and error awareness based on multiple physiological characteristics, It is characterized in that The following steps are involved: Synchronously collect the operator's EEG data, frown EMG data, eye movement data and facial micro-expression data; Process the synchronously collected EEG data, frown EMG data, eye movement data and facial micro-expression data to extract EEG features, frown EMG features, eye movement features and facial expression features, and fuse the extracted EEG features, frown EMG features, eye movement features and facial expression features to generate a physiological behavior feature set; as well as Based on the multi-stage and multi-type prediction of the physiological behavior feature set, the error stage and error consciousness type of the operator are obtained, and based on the error stage and the error consciousness type, the error alarm level and alarm action corresponding to the current error and error consciousness are matched to perform error and error consciousness warning; The error stage of the operator is obtained based on the multi-stage and multi-type prediction of the physiological behavior feature set, and further includes: when the time corresponding to the physiological behavior feature set is in the first interval, the error stage is determined to be the error trial stage; when the time corresponding to the physiological behavior feature set is in the second interval, the error stage is determined to be the pre-error trial stage; when the time corresponding to the physiological behavior feature set is in the third interval, the error stage is determined to be the pre-error trial stage, wherein the minimum value of the first interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the third interval; the error consciousness type of the operator is obtained based on the multi-stage and multi-type prediction of the physiological behavior feature set, and further includes: when the feature value of the behavior feature set is greater than a first threshold value, the error consciousness type is determined to be an aware of error type; when the feature value of the behavior feature set is less than a second threshold value, the error consciousness type is determined to be an unaware of error type, wherein the first threshold value is greater than the second threshold value; when the feature value of the behavior feature set is less than or equal to the first threshold value, and greater than or equal to the second threshold value, the error consciousness type is determined to be an uncertain type.

2. The method according to claim 1, It is characterized in that Also includes: Obtain the physiological and behavioral characteristics of the trials before the errors of different operation results in the test data as the basic threshold; The physiological behavior characteristics of the operator after the operation and the physiological behavior characteristics of the previous trial are obtained, and AI training is performed based on the physiological behavior characteristics after the operation, the physiological behavior characteristics of the previous trial and the basic threshold to adjust the physiological behavior characteristic threshold.

3. The method according to claim 1, It is characterized in that Before obtaining the error stage and error consciousness type of the operator based on the multi-stage and multi-type prediction of the physiological behavior feature set, the method further includes: Machine learning methods based on single trial analysis predict the next trial error and error status; The physiological and behavioral characteristics of the previous trial corresponding to different error intention states are extracted in a single trial, and the physiological and behavioral characteristics of the previous trial are trained one by one using multiple prediction models to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type prediction.

4. An error and error awareness warning device based on multiple physiological characteristics, It is characterized in that include: The acquisition module is used to synchronously collect the operator's EEG data, frown EMG data, eye movement data and facial micro-expression data; A processing module is used to process the synchronously collected EEG data, frown EMG data, eye movement data and facial micro-expression data, extract EEG features, frown EMG features, eye movement features and facial expression features, and fuse the extracted EEG features, frown EMG features, eye movement features and facial expression features to generate a physiological behavior feature set; A prediction module, used for predicting the error stage and error consciousness type of the operator based on the multi-stage and multi-type prediction of the physiological behavior feature set; The prediction module further includes: a first stage prediction unit, used to determine that the error stage is an error trial stage when the time corresponding to the physiological behavior feature set is in a first interval; a second stage prediction unit, used to determine that the error stage is a pre-error trial stage when the time corresponding to the physiological behavior feature set is in a second interval; a third stage prediction unit, used to determine that the error stage is a pre-error trial stage when the time corresponding to the physiological behavior feature set is in a third interval, wherein the minimum value of the first interval is greater than the maximum value of the second interval, and the minimum value of the second interval is greater than the maximum value of the third interval; the prediction module further includes: a first type prediction unit, used to determine that the error consciousness type is an aware error type when the feature value of the behavior feature set is greater than a first threshold; a second type prediction unit, used to determine that the error consciousness type is an unaware correct type when the feature value of the behavior feature set is less than a second threshold, wherein the first threshold is greater than the second threshold; a third type prediction unit, used to determine that the error consciousness type is an uncertain type when the feature value of the behavior feature set is less than or equal to the first threshold and greater than or equal to the second threshold; An alarm module is used to match the error alarm level and alarm action corresponding to the current error and error awareness based on the error stage and the error awareness type, so as to provide error and error awareness warning.

5. The device according to claim 4, It is characterized in that Also includes: A threshold adjustment module is used to obtain physiological behavior characteristics of different operation results before the error in the test data as a basic threshold; The physiological behavior characteristics of the operator after the operation and the physiological behavior characteristics of the previous trial are obtained, and AI training is performed based on the physiological behavior characteristics after the operation, the physiological behavior characteristics of the previous trial and the basic threshold to adjust the physiological behavior characteristic threshold.

6. The device according to claim 4, It is characterized in that Also includes: A modeling module is used to predict the error and error state of the next trial based on a machine learning method for single trial analysis, extract the physiological and behavioral characteristics of the previous trial corresponding to different error intention states in a single trial, and use multiple prediction models to train the physiological and behavioral characteristics of the previous trial one by one to establish an error and error awareness classification model, wherein the classification model is used for multi-stage and multi-type predictions.

Images