Multi-factor fusion visual fatigue evaluation system and method for SSVEP in BCI application

Abstract

The invention relates to a multi-factor fusion visual fatigue evaluation system and method in application of SSVEP (Steady-State Visual Evoked Potential) in BCI (Brain Center Interface), which establishes an experimental environment in which subjective and objective fusion factors of visual fatigue evaluation are normal form stimulation frequency, display resolution and refresh rate, and introduces a Like scale for collecting subjective fatigue and preference scores. The electroencephalogram acquisition module and the eye tracker are used for collecting objective data, an information entropy-CRITIC weight method is combined with subjective and objective indexes, a fatigue assessment system (display screen fitness DSF) is provided, different experimental environments are scored, the higher the DSF score is, the better the visual experience is, and when an SSVEP-based BCI application program is used, the visual experience is better. The DSF can provide powerful help for hardware and software selection guidance and visual protection; according to the method, more indexes are included from the objective angle, visual fatigue quantitative evaluation is explored, a subjective and objective index fusion method without prior experience is provided, and the method is beneficial to establishment of a gold standard for visual fatigue quantitative evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of biomedical engineering and brain-computer interface, and in particular relates to a system and method for evaluating visual fatigue using SSVEP with multiple factors fusion in BCI applications. Background Art

[0002] With the development of the brain-computer interface industry, a large number of related applications have entered people's field of vision. Brain-computer interfaces (BCIs) based on steady-state visual evoked potentials (SSVEPs) are widely used because they do not require pre-training and have a high information transmission rate (ITR); however, when the SSVEP stimulation paradigm is actually used, the subjects will experience visual fatigue as they use it, and fatigue will affect the transmission efficiency. SSVEP relies on the visual stimulation paradigm to make the subjects produce corresponding electroencephalogram (EEG) responses, and then conducts specific analysis. With the large-scale application of the SSVEP paradigm, the visual fatigue it brings to the subjects has received more and more attention. The fatigue caused by long-term use of SSVEP will seriously reduce the ITR and affect the effect of use.

[0003] Studies have shown that compared with the low-frequency (5Hz) stimulation paradigm, the high-frequency (12Hz) stimulation paradigm has a higher signal-to-noise ratio (SNR), and the speller using high-frequency (30Hz) can be used for a longer time (about 12 minutes), although its performance is still slightly reduced; at the same time, the ITR of the brain-computer interface is closely related to the parameters of the electronic screen. Different screen parameters, such as resolution, refresh rate, brightness, etc., also bring uncertainty to the application effect of the SSVEP-based BCI, among which the uncertainty is mainly reflected in the comfort of the eyes. If the screen refresh rate is too low, the stickiness and blockage of the computer will reduce the user experience. If the resolution is too low, the blurring of the image quality is more likely to cause visual fatigue. Therefore, understanding the subjective and objective experience of users under different paradigm stimulation frequencies, screen resolutions, and screen refresh rate combinations has a positive effect on reducing visual fatigue, improving the application experience of BCI, and designing an SSVEP paradigm that is more suitable for human physiological habits.

[0004] At present, the quantitative evaluation of visual fatigue in academia is still in the exploratory stage. Some people believe that subjective scoring is the gold standard, while others believe that there is a lack of a gold standard because there are large individual differences in the subjective scoring of subjects. Scholars use certain subjective or objective methods to conduct qualitative and quantitative evaluations of visual fatigue, and then analyze each indicator separately. Most quantitative studies on visual fatigue ([1]Makri D, Farmaki C, Sakkalis V.Visual fatigue effects onsteady state visual evoked potential-based brain computer interfaces.In20157th international IEEE / EMBS conference on neural engineering(NER)2015Apr 22(pp.70-73).IEEE;[2]Seo SP,Lee MH,Williamson J,Lee SW.Changes in fatigue and EEG amplitude during a chronic use of brain-computerinterface.In20197th International Winter Conference on Brain-ComputerInterface(BCI)2019Feb 18(pp.1-3).IEEE;[3]PengY,Wong CM,Wang Z,Wan F,Vai MI,Mak PU,Hu Y,Rosa AC.Fatigue evaluation using multi-scale entropy ofEEG inSSVEP-based BCI.IEEE Access.2019Aug 1; 7:108200-10; [4] Chai X, Zhang Z, Guan K, Zhang T, Xu J, Niu H. Effects of fatigue on steady state motion visual evoked potentials: optimized stimulus parameters for a zoom motion-based brain-computer interface. Computer Methods and Programs in Biomedicine.2020Nov 1; 196: 105650; [5] Peng Y, Wong CM, Wang Z, Rosa AC, Wang HT, Wan F. Fatigue detection in SSVEP-BCIs based on wavelet entropy of EEG. Ieee Access. 2021Jul 26; 9: 114905-13.) are all based on a single indicator. Since the academic community still lacks a gold standard for quantitatively evaluating visual fatigue, a fusion improvement scheme of the CRITIC weighting method is proposed. . Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a system and method for visual fatigue assessment by multi-factor fusion of SSVEP in BCI applications, establish an experimental environment in which the subjective and objective fusion factors of visual fatigue assessment are paradigm stimulation frequency, display resolution and refresh rate, introduce Likert scale to collect subjective fatigue and preference scores, objective EEG signal acquisition module and objective eye movement signal acquisition module to collect objective data, and use information entropy-CRITIC weighting method to combine subjective and objective indicators to propose a fatigue assessment system (display screen fitness DSF) to score different experimental environments. The higher the DSF score, the better the visual experience. When using BCI applications based on SSVEP, DSF can provide strong help for hardware and software selection guidance and visual protection. The present invention not only includes more indicators from an objective perspective, explores the quantitative assessment of visual fatigue, but also proposes a subjective and objective indicator fusion method without prior experience, which is helpful to establish a gold standard for quantitative evaluation of visual fatigue.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is:

[0007] A multi-factor fusion visual fatigue assessment system of SSVEP in BCI application, comprising: an objective EEG signal acquisition module, an objective eye movement signal acquisition module, a subjective Likert scale assessment module, and a screen parameter adjustment module;

[0008] Objective EEG signal acquisition module: simulates two paradigm stimulation frequencies of 7.5Hz and 15Hz;

[0009] Objective eye movement signal acquisition module: collects eye movement signals based on an eye tracker;

[0010] Subjective Likert scale evaluation module: subjective evaluation based on subjective fatigue questionnaire and subjective preference questionnaire;

[0011] Screen parameter adjustment module: a high refresh display screen whose screen parameters can be adjusted manually.

[0012] The described SSVEP multi-factor fusion visual fatigue assessment system in BCI applications is used in a closed dark room. A display with adjustable screen resolution and screen refresh rate is arranged in the dark room as a screen parameter adjustment module, and the screen brightness is calibrated using a HOPOO spectral brightness analyzer.

[0013] In the objective EEG signal acquisition module, electrodes are arranged in the occipital area, left and right temporal lobes and parietal lobes, and the acquisition electrodes are arranged in O1, O2, Oz, PO3, PO4, POz, T7, TP7, T8, TP8, Cz, FC1, FC2, F3, Fz and F4 areas in sequence. The ground electrode FPz is located on the forehead, and the reference electrode A1 is located on the left earlobe.

[0014] The objective EEG signal acquisition module sets 18 combination modes of three screen resolutions of 800×600, 1280×720, and 1920×1080 and three screen refresh rates of 120Hz, 240Hz, and 360Hz. The screen brightness contrast is adjusted to 287.6cd / m2, that is, the 50% mode. The combination of paradigm stimulation frequency, screen resolution, and screen refresh rate affects the visual perception of SSVEP-based BCI applications, that is, visual fatigue.

[0015] The evaluation method of the multi-factor fusion visual fatigue evaluation system using the above-mentioned SSVEP in BCI application includes the following steps:

[0016] Step 1: Guide the subject to sit at a designated position in a fully enclosed dark room, with the head fixed using a headrest;

[0017] Step 2: The subject wears the EEG cap and tests whether the EEG signal is recorded normally;

[0018] Step 3: Turn on the eye tracker and calibrate it;

[0019] Step 4: Adjust the corresponding screen parameters;

[0020] Step 5: Questionnaire to record subjective fatigue level before the experiment;

[0021] Step 6: Start the formal steady-state visual evoked potential stimulation experiment;

[0022] Step 7: After the experiment, record the subjective fatigue level and the preference for the evaluation system after the experiment;

[0023] Step 8: Organize all objective signal collection data, subjective fatigue inquiry data, and subjective preference inquiry data, set evaluation indicators, and conduct multi-factor fusion visual fatigue evaluation.

[0024] Before each round of evaluation experiments, the subjects' visual perception was first recorded using a Likert scale; secondly, the paradigm stimulation frequency, screen resolution, and screen refresh rate were adjusted; thirdly, in each group of experiments, the first three experiments were designed as white static "x" marks, and the subjects stared at the white static "x" marks on a dark background; fourthly, at the end of the experiment, the subjects were asked about their subjective feelings and preferences for the entire task using a Likert scale; visual fatigue accumulated over the course of the experiment, and the 4-23 formal experiments for each task were divided into four levels on average, with Level 1 for 4-8 experiments representing the lowest level of fatigue at the beginning, and Level 4 for 19-23 experiments representing the highest level of fatigue at the end.

[0025] The Likert scale is a set of subjective questions about the subject's fatigue level. The first six questions in the scale are composed of 1 to 10, with 1 indicating strongly disagree and 10 indicating strongly agree. By adding the scores of several questions, the subject's visual fatigue can be intuitively seen. The seventh question is composed of 1 to 5, with 1 indicating strongly dislike and 5 indicating very like. The first six questions in the scale are used to reflect visual fatigue, and the seventh question is used to reflect preference.

[0026] The EEG data were analyzed using the Canonical Correlation Analysis (CCA) algorithm. The CCA algorithm uses channel covariance information with a high signal-to-noise ratio to process the data. All data were standardized. The specific standardization procedure was as follows: in the 23 experiments, the first three stimulation experiments were not flashes, but a fixed white mark; the first three stimuli were used as reference values, and then the values ​​of the subsequent 20 real flashes were divided by the reference value.

[0027] The evaluation indicators in the eighth step include visual fatigue VFS, which is the sum of the subjects' Likert scale scores, and the subjective preference SP obtained directly from the Likert scale, which reflects the subjective quantification of visual fatigue; and include EEG data of θ+α band energy EB, EEG response amplitude EA, response signal-to-noise ratio ES, and pupil diameter PD data directly recorded by the eye tracker, which are used to reflect the objective quantification of participants' visual fatigue from the perspective of the optic nerve and cerebral cortex.

[0028] The eighth step of performing multi-factor fusion visual fatigue assessment includes the following steps:

[0029] In the first step, the indicators are divided into subjective quantitative VFS, SP and objective quantitative EB, EA, ES, and PD;

[0030] In the second step, for each indicator, one-way, two-way, and three-way repeated measures analysis of variance (ANOVA) with a significance level < 0.05 was used to conduct 1-level and 4-level analyses on the above 6 indicators of the 18 conditions;

[0031] In the third step, when there was a significant main effect, a post hoc Bonferroni adjustment was applied for multiple comparisons at p < 0.05;

[0032] The fourth step is to use the information entropy-CRITIC weight method to integrate subjective and objective indicators to obtain the health of the display screen; the data on the health of the display screen is binned, and the consistency test is used to verify the effect.

[0033] The information entropy-CRITIC weighting method is specifically:

[0034] 1) Divide the data into n groups by category, n=6, representing VFS, SP, EB, EA, ES, and PD. The subjective part is divided into before and after the experiment. The objective part is divided into the first 1 / 4 as the least fatigued state and the last 1 / 4 as the most fatigued state according to the number of experiments.

[0035] 2) The subjective original fatigue value is defined as the value after the experiment minus the value before the experiment. The objective original fatigue value is defined as the difference between the least fatigue state and the most fatigue state according to the specific data type.

[0036] 3) Standardize the objective data obtained, and the ratio of the objective original fatigue value to the objective initial calibration value is the standardized objective original fatigue value;

[0037] 4) Map the data to the interval [0, 1] for forward or reverse processing. The forward or reverse processing of the data is based on the characteristics of the data. The larger the value, the better the forward processing, and the smaller the value, the better the reverse processing;

[0038] 5) Use the CRITIC algorithm to calculate the weights corresponding to n groups of data: w1, w2, ..., w n ;

[0039] 6) Calculate the information entropy corresponding to n groups of data: I1, I2, ..., I n ;

[0040] 7) The corresponding information entropy is multiplied by the CRITIC weight respectively, and then the sum is distributed to n groups of data in proportion: Ie-w1, Ie-w2, ..., Ie-w n ;

[0041] 8) Multiply the obtained Ie-w weight by the corresponding n groups of data quantization values ​​and then add them together to obtain the final fatigue quantization score value, which is the health of the display screen.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Since the present invention adopts the information entropy-CRITIC weighting method to integrate subjective and objective indicators to evaluate different usage environments, it has the following two advantages: 1. Compared with the single indicator evaluation method currently adopted by most academic circles, the information considered by the integrated indicator is more comprehensive and more reliable; 2. The information entropy-CRITIC weighting method does not require the user's prior experience, which is more convenient for others to refer to and reproduce.

[0044] The evaluation method of the present invention is based on six evaluation indicators (VFS, SP, EB, EA, ES, PD) to explore the quantitative evaluation of visual fatigue under the combination of three different environmental factors (refresh rate, screen resolution and paradigm stimulation frequency), proposes a new indicator of display screen health, and quantitatively evaluates the impact of the combination of three different environmental parameters on visual fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the system of the present invention.

[0046] Figure 2 It is a schematic diagram of the experimental environment of the system of the present invention.

[0047] Figure 3 It is the layout diagram of the electrode positions of the system of the present invention.

[0048] Figure 4 is a flow chart of the method of the present invention.

[0049] Figure 5 It is a flow chart of the information entropy-CRITIC weighting method for quantitative evaluation of visual fatigue of the present invention.

[0050] Figure 6 It is the mean and scale of Likert scale (VFS) of 22 subjects under 18 combinations.

[0051] Figure 7 The Likert scale (SP) means and SD values ​​of 22 subjects in 18 conditions are shown.

[0052] Figure 8 The mean and scaled values ​​of the EEG theta+alpha band (EB) were compared for 22 subjects in 18 conditions.

[0053] Fig. 9 The mean and scaled EEG response amplitudes (EA) for 22 subjects in 18 conditions.

[0054] Fig.10 The mean and scaled EEG signal-to-noise ratio (ES) of 22 subjects under 18 conditions were compared.

[0055] Fig.11The pupil diameter (PD) is the mean and scale of 22 subjects in 18 conditions.

[0056] Fig.12 It is the consistency check evaluation tool of the present invention.

[0057] Fig.13 The present invention is based on the information entropy-CRITIC weighting method to evaluate the anti-fatigue score of displays at different resolutions and refresh rates at a paradigm stimulation frequency of 7.5 Hz.

[0058] Fig.14 This paper is based on the information entropy-CRITIC weighting method, and is used to evaluate the anti-fatigue performance of displays at different resolutions and refresh rates at a 15Hz paradigm stimulation frequency. DETAILED DESCRIPTION

[0059] The technical scheme of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0060] Reference Figure 1 , a multi-factor fusion visual fatigue assessment system of SSVEP in BCI application, including: objective EEG signal acquisition module, objective eye movement signal acquisition module, subjective Likert scale assessment module, screen parameter adjustment module;

[0061] Objective EEG signal acquisition module: simulate two paradigm stimulation frequencies of 7.5Hz and 15Hz; set 18 combination modes of three screen resolutions of 800×600, 1280×720, and 1920×1080 and three screen refresh rates of 120Hz, 240Hz, and 360Hz, and adjust the screen brightness contrast to 287.6cd / m2, i.e., 50% mode, to analyze the combination of paradigm stimulation frequency, screen resolution, and screen refresh rate that affects the visual perception of SSVEP-based BCI applications, i.e., visual fatigue;

[0062] Objective eye movement signal acquisition module: collects eye movement signals based on an eye tracker;

[0063] Subjective Likert scale evaluation module: subjective evaluation based on subjective fatigue questionnaire and subjective preference questionnaire;

[0064] Screen parameter adjustment module: a high refresh display screen whose screen parameters can be adjusted manually.

[0065] Reference Figure 2The described SSVEP multi-factor fusion visual fatigue assessment system in BCI applications is used in a closed dark room, which is built with opaque black cloth, and the control variables are not interfered by external light sources; a display with adjustable screen resolution and screen refresh rate is arranged in the dark room as a screen parameter adjustment module, and the screen brightness is calibrated using a HOPOO spectral brightness analyzer. The screen is 55 cm away from the subject, and the visual angle of the paradigm stimulus is 4° up and down, with a diameter of 148 pixels.

[0066] Reference Figure 3 In the objective EEG signal acquisition module, electrodes are arranged in the occipital area, left and right temporal lobes, and parietal lobes, and the acquisition electrodes are arranged in O1, O2, Oz, PO3, PO4, POz, T7, TP7, T8, TP8, Cz, FC1, FC2, F3, Fz, and F4 areas in turn, the ground electrode FPz is located on the forehead, and the reference electrode A1 is located on the left earlobe.

[0067] Reference Figure 4 The evaluation method of the multi-factor fusion visual fatigue evaluation system using the above-mentioned SSVEP in BCI application includes the following steps:

[0068] Step 1: Guide the subject to sit at a designated position in a fully enclosed dark room, with the head fixed using a headrest;

[0069] Step 2: The subject wears the EEG cap and tests whether the EEG signal is recorded normally;

[0070] Step 3: Turn on the eye tracker and calibrate it;

[0071] Step 4: Adjust the corresponding screen parameters;

[0072] Step 5: Questionnaire to record subjective fatigue level before the experiment;

[0073] Step 6: Start the formal steady-state visual evoked potential stimulation experiment;

[0074] Step 7: After the experiment, record the subjective fatigue level and the preference for the evaluation system after the experiment;

[0075] Step 8: Organize all objective signal collection data, subjective fatigue inquiry data, and subjective preference inquiry data, set evaluation indicators, and conduct multi-factor fusion visual fatigue evaluation.

[0076] Each subject was required to complete all 18 experimental tasks. Each set of experiments included 23 trials. The prompt time before each paradigm stimulus was 0.5 seconds, the formal stimulus time was 5 seconds, the rest time after stimulus was 0.5 seconds, and the complete test time was 6 seconds.

[0077] Before each round of evaluation experiments, the visual perception of the subjects was first recorded using the Likert scale; secondly, the paradigm stimulation frequency, screen resolution, and screen refresh rate were adjusted; thirdly, in each group of experiments, the first three experiments were designed as white static "x" marks, and the subjects stared at the white static "x" marks on a dark background. The data recorded in the first three experiments helped to normalize the subsequent data; fourthly, at the end of the experiment, the Likert scale was used to ask the subjects about their subjective feelings and preferences for the entire task; visual fatigue accumulated over the course of the experiment, and the 4-23 formal experiments of each task were divided into four levels on average, with level 1 in 4-8 experiments representing the lowest level of fatigue at the beginning, and level 4 in 19-23 experiments representing the highest level of fatigue at the end;

[0078] The canonical correlation analysis (CCA) algorithm was used to analyze the EEG data. After filtering and screening, the sum of the energy of the theta band (4-7 Hz) and the alpha band (8-13 Hz) was used as a band index, called EEG band energy (EB). At the same time, according to the stimulation frequency of the paradigm, the CCA response amplitude and signal-to-noise ratio at the stimulation frequency were used as quantitative indicators, called EEG amplitude (EA) and EEG signal-to-noise ratio (ES), respectively. The change in pupil diameter is used to reflect the degree of visual fatigue. Generally speaking, the average pupil diameter is 4 mm (generally 2-5 mm, up to 8 mm, and down to 1.5 mm), and the difference in pupil size between the two eyes is less than 0.25 mm. The pupil diameter (PD) index directly reflects the eye condition of the subject and can be used as a corresponding objective quantitative indicator. All data will be standardized for subsequent analysis. Specific standardization procedures: In the 23 trials, the first three are a fixed white mark, which is used as a reference value, and the values ​​of the subsequent 20 real flashes are divided by the reference value.

[0079] The mean and scale of the EEG θ+α band (EB), the mean and scale of the EEG response amplitude (EA), the mean and scale of the EEG signal-to-noise ratio (ES), and the mean and scale of the pupil diameter (PD) of 22 subjects in 18 conditions were compared. Statistics were performed using one-way repeated analysis of variance, **p<0.01. The difference between level 1 and level 4 represents the fatigue value generated by the subjects during the experiment; it can be seen that the signal-to-noise ratio of the overall evoked signal at low stimulation frequency is significantly lower than that at high stimulation frequency.

[0080] The Likert scale is a set of subjective questions about the subject's fatigue level. The first six questions of the scale are composed of 1 to 10, 1 indicating strongly disagree and 10 indicating strongly agree. By adding the scores of several questions, the subject's visual fatigue can be intuitively seen. The seventh question is composed of 1 to 5, 1 indicating very dislike and 5 indicating very like. The first six questions in the scale are used to reflect visual fatigue, and the seventh question is used to reflect preference. Comparing the Likert scale (SP) mean and SD values ​​of 22 subjects in 18 situations, it can be seen that the 7.5 Hz stimulation frequency is more popular with the participants than the 15 Hz stimulation frequency.

[0081] The evaluation indicators in the eighth step include visual fatigue VFS, which is the sum of the subjects' Likert scale scores, and the subjective preference SP obtained directly from the Likert scale, which reflects the subjective quantification of visual fatigue; and include EEG data of θ+α band energy EB, EEG response amplitude EA, response signal-to-noise ratio ES, and pupil diameter PD data directly recorded by the eye tracker, which are used to reflect the objective quantification of participants' visual fatigue from the perspective of the optic nerve and cerebral cortex.

[0082] In the eighth step, a multi-factor fusion visual fatigue assessment is performed, based on six indicators (VFS, SP, EB, EA, ES, PD) to explore the quantitative assessment of visual fatigue under three different combinations of environmental parameters (refresh rate, screen resolution and paradigm stimulation frequency);

[0083] In the first step, the indicators are divided into subjective quantification (VFS, SP) and objective quantification (EB, EA, ES, PD);

[0084] In the second step, for each indicator, one-way, two-way, and three-way repeated measures analysis of variance (ANOVA) with a significance level < 0.05 was used to conduct 1-level and 4-level analyses on the above 6 indicators of the 18 conditions;

[0085] In the third step, when there was a significant main effect, a post hoc Bonferroni adjustment was applied for multiple comparisons (p < 0.05);

[0086] In the fourth step, the information entropy-CRITIC weight method is used to integrate subjective and objective indicators to obtain the health of the display screen; the data on the health of the display screen is binned, and the consistency test is used to verify the effect.

[0087] Reference Figure 5 , the information entropy-CRITIC weighting method is specifically:

[0088] 1) Divide the data into n groups by category, n=6, representing VFS, SP, EB, EA, ES, and PD. The subjective part is divided into before and after the experiment. The objective part is divided into the first 1 / 4 as the least fatigued state and the last 1 / 4 as the most fatigued state according to the number of experiments.

[0089] 2) The subjective original fatigue value is defined as the value after the experiment minus the value before the experiment. The objective original fatigue value is defined as the difference between the least fatigue state and the most fatigue state according to the specific data type.

[0090] 3) Standardize the objective data obtained, and the ratio of the objective original fatigue value to the objective initial calibration value is the standardized objective original fatigue value;

[0091] 4) Map the data to the interval [0, 1] for forward or reverse processing. The forward or reverse processing of the data is based on the characteristics of the data. The larger the value, the better the forward processing, and the smaller the value, the better the reverse processing;

[0092] 5) Use the CRITIC algorithm to calculate the weights corresponding to n groups of data: w1, w2, ..., w n ;

[0093] 6) Calculate the information entropy corresponding to n groups of data: I1, I2, ..., I n ;

[0094] 7) The corresponding information entropy is multiplied by the CRITIC weight respectively, and then the sum is distributed to n groups of data in proportion: Ie-w1, Ie-w2, ..., Ie-w n ;

[0095] 8) Multiply the obtained Ie-w weight by the corresponding n groups of data quantization values ​​and then add them together to obtain the final fatigue quantization score value, which is the health of the display screen.

[0096] Reference Figure 6 , Figure 6 It is the different performance values ​​of the single indicator VFS under 18 groups of conditions. All the specific VFS values ​​here are obtained by adding up the evaluation scores of the subjects for each question in the questionnaire of the subjective visual fatigue Likert scale. The blue color represents the score result of the subjective visual fatigue Likert scale before the experiment, and the yellow color represents the score result of the subjective visual fatigue Likert scale after the experiment. It can be seen that the evaluation scores under the 18 groups of conditions have increased, indicating that the subjective fatigue level of the subjects has increased. The single indicator VFS can reflect visual fatigue to a certain extent from the perspective of subjective visual fatigue.

[0097] Reference Figure 7 , Figure 7These are the different performance values ​​of the single indicator SP under 18 groups of conditions. All the specific SP values ​​here are directly obtained from the subjects' evaluation scores on the Likert scale questionnaire questions on the subjective preference level. It can be seen that the subjects have different preferences for the 18 groups of conditions. The more the subjects like the conditions, the less visual fatigue they experience. The single indicator SP can reflect visual fatigue to a certain extent from the perspective of subjective preference.

[0098] Reference Figure 8 , Figure 8 It is the different performance values ​​of the single indicator EB under 18 groups of conditions. All the specific EB values ​​here are obtained by referring to the steps 1) to 4) of the information entropy-CRITIC weighting method. Blue represents the standardized + inverse result of the subject's least fatigued state under a single condition, and yellow represents the standardized + inverse result of the subject's most fatigued state under a single condition. It can be seen that the EB values ​​under the 18 groups of conditions have increased, the fatigue level has accumulated, and the energy has increased, which means that the objective fatigue level of the subjects has increased. The single indicator EB can reflect visual fatigue to a certain extent from the perspective of the objective θ+α frequency band energy value.

[0099] Reference Fig. 9 , Fig. 9 It is the different performance values ​​of the single indicator EA under 18 groups of conditions. All the specific EA values ​​here are obtained by referring to the steps 1) to 4) of the information entropy-CRITIC weighting method. Blue represents the standardized + inverse result of the subject's least fatigued state under a single condition, and yellow represents the standardized + inverse result of the subject's most fatigued state under a single condition. It can be seen that the EA values ​​under the 18 groups of conditions are all reduced, the fatigue level is accumulated, the brain activity of the subjects is reduced, and the response amplitude is correspondingly reduced, indicating that the objective fatigue level of the subjects has increased. The single indicator EA can reflect visual fatigue to a certain extent from the perspective of objective CCA amplitude.

[0100] Reference Fig.10 , Fig.10 It is the different performance values ​​of the single indicator ES under 18 groups of conditions. All the specific ES values ​​here are obtained by referring to the steps 1) to 4) of the information entropy-CRITIC weighting method. Blue represents the standardized + inverse result of the subject's least fatigued state under a single condition, and yellow represents the standardized + inverse result of the subject's most fatigued state under a single condition. It can be seen that the ES values ​​under the 18 groups of conditions are all reduced, the fatigue level is accumulated, the brain activity of the subjects is reduced, and the response signal-to-noise ratio is correspondingly reduced, indicating that the objective fatigue level of the subjects has increased. The single indicator ES can reflect visual fatigue to a certain extent from the perspective of the objective CCA signal-to-noise ratio value.

[0101] Reference Fig.11 , Fig.11It is the different performance values ​​of the single PD indicator under 18 groups of conditions. All the specific PD values ​​here are obtained by referring to the steps 1) to 4) of the information entropy-CRITIC weighting method. Blue represents the standardized + inverse results of the least fatigued state of the subject under a single condition, and yellow represents the standardized + inverse results of the most fatigued state of the subject under a single condition. It can be seen that the PD values ​​under the 18 groups of conditions are reduced, and the degree of fatigue is accumulated. Under a single paradigm stimulation, long-term gaze will cause the pupil diameter of the subject to become smaller, indicating that the objective fatigue degree of the subject has increased. The single PD indicator can reflect visual fatigue to a certain extent from the perspective of objective pupil diameter value.

[0102] Reference Fig.12 , CRITIC and information entropy-CRITIC weighted methods respectively integrated subjective evaluation data (VFS, SP), objective evaluation data (EB, EA, ES, PD) and overall subjective and objective data (all six indicators), and obtained the evaluation results of 18 groups of conditions, which were binned; then SPSS was used to perform Kappa consistency test. Compared with CRITIC, the information entropy-CRITIC weighted method improved the consistency of subjective (+45.90%) and objective (+202.34%) evaluation results, making the evaluation results more convincing.

[0103] Reference Fig.13 , Under the 7.5Hz paradigm stimulation frequency, at low refresh rates, low resolution scored the highest and had the best anti-fatigue effect, and the anti-fatigue effect worsened with increasing resolution. On the contrary, at high refresh rates, high resolution had the best anti-fatigue effect, and the anti-fatigue effect became better with increasing resolution. At medium refresh rates, low resolution had the best anti-fatigue performance, followed by high resolution.

[0104] Reference Fig.14 , Under the 15Hz paradigm stimulation frequency, when the refresh rate is low, high resolution has the highest score and the best anti-fatigue effect, and as the resolution increases, the anti-fatigue effect becomes better. Under medium and high refresh rates, medium resolution has the best anti-fatigue effect, followed by high resolution.

[0105] The above implementation modes are only used to illustrate the present invention rather than to limit the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. A multi-factor fusion visual fatigue assessment system based on SSVEP in BCI applications, characterized in that: include: Objective EEG signal acquisition module, objective eye movement signal acquisition module, subjective Likert scale evaluation module, screen parameter adjustment module; Objective EEG signal acquisition module: simulates two paradigm stimulation frequencies of 7.5Hz and 15Hz; Objective eye movement signal acquisition module: collects eye movement signals based on an eye tracker; Subjective Likert scale evaluation module: subjective evaluation based on subjective fatigue questionnaire and subjective preference questionnaire; Screen parameter adjustment module: a high refresh display screen whose screen parameters can be adjusted manually.

2. The evaluation system according to claim 1, characterized in that The described SSVEP multi-factor fusion visual fatigue assessment system in BCI applications is used in a closed dark room. A display with adjustable screen resolution and screen refresh rate is arranged in the dark room as a screen parameter adjustment module, and the screen brightness is calibrated using a HOPOO spectral brightness analyzer.

3. The evaluation system according to claim 1, characterized in that In the objective EEG signal acquisition module, electrodes are arranged in the occipital area, left and right temporal lobes and parietal lobes, and the acquisition electrodes are arranged in O1, O2, Oz, PO3, PO4, POz, T7, TP7, T8, TP8, Cz, FC1, FC2, F3, Fz and F4 areas in sequence. The ground electrode FPz is located on the forehead, and the reference electrode A1 is located on the left earlobe.

4. The evaluation system according to claim 1, characterized in that The objective EEG signal acquisition module sets 18 combination modes of three screen resolutions of 800×600, 1280×720, and 1920×1080 and three screen refresh rates of 120Hz, 240Hz, and 360Hz. The screen brightness contrast is adjusted to 287.6cd / m2, that is, the 50% mode. The combination of paradigm stimulation frequency, screen resolution, and screen refresh rate affects the visual perception of SSVEP-based BCI applications, that is, visual fatigue.

5. An evaluation method for a multi-factor fusion visual fatigue evaluation system in BCI application using the SSVEP described in claims 1-4, characterized in that: The following steps are involved: Step 1: Guide the subject to sit at a designated position in a fully enclosed dark room, with the head fixed using a headrest; Step 2: The subject wears the EEG cap and tests whether the EEG signal is recorded normally; Step 3: Turn on the eye tracker and calibrate it; Step 4: Adjust the corresponding screen parameters; Step 5: Questionnaire to record subjective fatigue level before the experiment; Step 6: Start the formal steady-state visual evoked potential stimulation experiment; Step 7: After the experiment, record the subjective fatigue level and the preference for the evaluation system after the experiment; Step 8: Organize all objective signal collection data, subjective fatigue inquiry data, and subjective preference inquiry data, set evaluation indicators, and conduct multi-factor fusion visual fatigue evaluation.

6. The evaluation method according to claim 5, characterized in that: Before each round of evaluation experiments, the visual perception of the subjects was first recorded using a Likert scale; secondly, the paradigm stimulation frequency, screen resolution, and screen refresh rate were adjusted; thirdly, in each group of experiments, the first three experiments were designed as white static "x" marks, and the subjects stared at the white static "x" marks on a dark background; Fourth, at the end of the experiment, the Likert scale was used to ask the subjects about their subjective feelings and preferences for the entire task; visual fatigue accumulated over the course of the experiment, and the 4-23 formal experiments for each task were evenly divided into four levels, with level 1 for 4-8 experiments representing the lowest level of fatigue at the beginning, and level 4 for 19-23 experiments representing the highest level of fatigue at the end; The EEG data were analyzed using the canonical correlation analysis (CCA) algorithm, which uses channel covariance information with a high signal-to-noise ratio to process the data. All data were standardized using the following procedure: in the 23 experiments, the first three stimulation experiments were not flashes but a fixed white mark; the first three stimuli were used as reference values, and then the values ​​of the subsequent 20 real flashes were divided by the reference values.

7. The evaluation method according to claim 6, characterized in that: The Likert scale described is a set of subjective questions about the subject's fatigue level. The first six questions of the scale are composed of 1 to 10, with 1 indicating strongly disagree and 10 indicating strongly agree; By adding up the scores of several questions, the subject's visual fatigue can be visually seen; The seventh question consists of 1 to 5, with 1 indicating very dislike and 5 indicating very like; the first six questions in the scale are used to reflect visual fatigue, and the seventh question is used to reflect preference.

8. The evaluation method according to claim 5, characterized in that: The evaluation indicators in the eighth step include visual fatigue VFS, which is the sum of the subjects' Likert scale scores, and the subjective preference SP obtained directly from the Likert scale, which reflects the subjective quantification of visual fatigue; and include EEG data of θ+α band energy EB, EEG response amplitude EA, response signal-to-noise ratio ES, and pupil diameter PD data directly recorded by the eye tracker, which are used to reflect the objective quantification of participants' visual fatigue from the perspective of the optic nerve and cerebral cortex.

9. The evaluation method according to claim 5, characterized in that: In the eighth step, multi-factor fusion visual fatigue assessment is performed. The following steps are involved: In the first step, the indicators are divided into subjective quantitative VFS, SP and objective quantitative EB, EA, ES, and PD; In the second step, for each indicator, one-way, two-way, and three-way repeated measures analysis of variance (ANOVA) with a significance level < 0.05 was used to conduct 1-level and 4-level analyses on the above 6 indicators of the 18 conditions; In the third step, when there was a significant main effect, a post hoc Bonferroni adjustment was applied for multiple comparisons at p < 0.05; The fourth step is to use the information entropy-CRITIC weight method to integrate subjective and objective indicators to obtain the health of the display screen; the data on the health of the display screen is binned, and the consistency test is used to verify the effect.

10. The evaluation method according to claim 9, characterized in that: The information entropy-CRITIC weighting method is specifically: 1) Divide the data into n groups by category, n=6, representing VFS, SP, EB, EA, ES, and PD. The subjective part is divided into before and after the experiment. The objective part is divided into the first 1 / 4 as the least fatigued state and the last 1 / 4 as the most fatigued state according to the number of experiments. 2) The subjective original fatigue value is defined as the value after the experiment minus the value before the experiment. The objective original fatigue value is defined as the difference between the least fatigue state and the most fatigue state according to the specific data type. 3) Standardize the objective data obtained, and the ratio of the objective original fatigue value to the objective initial calibration value is the standardized objective original fatigue value; 4) Map the data to the interval [0, 1] for forward or reverse processing. The forward or reverse processing of the data is based on the characteristics of the data. The larger the value, the better the forward processing, and the smaller the value, the better the reverse processing; 5) Use the CRITIC algorithm to calculate the weights corresponding to n groups of data: w1, w2, ..., w n ; 6) Calculate the information entropy corresponding to n groups of data: I1, I2, ..., I n ; 7) The corresponding information entropy is multiplied by the CRITIC weight respectively, and then the sum is distributed to n groups of data in proportion: Ie-w1, Ie-w2, ..., Ie-w n ; 8) Multiply the obtained Ie-w weight by the corresponding n groups of data quantization values ​​and then add them together to obtain the final fatigue quantization score value, which is the health of the display screen.

Images