Attention evaluation method and system based on multi-modal physiological signals, electronic equipment and medium

Abstract

The invention provides an attention evaluation method and system based on multi-modal physiological signals, electronic equipment and a medium. The method comprises the steps of obtaining eye movement data, collected by an eye movement tracker, of a subject and electroencephalogram data, collected by electroencephalogram equipment, of the subject in the process of presenting an attention evaluation task visual target to the subject; extracting eye movement data features based on the eye movement data and extracting electroencephalogram data features based on the electroencephalogram data; calculating correlation between the eye movement data features and the electroencephalogram data features; and determining a first attention evaluation result of the subject according to the correlation. By adopting the method and the system provided by the invention, multi-angle subject attention evaluation based on the multi-modal physiological signals can be realized, so that the concentration state of the subject is comprehensively evaluated, and the attention evaluation result is more objective and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of attention assessment, and in particular to an attention assessment method and system based on multimodal physiological signals, electronic equipment and medium. Background Art

[0002] In the prior art, the technical solutions for evaluating the level of attention are mainly: (1) manually clicking to locate the final position of the target object and giving the accuracy, reaction time, etc.; (2) judging the level of attention based on the accuracy, reaction time, etc.

[0003] However, since the location of the task target can be remembered through peripheral vision even when not focusing, the existing method of judging the level of attention using manual click data is not very accurate.

[0004] Therefore, how to design a method and / or system for automatically collecting data and judging the level of attention of the subject is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides an attention assessment method and system based on multimodal physiological signals to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides an attention assessment method based on multimodal physiological signals, comprising: obtaining eye movement data of the subject collected by an eye tracker and EEG data of the subject collected by an EEG device in the process of presenting a visual target of an attention assessment task to the subject; extracting eye movement data features based on the eye movement data and extracting EEG data features based on the EEG data; calculating the correlation between the eye movement data features and the EEG data features; and determining a first attention assessment result of the subject based on the correlation.

[0007] In some embodiments of the present invention, it also includes calculating a second attention evaluation result based on eye movement data, which step includes: calculating the overlap between the eye movement data and the moving path of the visual target, and standardizing the result of the overlap of the moving path as the second attention evaluation result; wherein the eye movement data includes a timestamp and corresponding gaze point coordinates.

[0008] In some embodiments of the present invention, the overlap between the eye movement data and the moving path of the visual target is calculated by calculating the overlap of trajectories, counting the gaze areas, and / or calculating the overlap in a heat map; wherein, by calculating the overlap of trajectories, the step of calculating the overlap of the eye movement data and the random moving path of the task target comprises: in the same graph, drawing the eye movement data trajectory based on the gaze point coordinates at different times contained in the eye movement data, drawing the motion trajectory of the randomly moving task target at the same starting point, and calculating the overlap between the eye movement data trajectory and the motion trajectory of the task target; counting the gaze areas, and calculating the overlap in a heat map; wherein, by calculating the overlap of trajectories, the step of calculating the overlap of the eye movement data and the random moving path of the task target comprises: drawing the eye movement data trajectory based on the gaze point coordinates at different times contained in the eye movement data, drawing the motion trajectory of the randomly moving task target at the same starting point, and calculating the overlap between the eye movement data trajectory and the motion trajectory of the task target; and calculating the overlap between the eye movement data trajectory and the motion trajectory of the task target by counting the gaze areas. The step of calculating the overlap between the eye movement data and the random moving path of the task target in an area manner includes: drawing the task target area according to the trajectory of the randomly moving task target, counting the number of times and the total time of fixation on the task target area in the eye movement data, and calculating the overlap of the fixation area according to the proportion of the number of times and the total time of fixation on the task target area in the moving time of the task target at any time; the step of calculating the overlap between the eye movement data and the random moving path of the task target in an area manner includes: generating a heat map according to the eye movement data, drawing the heat map according to the random moving path of the task target, and comparing the overlap of the two heat maps.

[0009] In some embodiments of the present invention, it also includes calculating a third attention evaluation result based on the EEG data, which step includes: extracting ERP waveforms of multiple trials according to a predetermined ERP time window within the time when the visual target appears in the EEG data, calculating the phase synchronization of different brain regions at different times, and calculating the third attention evaluation result based on the ERP waveform and the phase synchronization; wherein the EEG data contains a timestamp and a corresponding power spectral density.

[0010] In some embodiments of the present invention, a third attention evaluation result is calculated based on the ERP waveform and the phase synchronization, including: calculating the average power of the EEG data in each frequency band in different trials based on the power spectral density; comparing the average power of each frequency band before and after the task target appears, and using a statistical test method to evaluate whether there is a significant change in the average power; calculating the third attention evaluation result based on the statistical test evaluation results, the ERP waveform and the phase synchronization.

[0011] In some embodiments of the present invention, extracting eye movement data features based on the eye movement data includes: smoothing the eye movement data by moving average or Gaussian filtering; performing feature extraction on the smoothed eye movement data to obtain the eye movement data features; and / or extracting EEG data features based on the EEG data includes: filtering the EEG data to remove power supply noise and electromyography artifacts; performing feature extraction on the filtered EEG data to obtain the EEG feature data.

[0012] In some embodiments of the present invention, the eye movement data features include the standard deviation of the average gaze duration and the gaze point position, and the EEG data features include α wave power and β wave power; the correlation between the eye movement data features and the EEG data features includes a positive correlation and a negative correlation; calculating the correlation between the eye movement data features and the EEG data features includes: generating an eye movement feature graph in which the eye movement data features change over time; generating an ERP data graph in which the ERP data changes synchronously over time; drawing and analyzing a relationship scatter plot between the eye movement data features and the ERP data, and outputting the correlation results between the eye movement data represented by the eye movement data features and the EEG data represented by the ERP data based on the relationship scatter plot analysis results.

[0013] Corresponding to the above method, the present invention also provides an electronic device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any one of the above embodiments.

[0014] Corresponding to the above method, the present invention also provides an attention assessment system based on multimodal physiological signals, including an eye tracker, an electroencephalogram device and the electronic device described in claim 8, wherein the eye tracker is used to collect eye movement data of the subject, and the electroencephalogram device is used to collect electroencephalogram data of the subject.

[0015] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0016] The attention assessment method and system based on multimodal physiological signals proposed in the present invention can obtain attention assessment results based on separate eye movement data, separate EEG data and comprehensive eye movement data and EEG data respectively, and can realize multi-angle subject attention assessment based on multimodal physiological signals, thereby comprehensively evaluating the subject's concentration state, and the attention assessment results are more objective and accurate.

[0017] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0018] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0020] Figure 1 This is a flow chart of a method for evaluating attention based on multimodal physiological signals in one embodiment of the present invention.

[0021] Figure 2 It is a schematic diagram of the structure of the computer device in the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0023] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0024] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0025] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0027] In order to overcome the technical problems existing in the prior art, the present invention proposes an attention assessment method based on multimodal physiological signals. The method is based on an eye tracker and an EEG device worn by the participant at the same time, and simultaneously collects eye movement signals and EEG signals. First, the two signals are compared to see whether the task trajectories overlap, and then the two data are combined to analyze in which task his attention is focused. This method can analyze the attention of the subject (participant) by combining eye movement data and brain movement data, thereby improving the accuracy and reliability of the attention analysis results of the subject. Furthermore, the attention of the subject can be trained and improved based on the attention assessment method based on multimodal physiological signals. / / Eye movement signal collection uses: eye tracker, eye tracker. EEG signal collection uses: brain electrodes, EEG equipment.

[0028] The task target used in this method is a randomly generated trajectory that moves in a random direction and bounces back randomly if it hits a wall in a fixed area. Based on the task target and the collected eye movement signals, it is calculated whether the subject's attention is focused during the task time and an analysis is given. Based on the combination of eye movement and EEG technology for attention analysis, this method is suitable for scenarios such as personnel ability testing and training, disease auxiliary diagnosis and rehabilitation training.

[0029] This method can be specifically divided into the following steps: data collection, attention analysis and judgment, and auxiliary treatment (an extension of this scheme). (1) In the data collection step, an eye tracker is used to capture eye movement signals, and the captured eye movement signals are converted into eye movement data and recorded; an electroencephalogram (EEG) device is used to collect and analyze EEG data. (2) In the attention analysis step, the data of participants when specific task targets appear and when they do not appear are compared to determine whether their attention is focused. (3) In the judgment and auxiliary treatment step, the effect of the treatment intervention is evaluated and changes in attention are analyzed.

[0030] Figure 1 The flowchart of the method for evaluating attention of multimodal physiological signals in one embodiment of the present invention comprises the following steps:

[0031] Step S110: obtaining the eye movement data of the subject collected by the eye tracker and the EEG data of the subject collected by the EEG device in the process of presenting the visual target of the attention assessment task to the subject.

[0032] Step S120: extracting eye movement data features based on the eye movement data and extracting EEG data features based on the EEG data.

[0033] Step S130: Calculate the correlation between the eye movement data features and the EEG data features.

[0034] Step S140: determining the first attention evaluation result of the subject according to the correlation.

[0035] The attention evaluation method based on multimodal physiological signals proposed in the present invention can realize multi-angle subject attention evaluation based on multimodal physiological signals, thereby comprehensively evaluating the subject's concentration state, and the attention evaluation result is more objective and accurate.

[0036] In some embodiments of the present invention, for the collected eye movement data and EEG data, the method further includes: pre-smoothing the eye movement data by moving average or Gaussian filtering; and pre-filtering the EEG data to remove power supply noise and electromyographic artifacts.

[0037] Accordingly, extracting eye movement data features based on the eye movement data includes: smoothing the eye movement data in a moving average manner or a Gaussian filtering manner; extracting features from the smoothed eye movement data to obtain the eye movement data features. And / or, extracting EEG data features based on the EEG data includes: filtering the EEG data to remove power supply noise and electromyographic artifacts; extracting features from the filtered EEG data to obtain the EEG feature data.

[0038] By adopting the embodiment of the invention, the collected initial data can be processed through a variety of preprocessing methods, so that the data can more realistically fit the actual performance of the subjects.

[0039] In some embodiments of the present invention, it also includes calculating a second attention evaluation result based on eye movement data, which step includes: calculating the overlap between the eye movement data and the moving path of the visual target, and standardizing the result of the overlap of the moving path as the second attention evaluation result; wherein the eye movement data includes a timestamp and corresponding gaze point coordinates.

[0040] Furthermore, in some embodiments, the degree of overlap between the eye movement data and the moving path of the visual target is calculated by calculating the trajectory overlap, counting the gaze areas, and / or calculating the overlap in a heat map. Specifically, (1) the step of calculating the overlap of eye movement data and the random moving path of the task target by calculating the overlap of trajectories includes: in the same graph, drawing the trajectory of eye movement data based on the coordinates of the gaze points at different times contained in the eye movement data, drawing the motion trajectory of the randomly moving task target at the same starting point, and calculating the overlap of the eye movement data trajectory and the motion trajectory of the task target; (2) the step of calculating the overlap of eye movement data and the random moving path of the task target by counting the gaze areas includes: drawing the task target area according to the trajectory of the randomly moving task target, counting the number and total time of gazing at the task target area in the eye movement data, and calculating the overlap of the gaze area according to the proportion of the number and total time of gazing at the task target area in the duration of the task target moving at any time; (3) the step of calculating the overlap in the heat map includes: generating a heat map based on the eye movement data, drawing a heat map based on the random moving path of the task target, and comparing the overlap of the two heat maps.

[0041] By adopting the embodiment of the invention, the overlap between the eye movement data and the random moving path of the task target can be calculated based on multiple angles. Furthermore, in some embodiments of the present invention, the overlap calculated from the above multiple angles can be weighted averaged according to a preset weight ratio to obtain a comprehensive overlap, and the second attention evaluation result can be calculated based on the comprehensive overlap.

[0042] In some embodiments of the present invention, the method also includes calculating a third attention evaluation result based on the EEG data, which step includes: extracting ERP waveforms of multiple trials according to a predetermined ERP time window within the time when the visual target appears in the EEG data, calculating the phase synchronization of different brain regions at different times, and calculating the third attention evaluation result based on the ERP waveform and the phase synchronization; wherein the EEG data contains a timestamp and a corresponding power spectral density.

[0043] Furthermore, in some embodiments of the invention, a third attention evaluation result is calculated based on the ERP waveform and the phase synchronization, including: calculating the average power of the EEG data in each frequency band in different trials based on the power spectral density; comparing the average power of each frequency band before and after the appearance of the task target, and using a statistical test method to evaluate whether there is a significant change in the average power; calculating the third attention evaluation result based on the statistical test evaluation results, the ERP waveform and the phase synchronization.

[0044] By adopting this embodiment of the invention, a third attention evaluation result can be calculated based on separate EEG data.

[0045] Furthermore, based on the above embodiments, the method may also include: after obtaining the first attention assessment result, the second attention assessment result and the third attention assessment result, weighted processing the first attention assessment result, the second attention assessment result and the third attention assessment result according to preset weights to obtain a comprehensive attention assessment result.

[0046] In some embodiments of the present invention, the eye movement data features include average gaze duration and standard deviation of gaze point position, and the EEG data features include α wave power and β wave power.

[0047] Correspondingly, the correlation between eye movement data features and EEG data features is divided into positive correlation and negative correlation.

[0048] Calculating the correlation between eye movement data features and EEG data features includes: (1) generating an eye movement feature graph showing changes in eye movement data features over time; (2) generating an ERP data graph showing synchronous changes in ERP data over time; (3) drawing and analyzing a relationship scatter plot between eye movement data features and ERP data, and outputting a correlation result between eye movement data represented by eye movement data features and EEG data represented by ERP data based on the relationship scatter plot analysis result.

[0049] By adopting the embodiment of the invention, eye movement data and EEG data can be comprehensively considered based on the correlation between the extracted data features. Since the eye movement data features and the EEG data features are often positively correlated or negatively correlated with the degree of concentration, for example, the lower the α wave power, the higher the concentration, and the higher the β wave power, the higher the attention. If the ERP value of the participant at each time point in the task is more highly correlated with the eye movement data, it proves that the user is in a concentrated state during this time period. The evaluation results of the subject's attention state based on correlation are more representative and accurate.

[0050] In a specific implementation process, the correlation between the eye movement data features and the EEG data features can be calculated based on the pearsonr function provided by the SciPy database.

[0051] In some further embodiments of the present invention, the method further includes: in the process of calculating the first attention assessment results, pre-setting different correlation threshold assessment standards for different age or cognitive ability groups, and calculating the first attention assessment results for different age or cognitive ability groups according to the correlation threshold assessment standards.

[0052] By adopting this embodiment of the invention, attention assessment can be performed based on different relevance threshold assessment standards for different age or cognitive ability groups.

[0053] In a specific embodiment of the present invention, the specific method of implementing data collection is as follows: (1) The subject wears an eye tracker (such as Tobii, SensoMotoric Instruments, etc.) and an electroencephalogram (EEG) device at the same time. (2) The eye movement data is accurately measured and collected using the eye tracker. The recorded eye movement data includes: timestamp, coordinates and event type. Among them, the event type refers to a specific event in the attention assessment process, such as target appearance, target disappearance, etc. Based on the timestamp and coordinate changes contained in the eye movement data, the subject's scanning, fixation and review behaviors can be analyzed. Based on the eye movement data, the subject's gaze point, eye stability, scanning path, fixation time, review behavior, eye movement speed and other information can be analyzed. (3) The subject's electroencephalogram (EEG) data is collected using the electroencephalogram (EEG) device, and the power of different frequency bands (such as δ, θ, α, β, and γ waves) is calculated based on the electroencephalogram (EEG) data, and the ERP signal is extracted. For example, when concentrating attention, the power of the β wave usually increases, while the power of the α wave decreases. Among them, the extracted ERP (event-related potential, ERP) signal refers to the event-related potential, which is to extract the average waveform of the EEG signal before and after a specific event (such as the appearance of a target) to analyze the attention-related potential changes. The ERP signal is obtained based on the electroencephalogram and is a time domain analysis method for EEG signals.

[0054] In a specific embodiment of the present invention, the specific implementation method of collecting eye movement data is as follows: the participant needs to follow a specific target object (for example, an icon of a specific color) on the screen. First, wear a high-precision eye tracker (such as Tobii Pro or EyeLink). When the eyes follow the movement of the task target, record the screen coordinates of each gaze point, usually two-dimensional coordinates (X coordinate and Y coordinate), and the timestamp of each gaze point. Record the user's scanning path, that is, the movement trajectory of the eyes from one gaze point to another. The duration of each scan indicates the time the eyes spend in the scanning process. Calculate the number of times the user gazes in a specific task area (AOI, Area of ​​Interest), and count the total time the user gazes in a specific area.

[0055] In a specific embodiment of the present invention, the specific implementation method of collecting EEG data is as follows: the participant needs to remember the position where a specific target object (for example, an icon of a specific color) moves. First, wear an EEG device (such as a head-mounted electroencephalograph) and perform Fourier transform on the EEG signal to obtain the power of each frequency component. The power spectral density diagram shows the power of different frequency bands (such as δ, θ, α, β, and γ waves). Among them, the power spectral density (PSD) is used to describe the power distribution of the signal at different frequencies, and is used to indicate the change of signal power with frequency. The power spectral density diagram can record the power of different frequency bands.

[0056] When the subjects are in different states of attention, the active bands of EEG data will be different. When the subjects are in a state of focused attention: you may see a significant increase in the power of beta waves (13-30Hz), while the power of alpha waves (8-13Hz) and theta waves (4-8Hz) decreases. When the subjects are in a relaxed state: you may see an increase in the power of alpha waves (8-13Hz) and theta waves (4-8Hz), while the power of beta waves (13-30Hz) decreases.

[0057] In one embodiment of the present invention, by counting the number and duration of fixations, the user's attention to the task target area can be evaluated. Through deviation analysis, the consistency and overall deviation between the user's eye movement trajectory and the task target trajectory can be understood. In-depth understanding of the user's visual behavior and task execution effect can be achieved, and more accurate determination of whether the user's vision is within the task area can be made.

[0058] The process of analyzing eye movement data includes the following steps:

[0059] (1) Use moving average or Gaussian filtering to smooth the eye movement data to reduce noise and errors.

[0060] (2) Draw the eye movement data trajectory and the task target trajectory in the same chart, using different colors or line types to distinguish them. The eye movement trajectory is drawn in one color, such as blue, and the task target trajectory is drawn in another color, such as red. Calculate the minimum distance from the eye movement data trajectory point to the task target trajectory point. For each eye movement trajectory point, calculate the distance to each point of the task target trajectory and take the minimum value.

[0061] (3) The matching degree is evaluated by calculating the overlap between the eye movement trajectory and the task target trajectory, and algorithms such as Dynamic Time Warping (DTW) are used to measure the similarity between the trajectories.

[0062] (4) Count the number of times and total time that the eye movement data is fixed on the task target area, compare it with the expected fixation area of ​​the task target trajectory, calculate the average distance and standard deviation of the eye movement trajectory from the task target trajectory, and evaluate the overall degree of deviation.

[0063] (5) Generate a heat map for the eye movement data and compare it with the heat map of the task target area to check whether the eye movement focus area is consistent with the task target.

[0064] Furthermore, eye movement behaviors (including but not limited to fixation, scanning, and review) and eye movement speed can be identified based on eye movement data, and the proportion of the duration of fixation behavior in the task target existence duration in the eye movement behavior can be comprehensively considered to analyze the subject's attention level.

[0065] In the specific implementation process, the specific steps for analyzing eye movement data are as follows:

[0066] (1) Number and duration of fixations: Calculate the number of fixations and traverse each fixation point in the eye movement data to check whether it is in the task target area. The number of fixations = the number of fixations in the task target area. Calculate the total fixation time and sum the duration of all fixations in the task target area. The total fixation time = the sum of the duration of all fixations in the task target area. Example: The task target area is defined as a rectangular area from (250,250) to (300,300). There are 5 fixations in the eye movement data, and the total duration is 15 seconds.

[0067] (2) Deviation analysis: Calculate the minimum distance from each eye movement trajectory point to the task target trajectory. For each point in the eye movement trajectory, calculate the distance to each point on the task target trajectory and select the minimum value.

[0068]

[0069] Where (x1, y1) is the coordinate of the eye movement track point, and (x2, y2) is the coordinate of the point on the task target track. Calculate the average distance and standard deviation, and calculate the average and standard deviation of all minimum distances. Average distance = Σ(minimum distance) / N.

[0070]

[0071] (3) Trajectory overlap: Collect eye movement trajectory data: usually a series of coordinate points (x, y), and collect task target trajectory data: also a series of coordinate points (x, y). Ensure that the data format of the two trajectories is consistent, such as using the same time interval or sampling rate, and smooth the trajectories to reduce the impact of noise. DTW can be used to calculate the similarity of the two trajectories. Based on the DTW results, the overlap is calculated by inverting the DTW distance.

[0072] In one embodiment of the present invention, whether the participant's attention is focused before and after the target appears is verified by counting whether the average power of the frequency band before and after the target appears. ERP can be used to characterize the change of the average power of the frequency band.

[0073] The analysis of EEG data includes the following steps: (1) Continue to record EEG data before the task starts, when the task target appears, and after that. Use filters to remove power supply noise and electromyographic artifacts, extract the time period related to the appearance of the task target from the raw data, and remove DC drift or low-frequency trends in the data. (2) Mark the time point when the task target appears in the EEG data as an event marker, select a time window around the time when the task target appears (for example, -200 to 1000 milliseconds), and average the data of multiple trials to extract the ERP waveform. The ERP time window refers to the time period specified in the ERP system in which a specific operation or task can be performed. (3) Perform Fourier transform on the data within the time window and analyze the power of different frequency bands (such as alpha, beta, and gamma). Compare the changes in frequency band power before and after the task target appears. (4) Analyze the amplitude and latency of key components of the ERP waveform (such as P300 and N200). Use statistical methods (such as t-test or analysis of variance) to compare the frequency band power in different time periods. Analyze the phase synchronization of different brain regions and evaluate attention allocation.

[0074] In the specific implementation process, the specific methods of analyzing EEG data include: (1) The task target appears at a specific time point in the experiment. This time point can be recorded or an event code can be generated (for example, timing at the beginning of the task). The EEG signal in the time window is converted to the frequency domain using Fourier transform to obtain the power spectral density of each frequency. (2) The average power of each frequency band in all trials is calculated, and the power of the frequency band before and after the task target appears is compared. Statistical tests (such as t-tests) are used to evaluate whether there are significant changes.

[0075] In some embodiments of the present invention, the step of comprehensively analyzing the eye movement data and the EEG data includes: (1) ensuring that the eye movement data and the EEG data have been extracted and aligned to the same time period. Assume that we already have the following data structure: eye movement data, including timestamp, gaze point position, gaze duration, etc., EEG data, including timestamp, EEG signal and ERP data. (2) extracting useful features from the eye movement data and the EEG data, such as the average gaze duration and the standard deviation of the gaze point position. EEG data features such as alpha wave power and beta wave power. (3) calculating the overlap rate by calculating the correlation between the eye movement data features and the EEG data features, for each time period, extracting the corresponding eye movement features and EEG features, and calculating the Pearson correlation coefficient: using the pearsonr function in the scipy.stats library to calculate the correlation, analyzing the relationship between the eye movement features and the EEG features, the value is between -1 and 1, close to 1 indicates a strong positive correlation, and close to -1 indicates a strong negative correlation. Among them, the pearsonr correlation coefficient is used to measure the linear relationship between the two data sets. (4) Evaluate the degree of concentration based on the relationship between the eye movement data features and the EEG data features. If the beta wave power is high, it generally indicates high attention. If the alpha wave power is low, it also indicates high attention. Longer fixation duration also indicates high attention. We analyze the eye movement data and EEG data together. If the ERP value of the participant at each time point in the task is more correlated with the eye movement data, it proves that the user is in a state of concentration during this time period. (5) Output the ERP data graph of the ERP data changing over time, the eye movement feature graph of the eye movement feature changing over time, the eye movement feature graph of the eye movement feature changing synchronously with the ERP data and the ERP data, and analyze the relationship between the eye movement feature and the ERP data scatter plot. (6) Give different feedback on the output graphs. Give some suggestions for high, medium, and low attention and low attention. High attention: correlation coefficient > 0.7, medium attention: 0.3 < correlation coefficient ≤ 0.7, low attention: correlation coefficient ≤ 0.3.

[0076] Furthermore, based on the above embodiments, auxiliary diagnosis and treatment can be performed based on the attention assessment method based on multimodal physiological signals proposed in the present invention.

[0077] Consider the physiological and psychological differences of different populations. For example, children may have a shorter attention span, while the elderly may need to pay special attention to the impact of cognitive decline. Adjust the threshold and standard of the attention assessment tool according to the characteristics of the population. The present invention proposes two auxiliary diagnosis and treatment methods for different populations.

[0078] Method 1: Adjust the evaluation criteria and thresholds. For example: (1) The relevance threshold for children can be adjusted to 0.5, which is a lower relevance threshold. (2) The relevance threshold for the elderly can be adjusted to 0.6, which is a medium relevance threshold. (3) The relevance threshold for people with attention disorders can be adjusted to 0.4, which is a lower adaptive threshold.

[0079] Method 2: Design visual tasks suitable for different age groups or cognitive abilities. Basic principles: Use simpler visual tasks for children, and consider changes in memory and reaction speed for the elderly. For people with attention disorders, conduct long-term tracking and evaluation to observe changes in attention over time.

[0080] For example, for children: use simple image recognition tasks and add interactive elements. For older adults: design tasks that involve short-term memory and reaction speed, such as recalling simple patterns. For people with attention disorders: use dynamic and varied tasks and record performance in different environments.

[0081] Corresponding to the above method, the present invention also provides an attention assessment system based on multimodal physiological signals, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the method described above. Figure 2 It is a schematic diagram of the structure of the computer device in the present invention.

[0082] The attention assessment method and system based on multimodal physiological signals proposed in the present invention can obtain attention assessment results based on separate eye movement data, separate EEG data and comprehensive eye movement data and EEG data respectively, and can realize multi-angle subject attention assessment based on multimodal physiological signals, thereby comprehensively evaluating the subject's concentration state, and the attention assessment results are more objective and accurate.

[0083] The key points and advantages of this solution include but are not limited to: (1) It can analyze attention by combining eye movement data and EEG data, which can improve the accuracy of attention assessment. (2) It can analyze and calculate eye movement data and EEG data from different angles, and can intuitively give clear analysis results in the form of icons, etc., and combine analysis from different angles to comprehensively evaluate the subject's attention state.

[0084] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0085] See also Figure 2 The computer device 00 includes: a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0086] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0087] Corresponding to the above method, the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0088] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0089] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0090] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0091] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating attention based on multimodal physiological signals, characterized in that: include: Acquiring eye movement data of the subject collected by an eye tracker and EEG data of the subject collected by an EEG device during the process of presenting a visual target of an attention assessment task to the subject; Extracting eye movement data features based on the eye movement data and extracting EEG data features based on the EEG data; Calculate the correlation between the eye movement data features and the EEG data features; The first attention evaluation result of the subject is determined according to the correlation.

2. The method according to claim 1, characterized in that The step also includes calculating a second attention evaluation result based on the eye movement data, which includes: The overlap between the eye movement data and the moving path of the visual target is calculated, and the result after the normalization of the moving path overlap is used as the second attention evaluation result; wherein the eye movement data includes a timestamp and corresponding fixation point coordinates.

3. The method according to claim 2, characterized in that Calculating the overlap between the eye movement data and the moving path of the visual target is achieved by calculating the overlap of trajectories, counting the gaze areas, and / or calculating the overlap in a heat map; wherein, The step of calculating the degree of coincidence between the eye movement data and the random moving path of the task target by calculating the degree of coincidence of the trajectories includes: in the same graph, drawing the trajectory of the eye movement data based on the coordinates of the gaze points at different times contained in the eye movement data, drawing the motion trajectory of the randomly moving task target at the same starting point, and calculating the degree of coincidence between the trajectory of the eye movement data and the motion trajectory of the task target; The step of calculating the overlap between the eye movement data and the random moving path of the task target by counting the fixation areas includes: drawing the task target area according to the trajectory of the randomly moving task target, counting the number of times and the total time of fixation on the task target area in the eye movement data, and calculating the overlap between the fixation areas according to the proportion of the number of times and the total time of fixation on the task target area in the moving time of the task target at any time; The step of calculating the degree of overlap between the eye movement data and the random moving path of the task target by calculating the degree of overlap in the heat map includes: generating a heat map according to the eye movement data, drawing a heat map according to the random moving path of the task target, and comparing the degree of overlap between the two heat maps.

4. The method according to claim 1, characterized in that The step also includes calculating a third attention evaluation result based on the EEG data, the step comprising: The ERP waveforms of multiple trials are extracted according to a predetermined ERP time window within the time when the visual target appears in the EEG data, the phase synchronization of different brain regions at different times is calculated, and the third attention evaluation result is calculated based on the ERP waveform and the phase synchronization; wherein the EEG data includes a timestamp and a corresponding power spectral density.

5. The method according to claim 4, characterized in that The third attention evaluation result is calculated based on the ERP waveform and the phase synchronization, including: calculating the average power of the EEG data in each frequency band in different trials based on the power spectral density; comparing the average power of each frequency band before and after the task target appears, and using a statistical test method to evaluate whether there is a significant change in the average power; calculating the third attention evaluation result based on the statistical test evaluation results, the ERP waveform and the phase synchronization.

6. The method according to claim 1, characterized in that Extracting eye movement data features based on the eye movement data includes: smoothing the eye movement data by moving average or Gaussian filtering; extracting features from the smoothed eye movement data to obtain the eye movement data features; and / or, Extracting EEG data features based on the EEG data includes: filtering the EEG data to remove power supply noise and electromyographic artifacts; and extracting features from the filtered EEG data to obtain the EEG feature data.

7. The method according to claim 1, characterized in that The eye movement data features include the average fixation duration and the standard deviation of the fixation point position, and the EEG data features include α wave power and β wave power; the correlation between the eye movement data features and the EEG data features includes a positive correlation and a negative correlation; Calculate the correlation between eye movement data features and EEG data features, including: Generate an eye movement feature map showing the temporal changes of eye movement data features; Generate an ERP data graph showing the ERP data changing synchronously over time; Draw and analyze the relationship scatter plot between the eye movement data features and the ERP data, and output the correlation results between the eye movement data represented by the eye movement data features and the EEG data represented by the ERP data based on the relationship scatter plot analysis results.

8. An electronic device comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 7.

9. An attention evaluation system based on multimodal physiological signals, characterized in that: It comprises an eye tracker, an electroencephalogram device and the electronic device as claimed in claim 8, wherein the eye tracker is used to collect eye movement data of the subject, and the electroencephalogram device is used to collect electroencephalogram data of the subject.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images