Method of quantifying attention

Abstract

A method of quantifying attention includes receiving electroencephalogram (EEG) data corresponding to signals collected from a brain of a subject, the signals being synchronized with stimuli applied to the subject. The EEG data is segmented into a plurality of segments, each segment corresponding to a single stimulus. The method also includes dividing each segment of the EEG data into a first time window having a fixed onset relative to the respective stimulus and a second time window having a varying onset relative to the respective stimulus. The method also includes processing the first time window and the second time window to determine a likelihood of a given segment, the given segment describing a state of concentration of the brain.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 069,742, filed August 25, 2020, the entire contents of which are incorporated herein by reference.

[0003] Technical Field and Background Technology

[0004] In some embodiments, this invention relates to an electroencephalogram (EEG) analysis, and more specifically, but not exclusively, to systems and methods for quantifying attention based on such analysis. Some embodiments relate to systems and methods for quantifying fatigue and / or inattentiveness.

[0005] Electroencephalography (EEG), a non-invasive recording technique, is one of the commonly used systems for monitoring brain activity. In this technique, EEG data is collected simultaneously from multiple channels at a high temporal resolution, resulting in a high-dimensional data matrix used to represent the brain activity of a single trial. Besides its unparalleled temporal resolution, EEG is wearable and more affordable than other neuroimaging techniques, and has been used for various purposes, such as in brain-computer interface (BCI) applications, where brain activity is decoded in response to a single event (trial).

[0006] Traditional EEG classification techniques use machine learning algorithms to classify single-trial spatial-temporal activity matrices based on the statistical properties of these matrices. These methods are based on two main components: a feature extraction mechanism for efficient dimensionality reduction and a classification algorithm. A typical classifier uses a sample of data to learn a mapping rule that can classify other test data into one of two or more categories. Classifiers can be broadly categorized into linear and nonlinear methods. Nonlinear classifiers (such as neural networks, hidden Markov models, and k-nearest neighbors) can approximate various functions, allowing for the discrimination of complex data structures. While nonlinear classifiers have the potential to capture complex discriminative functions, their complexity can lead to overfitting and impose heavy computational requirements, making them less suitable for real-time applications.

[0007] On the other hand, linear classifiers are less complex and therefore more robust to overfitting. Linear classifiers perform particularly well on data that can be linearly separated. Fisher Linear Discriminant (FLD), Linear Support Vector Machine (SVM), and Logistic Regression (LR) are examples of classifiers. FLD finds a linear combination of features that map data from two classes onto a separable projection axis. The separating criterion is defined as the ratio of the mean distance between classes to the variance within each class. SVM finds a separating hyperplane that maximizes the margin between the two classes. As the name suggests, LR projects data onto a logistic function.

[0008] International Publication No. WO2014 / 170897 (the contents of which are incorporated herein by reference) discloses a method for single-trial classification of EEG signals of human subjects generated in response to a series of images including target and non-target images. The method includes: obtaining an EEG signal within a spatial-temporal representation, the spatial-temporal representation including time points and the corresponding spatial distribution of the EEG signal; independently classifying the time points using a linear discriminant classifier to calculate spatial-temporal discriminant weights; using the spatial-temporal discriminant weights to amplify the spatial-temporal representation at the time-space points to create a spatial weight representation; performing dimensionality reduction using Principal Component Analysis (PCA) in a time domain for each spatial channel of the EEG signal to create a PCA projection; applying the PCA projection to the spatial weight representation on a first plurality of principal components to create a temporally approximate spatial weight representation, which includes PCA coefficients for the plurality of principal temporal projections for each spatial channel; and using a linear discriminant classifier to classify the temporally approximate spatial weight representation on the plurality of channels to generate a binary decision series indicating whether each image in the image series belongs to a target image or a non-target image.

[0009] International Publication No. WO2016 / 193979 discloses an image classification method, the contents of which are incorporated herein by reference. A computer vision program is applied to an image to detect candidate image regions suspected of being occupied by a target. Each candidate image region is presented to an observer as a visual stimulus while neurophysiological signals are collected from the observer's brain. The neurophysiological signals are processed to identify neurophysiological events indicative of target detection by the observer. The presence of a target in the image is determined based on the identification of these neurophysiological events.

[0010] International Publication No. WO2018 / 116248 discloses a technique for training an image classification neural network. An image is presented to an observer as a visual stimulus, and neurophysiological signals are collected from his or her brain. These signals are processed to identify neurophysiological events that indicate the observer has detected a target in the image, and the image classification neural network is trained to identify targets in the image based on this recognition. Summary of the Invention

[0011] According to one aspect of some embodiments of the present invention, a method for estimating attention is provided. The method includes: receiving electroencephalogram (EEG) data corresponding to signals collected from the brain of a subject, the signals being synchronized with stimuli applied to the subject, the EEG data being segmented into multiple segments, each segment corresponding to a single stimulus; dividing each segment into a first time window and a second time window, the first time window having a fixed starting point and the second time window having a variable starting point, the fixed starting point and the variable starting point being related to their respective stimuli; and processing the first time window and the second time window to determine a probability of a given segment describing a state of concentration in the brain.

[0012] According to some embodiments of the present invention, the starting point of change is a random starting point.

[0013] According to some embodiments of the invention, the method includes receiving additional EEG data collected from the brain of the subject while intentionally not focusing on a portion of the stimulus. The additional EEG data is also segmented into multiple segments, each segment corresponding to a single stimulus. According to some embodiments of the invention, the method includes processing the segments of the additional EEG data to determine an additional probability of a given segment describing the state of focus of the brain; and combining the probability with the additional probability.

[0014] According to some embodiments of the present invention, the method includes representing each segment of the additional EEG data as a time-domain data matrix, wherein the processing includes processing the time-domain data matrix.

[0015] According to some embodiments of the present invention, the method includes representing each segment of the additional EEG data as a frequency domain data matrix, wherein the processing includes processing the frequency domain data matrix.

[0016] According to some embodiments of the present invention, the method includes representing each segment of the additional EEG data as a time-domain data matrix and a frequency-domain data matrix, wherein the processing includes processing the time-domain data matrix and the frequency-domain data matrix separately to provide two separate scores describing the additional possibilities, and wherein the combination includes combining a score describing the possibilities with the two separate scores describing the additional possibilities.

[0017] According to some embodiments of the present invention, the method includes receiving additional physiological data and processing the additional physiological data, wherein the probability is further based on the processed additional physiological data.

[0018] According to some embodiments of the present invention, the additional physiological data involves at least one physiological parameter selected from a group consisting of the number and time distribution of blinks, the duration of blinks, pupil size, muscle activity, movement, and heart rate.

[0019] According to some embodiments of the present invention, the method includes extracting spatial-temporal frequency features from the plurality of segments and clustering the spatial-temporal frequency features into a plurality of clusters of different states of consciousness.

[0020] According to some embodiments of the present invention, the state of consciousness includes at least one state of consciousness selected from the group consisting of a fatigued state, a focused state, a distracted state, a daydreaming state, a blank mind state, a conscious state, and a drowsy state.

[0021] According to some embodiments of the present invention, the first time window has a fixed width. According to some embodiments of the present invention, the second time window has a fixed width. According to some embodiments of the present invention, each of the first time window and the second time window has the same fixed width.

[0022] According to some embodiments of the present invention, the second time window has a variable width.

[0023] According to some embodiments of the present invention, the processing includes applying a linear classifier. According to some embodiments of the present invention, the nonlinear classifier includes a machine learning program.

[0024] According to some embodiments of the present invention, the processing includes applying a nonlinear classifier. According to some embodiments of the present invention, the nonlinear classifier includes a machine learning program.

[0025] According to one aspect of some embodiments of the present invention, a method for estimating attention is provided. The method includes: receiving electroencephalogram (EEG) data corresponding to signals collected from the brain of a subject, the signals being synchronized with stimuli applied to the subject, the EEG data being segmented into a plurality of segments, each segment corresponding to a single stimulus. The method further includes accessing a computer-readable medium storing a set of machine learning programs, each of the machine learning programs being trained to estimate attention, particularly for the subject, and each machine learning program being associated with a parameter indicating a performance characteristic of the machine learning program. The method further includes providing the plurality of segments to each machine learning program in the set of machine learning programs, and providing a score for each segment received from the machine learning program, the score indicating a probability of the plurality of segments describing a state of concentration in the brain, thereby providing an array of scores for each segment. The method further includes combining the scores based on the parameter indicating the performance to provide a combined score; and generating an output with the combined score.

[0026] According to one aspect of some embodiments of the present invention, a method for determining attention for a specific task is provided. The method includes: receiving electroencephalogram (EEG) data corresponding to signals collected from the brain of a subject, the signals engaging in brain activity over a time period, the time period including time intervals during which the subject performs a task of interest and time intervals during which the subject performs multiple background tasks; segmenting the EEG data into a plurality of partially overlapping segments according to a predetermined segmentation protocol independent of the subject's brain activity; assigning a vector of multiple values ​​to each segment, wherein one of the multiple values ​​identifies a task type corresponding to an overlapping time interval of the segment, and the other values ​​of the vector are multiple features extracted from the segment; providing the vectors assigned to the plurality of segments to a first machine learning program to train the first machine learning program to determine a probability of the segment corresponding to the time interval during which the subject performs the task of interest; and storing the trained first machine learning program in a computer-readable medium.

[0027] According to some embodiments of the present invention, at least one value of the vector is a frequency domain feature.

[0028] According to some embodiments of the present invention, the first machine learning program is a logistic regression program.

[0029] According to some embodiments of the present invention, the EEG data is arranged on M channels, each channel corresponding to a signal generated by an EEG sensor, and the vector includes at least 10M features, or at least 20M features, or at least 40M features, or at least 80M features.

[0030] According to some embodiments of the present invention, the task of interest is selected from a first group comprising a visual processing task, an auditory processing task, a working memory task, a long-term memory task, a language processing task, and any combination thereof.

[0031] According to some embodiments of the present invention, the task of interest is one of the qualifications in the first group, and the plurality of background tasks include all other qualifications in the first group.

[0032] According to some embodiments of the present invention, the method includes calculating a Fourier transform for each segment and providing the Fourier transform to a second machine learning program to train the second machine learning program to determine a probability of the segment corresponding to a time interval of the subject's attention.

[0033] According to one aspect of some embodiments of the present invention, a method is provided for determining a distracted or inattentive brain state. The method includes: receiving electroencephalogram (EEG) data corresponding to signals collected from the brain of a subject, the signals engaging in brain activity over a time period, the time period including a time interval in which the subject performs a prohibited task. The method further includes segmenting the EEG data into multiple segments, each segment being surrounded by a time interval in which no prohibited task begins; and, in response to an initial occurrence immediately following the segment, assigning a label to each segment based on a success or failure of the prohibited task. The method includes using the segments and the labels to train a machine learning program to estimate a probability of the segments corresponding to a time window in which the brain is in a distracted or inattentive state; and storing the trained machine learning program in a computer-readable medium.

[0034] According to one aspect of some embodiments of the present invention, a method for determining a state of consciousness is provided. The method includes: receiving electroencephalogram (EEG) data corresponding to signals collected from the brain of a subject, the signals being involved in brain activity over a time period; segmenting the EEG data into multiple segments according to a predetermined protocol independent of the subject's brain activity; extracting multiple classification features from the multiple segments and clustering the multiple classification features into multiple clusters; and ranking the multiple clusters according to a state of consciousness of the subject.

[0035] According to one aspect of some embodiments of the present invention, a method is provided for determining the state of consciousness of a specific subject within a group of subjects. The method includes: receiving electroencephalogram (EEG) data for the subjects in the group; extracting multiple categorical features from the EEG data; and clustering the multiple categorical features into L clusters, each cluster being characterized by a central vector of the multiple categorical features, thereby providing multiple L central vectors, one L central vector for each subject. The method further includes clustering the multiple central vectors into L central vector clusters; and for the specific subject, re-clustering the multiple categorical features using the L central vector clusters as multiple initial cluster seeds; and ranking the multiple clusters according to a state of consciousness of the subject.

[0036] According to some embodiments of the present invention, the method includes supplementing the plurality of classification features with the center vectors of the L center vector clusters prior to the re-clustering.

[0037] According to some embodiments of the present invention, the method includes segmenting the EEG data into multiple segments according to a predetermined protocol independent of the subject's brain activity.

[0038] According to some embodiments of the present invention, the predetermined protocol includes a sliding window.

[0039] According to some embodiments of the present invention, the predetermined protocol includes segmentation based solely on the EEG data.

[0040] According to some embodiments of the present invention, the segmentation is based on energy bursts within the EEG data.

[0041] According to some embodiments of the present invention, the segmentation is adaptive. For example, different segments may have different widths.

[0042] According to some embodiments of the present invention, the ranking is based on the membership levels of the multiple clusters according to the multiple fragments of the EEG data.

[0043] According to some embodiments of the present invention, the state of consciousness includes at least one state of consciousness selected from the group consisting of a fatigued state, a focused state, a distracted state, a daydreaming state, a blank mind state, a conscious state, and a drowsy state.

[0044] According to one aspect of some embodiments of the present invention, a computer software product is provided. The computer software product includes a computer-readable medium storing a plurality of program instructions that, when read by a data processor, cause the data processor to perform the method described above, and optionally or preferably further described below.

[0045] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. While similar or equivalent methods and materials described herein may be used to practice or test embodiments of the invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification (including definitions) shall prevail. Furthermore, materials, methods, and embodiments are illustrative only and are not necessarily limiting.

[0046] The implementation of the methods and / or systems of the embodiments of the present invention may involve manually, automatically, or in combination thereof, performing or completing selected tasks. Furthermore, in the actual instruments and apparatus of the embodiments of the methods and / or systems of the present invention, several selected tasks may be implemented using an operating system via hardware, software, or firmware, or in combination thereof.

[0047] For example, hardware for performing a selected task according to embodiments of the present invention can be implemented as a chip or circuit. As software, the selected task according to embodiments of the present invention can be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In exemplary embodiments of the present invention, one or more tasks according to exemplary embodiments of the methods and / or systems described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data and / or non-volatile memory for storing instructions and / or data, such as a hard disk and / or removable media. Optionally, a network connection is also provided. A display and / or a user input device such as a keyboard or mouse are also optionally provided. Attached Figure Description

[0048] Some embodiments of the invention are described herein by way of example only with reference to the accompanying drawings. Reference will now be made in detail to the drawings, with emphasis placed on the details shown as examples and for the purpose of illustrative discussion of embodiments of the invention. In this regard, the description taken in conjunction with the drawings will enable those skilled in the art to clearly understand how embodiments of the invention can be practiced.

[0049] In the attached diagram:

[0050] Figure 1 This is a method for estimating attention according to some embodiments of the present invention;

[0051] Figure 2 This is a flowchart of a method for estimating attention, in an embodiment of the invention, the method using labeled electroencephalogram (EEG) data;

[0052] Figure 3A and Figure 3B This is a schematic diagram of an architecture of a convolutional neural network (CNN) used in experiments according to some embodiments of the present invention;

[0053] Figure 4 The test scores shown are measures of a subject’s ability to succeed in a single trial, as obtained in experiments conducted according to some embodiments of the invention;

[0054] Figure 5 A comparison of the accuracy of a linear classifier and a CNN is shown, as obtained in experiments conducted according to some embodiments of the invention;

[0055] Figure 6 This is a chart prepared in experiments according to some embodiments of the present invention, in order to demonstrate the improvement in performance accuracy through data accumulation;

[0056] Figure 7 The normalized test scores, which are the average values ​​among subjects before (t<0) and after (t>0) and at rest (t=0) in an experiment conducted according to some embodiments of the invention, are shown.

[0057] Figure 8 The comparison between different scores obtained in experiments according to some embodiments of the present invention is shown;

[0058] Figure 9 Four classification methods for detecting the performance of a state of concentration are shown in experiments conducted using some embodiments of the present invention.

[0059] Figure 10 An attention index is shown, defined as the score obtained for each subject using a classifier that provides the highest performance to the subject, averaged over multiple subjects, as obtained in experiments conducted according to some embodiments of the invention;

[0060] Figures 11A to 11D The evoked response potentials (ERPs) of four subjects are shown, as obtained in experiments conducted according to some embodiments of the invention;

[0061] Figure 12The performance of a trial classifier is shown, as obtained in experiments conducted according to some embodiments of the invention;

[0062] Figure 13 Features were shown that were found to affect the logistic regression function used during experiments conducted according to some embodiments of the invention;

[0063] Figure 14A and Figure 14B The performance of a specific task attention classifier used during experiments according to some embodiments of the invention is shown;

[0064] Figure 15 The performance of the attention classifier used during experiments according to some embodiments of the invention is shown;

[0065] Figure 16 This is a schematic diagram of a clustering procedure according to some embodiments of the present invention;

[0066] Figure 17 Cluster membership levels for data segmentation of clusters associated with energy in the alpha band are shown, as obtained in experiments conducted according to some embodiments of the invention;

[0067] Figure 18 This is a schematic diagram of a graphical user interface (GUI) suitable for presenting the output of a clustering program according to some embodiments of the present invention;

[0068] Figure 19 The performance of a fatigue classifier used during experiments according to some embodiments of the present invention is shown;

[0069] Figure 20 This illustrates a distraction signal obtained in an experiment according to some embodiments of the present invention;

[0070] Figure 21 The performance of a distraction classifier used in experiments according to some embodiments of the present invention is shown;

[0071] Figure 22A and Figure 22B Exemplary combined outputs for estimating brain states according to some embodiments of the present invention are shown;

[0072] Figure 23 This is a flowchart describing a method for determining attention and / or attention for a specific task according to some embodiments of the present invention;

[0073] Figure 24A and Figure 24BThis is a flowchart describing a method for estimating the state of consciousness of the brain according to some embodiments of the present invention;

[0074] Figure 25 This is a flowchart describing a method for determining a state of inattentiveness or lack of focus according to some embodiments of the present invention. Detailed Implementation

[0075] In some embodiments, this invention relates to an electroencephalogram (EEG) analysis, and more specifically, but not exclusively, to systems and methods for quantifying attention based on such analysis. Some embodiments relate to systems and methods for quantifying fatigue and / or inattentiveness.

[0076] Before explaining at least one embodiment of the present invention in detail, it should be understood that the application of the present invention is not necessarily limited to the details of the construction and arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or embodiments. The present invention can have other embodiments or can be practiced or performed in various ways.

[0077] Human observers engage in a large number of tasks at a relatively high task rate (e.g., as airport X-ray security personnel repeatedly presenting images). Typically, their level of attention to the tasks they are instructed to perform decreases, whether momentarily or over a period of time. This decrease can be due to factors such as drowsiness, inattention, or distraction. Events of decreased attention can be overt or covert. Overt events are those that can be detected by monitoring the subject's external organs. For example, when the task involves viewing images on a screen, a noticeable drop in attention occurs when the subject stops looking at the screen, and can therefore be detected by monitoring the subject's gaze or head direction.

[0078] Covert events are those that involve decreased attention, in which the subject's external organs appear to be in the same state as when attention is high, and therefore cannot be detected by monitoring the external organs. For example, when the task involves viewing images on a screen, covert decreased attention occurs while the subject is still looking at the screen, but their brain is in a state where it cannot provide sufficient attention to the images on the screen.

[0079] The inventors have discovered a technique that can estimate attention by analyzing electroencephalogram (EEG) data. This technique can be used to detect covert attention deficit events, and optionally and preferably also to detect overt attention deficit events.

[0080] At least some of the operations described herein can be implemented by a data processing system, such as a dedicated circuit or a general-purpose computer, configured to receive data and perform the operations described below. At least some of the operations can be implemented via cloud computing facilities in remote locations.

[0081] Computer programs implementing the methods of this embodiment can typically be distributed to users via communication networks or on distribution media such as, but not limited to, floppy disks, CD-ROMs, flash memory devices, and portable hard disk drives. From the communication network or distribution media, the computer program can be copied to a hard disk or similar intermediate storage medium. The computer program can be executed by loading code instructions from its distribution medium or its intermediate storage medium into the computer's execution memory, configuring the computer to operate according to the methods of the invention. All of these operations are well known to those skilled in the art of computer systems.

[0082] The processing operations described in this article can be performed by processor circuits such as DSPs, microcontrollers, FPGAs, ASICs, or any other conventional and / or special-purpose computing systems.

[0083] The method of this embodiment can be embodied in various forms. For example, it can be embodied on a tangible medium such as a computer for performing the method operations. It can be embodied on a computer-readable medium, including computer-readable instructions for performing the method operations. It can also be embodied in an electronic device with digital computing capabilities, which is arranged to run a computer program on a tangible medium or execute instructions on a computer-readable medium.

[0084] The method of this embodiment can be embodied in various forms. For example, the method can be embodied on a tangible medium, such as a computer, for performing the method operations. The method can be embodied on a computer-readable medium, including computer-readable instructions for performing the method operations. The method can also be embodied in an electronic device with digital computing capabilities, the electronic device being arranged to run a computer program on a tangible medium or to execute instructions on a computer-readable medium.

[0085] The method begins at step 10 and optionally and preferably continues to step 11, whereby electroencephalogram (EEG) data is received. The EEG data may be either EEG data or magnetoencephalogram (MEG) data.

[0086] The EEG data is a digital form of the EEG signal, which is optionally and preferably collected simultaneously from multiple sensors (e.g., at least 4, 16, 32, or 64 sensors), and optionally and preferably collected with a sufficiently high temporal resolution. In the case of EEG, the sensor may be an electrode; in the case of MEG, the sensor may be a superconducting quantum interference device (SQUID).

[0087] In some embodiments of the invention, the signal is sampled at a sampling rate of at least 150 Hz, at least 200 Hz, or at least 250 Hz, for example, about 256 Hz. Optionally, a low-pass filter is employed to prevent high-frequency aliasing. A typical cutoff frequency of the low-pass filter is, but is not limited to, about 100 Hz.

[0088] When the neurophysiological signal is an EEG signal, one or more of the following frequency bands can be defined: delta band (typically from about 1 Hz to about 4 Hz), theta band (typically from about 3 to about 8 Hz), alpha band (typically from about 7 to about 13 Hz), low beta band (typically from about 12 to about 18 Hz), beta band (typically from about 17 to about 23 Hz), and high beta band (typically from about 22 to about 30 Hz). Higher frequency bands can also be considered, such as, but not limited to, the gamma band (typically from about 30 to about 80 Hz).

[0089] The EEG data corresponds to signals collected from the brain of a specific subject, synchronized with stimuli applied to that subject. When a stimulus is presented to an individual, for example during a task requiring the individual to identify a stimulus, a neural response is elicited in that individual's brain. The stimulus can be of any type, including but not limited to a visual stimulus (e.g., by displaying an image), an auditory stimulus (e.g., by generating a sound), a tactile stimulus (e.g., by physical contact with the individual or by changing the temperature to which the individual is exposed), an olfactory stimulus (e.g., by generating an odor), or a gustatory stimulus (e.g., by providing the subject with an edible substance). The response changes when attention to the stimulus is low, thus, by measuring neural activity, an individual's level of engagement in the task can be assessed.

[0090] The signal can be collected using the method described above, or the method can receive previously recorded data. For example, the method can use data collected during a training session involving the specific subject. The EEG data can optionally and preferably be segmented into multiple multichannel segments, each multichannel segment corresponding to a single stimulus applied to the subject. For example, the data can be segmented into multiple trials, where each multichannel segment contains N time points collected across M spatial channels, where each channel corresponds to a signal provided by one of multiple sensors. The multiple trials are typically segmented from a predetermined time before the stimulus begins (e.g., 300 milliseconds, 200 milliseconds, 100 milliseconds, 50 milliseconds) to a predetermined time after the stimulus begins (e.g., 500 milliseconds, 600 milliseconds, 700 milliseconds, 800 milliseconds, 900 milliseconds, 1000 milliseconds, 1100 milliseconds, 1200 milliseconds).

[0091] The method continues to step 12, where two time windows are defined for each segment. A first time window has a fixed starting point associated with the corresponding stimulus, and a second time window has a variable (e.g., random) starting point associated with the corresponding stimulus. The first time window preferably begins before the stimulus begins and ends after the stimulus begins. Therefore, it is referred to herein as a "true" test because it includes the beginning of the stimulus and thus contains data related to the brain's response to the stimulus. The starting point of the second time window varies from segment to segment and does not necessarily include the beginning of the stimulus. Therefore, the second time window is referred to herein as a "sham" test because the data it contains may or may not be related to the brain's response to the stimulus.

[0092] The first time window is preferably fixed at both its start and end points relative to the time window. The second time window varies relative to the start point of the time window, but has a fixed width in various exemplary embodiments of the present invention. In some embodiments of the present invention, the widths of the two time windows are the same or approximately the same.

[0093] Representative examples of the widths of the first and second time windows include, but are not limited to, approximately 10%, 20%, 30%, or 40% of the length of the segment. In some embodiments of the invention, the widths of the fixed and variable time windows are Δt, where Δt is approximately 100 milliseconds, or approximately 125 milliseconds, or approximately 150 milliseconds, or approximately 175 milliseconds, or approximately 200 milliseconds, or approximately 225 milliseconds, or approximately 250 milliseconds, or approximately 275 milliseconds, or approximately 300 milliseconds, or approximately 325 milliseconds, or approximately 350 milliseconds, or approximately 375 milliseconds, or approximately 400 milliseconds. In some embodiments of the invention, the starting point of the fixed time window is t1 milliseconds before the start of the stimulus, where t1 is approximately 200, or approximately 175, or approximately 150, or approximately 125, or approximately 100, or approximately 75, or approximately 50.

[0094] The method may optionally and preferably proceed to step 13, where the time window defined in step 12 is processed to determine a possibility of a given segment to describe a state of concentration of the brain.

[0095] The processing is preferably automatic and can be based on supervised or unsupervised learning of the data window. Learning techniques that can be used to determine the state of focus include, but are not limited to, Common Spatial Patterns (CSP), autoregressive models (AR), and Principal Component Analysis (PCA). CSP extracts spatial weights to distinguish between two classes by maximizing the variance of one class while minimizing the variance of the second class. In contrast, AR focuses on the temporal correlation, rather than spatial correlation, of a signal that may contain discriminative information. A linear classifier can be used to select the discriminative AR coefficients.

[0096] PCA is particularly useful for unsupervised learning. PCA maps the data to a new, typically uncorrelated space, where axes are ordered by the variance of the projected data samples along the axes, and only the axes that reflect the majority of the variance are retained. The result is a new data representation that retains the maximum information about the original data while providing effective dimensionality reduction.

[0097] Another approach for identifying a target detection event employs Independent Component Analysis (ICA) to extract a spatially weighted set and obtain the maximally independent spatial-temporal source. A parallel ICA stage is performed in the frequency domain to learn spatial weights for the independent temporal components. PCA can be applied separately to the spatial and spectral sources to reduce the dimensionality of the data. Each feature set can be classified individually using Fisher Linear Discriminant (FLD), and then optionally and preferably combined using Naive Bayes fusion (through multiplication of posterior probabilities).

[0098] In various exemplary embodiments of the present invention, the method employs a Spatially Weighted Fisher Linear Discriminant (SWFLD) classifier on the data window. The classifier can be obtained by performing at least some of the following operations: Time points can be classified independently to compute a spatial-temporal matrix of discriminant weights. This matrix can then be used to amplify the original spatial-temporal matrix by increasing the discriminant weights at each spatial time point, thereby providing a spatial weight matrix.

[0099] Preferably, SWFLD is supplemented by PCA. In these embodiments, PCA is optionally and preferably applied separately and independently in the time domain for each spatial channel. This represents the time series data as a linear combination of components. PCA is also optionally and preferably applied independently to each row vector of the spatial weight matrix. These two independent applications of PCA provide a projection matrix that can be used to reduce the dimensionality of each channel, thereby providing a dimensionality-reduced data matrix.

[0100] The rows of this reduced-dimensional matrix can then be concatenated to provide a feature representation vector, which represents the temporal approximation and spatial weighting activity of the signal. An FLD classifier can then be trained on this feature vector to classify the spatial-temporal matrix into one of two categories. In this embodiment, one category corresponds to real trials, and the other to fake trials.

[0101] In some embodiments of the invention, a non-linear procedure is employed. In these embodiments, the procedure may include an artificial neural network. An artificial neural network is a category of machine learning procedures based on the concept of interconnected computer program objects (called neurons). In a typical artificial neural network, a neuron contains multiple data values, each of which influences the value of the connected neuron according to a predefined weight (also called "connection strength") and whether the sum of connections to each particular neuron satisfies a predefined threshold. By determining appropriate connection strengths and thresholds (a process also known as training), the artificial neural network can achieve effective recognition of data patterns. Typically, these neurons are grouped into multiple layers. Each layer of the network may have a different number of neurons, and these may or may not be related to the specific quality of the input data. An artificial neural network with a multi-layered structure belongs to a category of artificial neural networks, also known as a deep neural network.

[0102] In an implementation called a fully connected network, each neuron in a particular layer is connected to every neuron in the next layer and provides input values ​​to each neuron in the next layer. These input values ​​are then summed, and this sum is used as an input to an activation function (e.g., but not limited to ReLU or Sigmoid). The output of the activation function is then used as the input to the neurons in the next layer. This computation continues through the layers of the neural network until it reaches the last layer. At this point, the output of the fully connected network can be read from the values ​​of the last layer.

[0103] A convolutional neural network (CNN) consists of one or more convolutional layers, in which the transformation of a neuron's value for subsequent layers is generated by a convolution operation. The convolution operation involves applying a convolutional kernel (also called a filter in the literature) multiple times, each time to a different block of neurons within the layer. The kernel typically slides across the layer until it visits all patch combinations. The output provided by the application of the kernel is called an activation map of the layer. Some convolutional layers are associated with more than one kernel. In these cases, each kernel is applied independently, and the convolutional layer is said to provide multiple activation maps, one for each kernel. Such a stack is often mathematically described as a subject with D+1 dimensions, where D is the number of horizontal dimensions for each activation map. The additional dimension is often called the depth of the convolutional layer.

[0104] In some embodiments of the present invention, the artificial neural network used in the method is a deep learning neural network, more preferably a CNN.

[0105] According to some embodiments of the present invention, an artificial neural network can be trained by providing labeled window data to an artificial neural network training program. For example, each window can be represented as a space-time matrix with N columns and M rows (or vice versa), where each matrix element stores a value representing the EEG signal sensed by a specific EEG sensor at a specific time point within the window. Each window provided to the training program is labeled. In some embodiments of the invention, binary labels are used during training. For example, a window can be labeled as a first window type with a fixed starting point (corresponding to a true trial) or a second window type with a varying starting point (corresponding to a sham trial). Since two types of windows can be defined in principle for each segment, the number of labeled windows provided to the artificial neural network training program can be twice the number of segments in the data, thereby improving the classification accuracy of the training process.

[0106] The training process adjusts the parameters of the artificial neural network (e.g., weights, convolutional kernels, etc.) to produce an output that classifies each window as closely as possible to its label. The final result of the training is a trained artificial neural network with adjusted weights assigned to each component (neuron, layer, kernel, etc.) of the network. The trained artificial neural network can then be stored in a computer-readable medium in step 14 and can be used later without retraining. For example, once extracted from the computer-readable medium, the trained artificial neural network can receive an unlabeled EEG data segment and produce a score, typically in the range [0, 1], which estimates the probability that the segment describes a state of concentration in the brain. Unlike artificial neural network training procedures that provide a first time window and a second time window for each segment of each EEG data, the subsequently used trained artificial neural network does not require two time windows for each segment. Instead, the trained artificial neural network can be provided by the EEG data segment itself, optionally and preferably followed by some preprocessing operations, such as, but not limited to, filtering and removal or artifact removal.

[0107] A representative example of a CNN architecture applicable to this embodiment is provided in the Examples section below.

[0108] In step 15, method 10 ends.

[0109] Figure 2 This is a flowchart of a method according to an embodiment of the present invention, wherein the method uses labeled EEG data. In these embodiments, the method begins at step 20 and continues to step 21, where the method receives EEG data collected from the subject's brain while instructing the subject to intentionally de-intend to a portion of the applied stimulus. As for the data received at step 11 ( Figure 1 The EEG data received in step 21 is also segmented into multi-channel segments, each corresponding to a single stimulus. Unlike the data received in step 11, the EEG data segments received in step 21 are labeled according to the subject's level of deliberate inattentiveness. Specifically, each segment of these EEG data is optionally and preferably labeled using a binary label, which indicates whether the subject was intentionally inattentive during the time interval contained in the respective segment. The EEG data received in step 21 is therefore referred to as labeled EEG data.

[0110] In some embodiments of the invention, the method continues to step 22, where additional physiological data is received. The additional physiological data may include any type of data related to attention. For example, such data may include data indicating the occurrence of a significant decrease in attention. Representative examples of additional physiological data suitable for this embodiment include, but are not limited to, data related to a physiological parameter selected from a group consisting of the number and timing distribution of blinks, blink duration, pupil size, muscle activity, movement, and heart rate.

[0111] The method may proceed to step 23, where the labeled segments of EEG data are processed to determine the probability of a given segment describing a state of concentration in the brain. Step 23 is preferably automatic and may be based on any of the aforementioned supervised or unsupervised learning techniques, except that in method 20, the segments are labeled based on the subject's intentional state of concentration, rather than on the predefined window type.

[0112] Preferably, step 23 is performed via an artificial neural network, as further detailed above. Since each segment is assigned a label (e.g., "0" for a focused state, "1" for a distracted state), the number of labeled segments provided to the artificial neural network training program in method 20 is equal to or less than the total number of segments in the data received at step 21. In embodiments of the invention, additional physiological data is received in step 22, which is also provided to the artificial neural network training program. Preferably, the values ​​of the additional physiological data are associated with corresponding windows based on the time points at which they are recorded. The additional physiological data serves as additional labels for the segments, thus improving accuracy. For example, when the additional physiological data relates to blinking, the presence of prolonged blinking or multiple short blinks may indicate that the brain is likely in a distracted state, and a corresponding label can be assigned accordingly.

[0113] In method 10 above, the input to the artificial neural network training program includes the window defined in step 12. Therefore, the input is in the time domain, for example, using the aforementioned spatial-temporal matrix. In method 20, the input does not necessarily have to be in the time domain because it is not based on a time window for each segment. Therefore, in some embodiments of the invention, the input to the artificial neural network training program is arranged in the time domain, and in some embodiments of the invention, the input to the artificial neural network training program is arranged in the frequency domain. Embodiments in which two artificial neural networks are trained are also envisioned: a time-domain artificial neural network is trained by providing the artificial neural network training program with data arranged in the time domain, and a frequency-domain artificial neural network is trained by providing the artificial neural network training program with data arranged in the frequency domain.

[0114] In the time domain, the input data can be arranged according to the principles described in method 10 above. In the frequency domain, the input data can be arranged by applying a Fourier transform to each plurality of channel segments to generate a spatial spectral matrix, wherein each matrix element stores a value representing the EEG signal sensed by a specific EEG sensor at a specific frequency band.

[0115] Within a frequency range of approximately 1 Hz to approximately 30 Hz, the typical number of frequency bins ranges from approximately 10 to approximately 100. Therefore, both temporal and frequency-domain artificial neural networks are trained to score each segment based on the probability of the brain being in a focused state within the time intervals it contains. The difference between these networks is that the input to the temporal network is based on the temporal bins, while the input to the frequency-domain artificial network is based on the frequency bins.

[0116] The trained artificial neural networks can then be stored in a computer-readable medium and used later without retraining, as further described above. Method 20 ends at step 25.

[0117] The inventors discovered that although both methods 10 and 20 provide a possibility of a state of concentration in the brain, the interpretation of the resulting possibility (e.g., the output of a trained artificial neural network) is different.

[0118] Method 10 determines the probability based on a statistical observation, namely, that a time window unrelated to the stimulus can be used to classify brain states according to the task required of the subject. Therefore, the probability provided by Method 10 assesses the similarity between a given trial and trials in which the subject successfully performs the task. In a sense, the probability provided by Method 10 is a measure of the subject's ability to succeed in a single trial. The inventors refer to this measure as "trialability," and the artificial neural network trained using Method 10 is called a trial network.

[0119] Method 20 determines the probability based on basic fact labels, thus providing the possibility that the subject's inability to successfully perform the task is due to inattentiveness, rather than, for example, some other reason.

[0120] The scores provided by the artificial network trained using methods 10 and 20 can optionally and preferably be combined. For example, unlabeled EEG data can be segmented into a group of segments collected from the brain of a specific subject and synchronized with stimuli applied to the subject over a period of time, where each segment corresponds to a single stimulus. A given set of unlabeled EEG data can be provided to the trained network. Each of these networks generates a score for a given unlabeled segment, thus providing a score array for the given unlabeled segment, one score per network. The score arrays can then be combined to provide a combined score that describes the subject's state of concentration during time intervals overlapping with the given unlabeled segment.

[0121] Preferably, the combination of scores is based on the performance characteristics of the trained artificial neural network for the specific subject. Therefore, in various exemplary embodiments of the invention, each trained artificial network undergoes a validation process to determine its performance characteristics. This can be done after the training of the artificial neural network. Typically, the data obtained before network training can be divided into a training data set and a validation data set. The training data set is provided to the training procedure, and the validation data set is provided to the trained network to compare the output of the trained network with the actual attention of the subject and to verify the network's ability to predict the subject's state of concentration.

[0122] In some embodiments of the invention, validation may include applying statistical analysis to the output, which is generated in response to the validation data set through each trained artificial neural network. Such analysis may include a computational statistical measure, such as a measure that characterizes the receiver operating characteristic (ROC) curve generated by the scores of the segments. For example, the measure may be the area under the ROC curve (AUC). Other or additional statistical measures may be calculated during validation and used to combine the scores according to some embodiments of the invention, including, but not limited to, at least one statistical measure selected from the group consisting of true positives, true negatives, false negatives, false positives, sensitivity, specificity, total accuracy, positive predictive value, negative predictive value, and Matthews correlation coefficient.

[0123] In some embodiments of the invention, performance features associated with each network trained by methods 10 and 20 are also stored in a computer-readable medium and linked with the trained networks to facilitate the combination of the scores. Additionally, or alternatively, a weighted reassembly calculated based on the performance features may be stored in a computer-readable medium and linked with the trained networks to facilitate the combination of the scores.

[0124] A representative example of a weight set that can be calculated according to some embodiments of the present invention is one containing weight w. i A set of {W} is defined as a ratio (P) i - P0) / (Σ) i P i - nP0), where P i Σ is the performance characteristic of the i-th network (e.g., the AUC of the i-th network). i P i The sum of all network performance characteristics is given, n is the number of networks used to generate the combined score (i = 1, 2, ..., n), and P0 is optionally and preferably a parameter not specific to the subject. For example, for performance characteristics in the range [0, 1], P0 can be set to about 0.5.

[0125] The combined score of a given unlabeled segment can optionally and preferably be calculated as a weighted sum of the scores, thereby using a ratio w i As the weight of the sum. Specifically, use S. i S represents the score provided by the i-th network to the given unlabeled segment, and the combined score S of the segments. TOT For S TOT = w1S1 + w2S2 + ... + w n S n , where n is the number of trained networks used to calculate the segment.

[0126] In some embodiments of the present invention, the score provided by the experimental network is combined with the score provided by a time-domain artificial neural network trained using method 20; in some embodiments of the present invention, the score provided by the experimental network is combined with the score provided by a frequency-domain artificial neural network trained using method 20; in some embodiments of the present invention, the score provided by the time-domain artificial neural network trained using method 20 is combined with the score provided by a frequency-domain artificial neural network trained using method 20; and in some embodiments of the present invention, the score provided by the experimental network is combined with the score provided by both the time-domain artificial neural network trained using method 20 and the frequency-domain artificial neural network trained using method 20.

[0127] The inventors of this invention have discovered that EEG data can also be used to estimate a subject's attention even when the EEG data is out of sync with the stimulus. This is advantageous because it allows for estimating the likelihood that a subject's brain is in a state of focus while the subject is performing a task undriven by the stimulus. For example, the subject may perform a task randomly or at time intervals chosen by the subject. This technique is useful when it is desirable to estimate the likelihood that a subject is focused on a specific task of interest or when it is necessary to estimate the likelihood that a subject is focused on a non-specific task. The technique of this embodiment is also useful when it is desirable to estimate the likelihood that a subject's brain is in a state of fatigue or inattentiveness.

[0128] Figure 23 This is a flowchart describing a method suitable for determining attention and / or focus for a specific task according to some embodiments of the present invention. The method begins at step 230 and continues to step 231, as further detailed above, where EEG data is received. The EEG data corresponds to signals collected from the brain of a subject participating in brain activity. During the brain activity, optionally and preferably, there are time intervals for the subject to perform the task of interest and time intervals for the subject to perform background tasks. The task of interest may, for example, be selected from a group consisting of a visual processing task, an auditory processing task, a working memory task, a long-term memory task, a language processing task, and one or a combination of two tasks. The background tasks may also be selected from the same group of tasks, provided that they do not include the task of interest itself.

[0129] The method optionally and preferably continues to step 232, where the EEG data is segmented into multiple segments, preferably partially overlapping segments. In some embodiments of the invention, the segmentation is performed according to a predetermined segmentation protocol independent of the subject's activity.

[0130] The protocol is independent of the subject's activity because no signals that induce the subject's activity are used to trigger the start or end of a segment or otherwise define the segment. This differs from segmentation in conventional induced response potential (ERP) tests, where the segmentation process locks in the signals used to generate or transmit stimuli to the subject.

[0131] A representative example of a segmentation protocol independent of the subject's activity and applicable to this embodiment includes, but is not limited to, using a sliding window of a predetermined width (or a predetermined group of widths) and a predetermined overlap (or a predetermined group of overlaps). Embodiments of the segmentation protocol based solely on the EEG data are also envisioned. For example, segments can be defined when the EEG data or its properties meet some predetermined criteria (e.g., exceeding a certain threshold, falling within a threshold range, etc.).

[0132] The method can proceed to step 233, where a vector is assigned to each segment. One component of the vector identifies a type of task (either a task of interest or a background task), the task corresponding to a time interval overlapping with the segment, and the other components of the vector are multiple features extracted from the segment. For example, one component of the vector could be a label indicating that the task performed by the subject during the corresponding time interval was the task of interest, while the other components could be multiple extracted features. Another example is a vector where one component is a label indicating that the task performed by the subject during the corresponding time interval was one of multiple background tasks, while the other components are multiple extracted features.

[0133] The extracted features can be of various types, such as, but not limited to, temporal features, frequency features, spatial features, spatial-temporal features, temporal-spatial features, spatial-temporal-frequency features, statistical features, ranking features, and counting features. Preferably, the number of features is greater than the number of EEG channels, more preferably greater than 10 times the number of EEG channels, more preferably greater than 20 times the number of EEG channels, more preferably greater than 40 times the number of EEG channels, and more preferably greater than 80 times the number of EEG channels. Representative examples of features applicable to this embodiment are provided in the Examples section below (see Table 5.1).

[0134] In some embodiments of the invention, the method proceeds to step 234, where a Fourier transform is calculated for each segment, thereby providing the spectrum of the EEG data within the segment. Optionally and preferably, a low-pass filter is applied to the Fourier transform. The cutoff frequency of the low-pass filter may be from about 40 Hz to about 50 Hz, for example, about 45 Hz.

[0135] The method optionally and preferably proceeds to step 235, whereby the vector assigned to the segment is used to train a machine learning program to determine a probability of a segment corresponding to a time interval in which the subject performs the task of interest. In various exemplary embodiments of the invention, the training process is specific to both the subject and the task of interest for which attention is to be estimated. Therefore, when there is more than one subject, the training process is preferably repeated individually for each subject, thereby generating multiple trained machine learning programs. Similarly, when it is desired to determine a probability of a segment corresponding to a time interval in which the subject performs another specific task, the training process is preferably repeated for other specific tasks, thereby generating a separate trained machine learning program for each task of interest.

[0136] The training is specific to the subject because the features forming the vectors are extracted from EEG data describing the subject's brain activity. The training is specific to the task of interest, and the components of the vectors identify whether the task is the task of interest or one of several background tasks, and are set based on tasks previously identified as the task of interest.

[0137] The machine learning program can be any of the types of machine learning programs described above. In the experiments conducted by the inventors, a logistic regression-type machine learning program was used. In an embodiment employing a logistic regression program, the training process uses a set of coefficients that defines a logistic regression function so that once the function is applied to features of a vector corresponding to a given segment, the logistic regression function returns the labeled components of the vector. The number of coefficients in the set of coefficients is typically the same as the number of features in the vector.

[0138] In some embodiments of the invention, the method proceeds to step 236, in which the spectrum obtained in step 234 (optionally and preferably after filtering) is used to train another machine learning program to determine a probability of a segment corresponding to a time interval in which the subject is attentive. The machine learning program trained in step 236 can be any of the types of machine learning programs described above. In the experiments conducted by the inventors, a CNN was used.

[0139] Similar to the training at step 234, the training at step 236 is specific to the subject; therefore, for multiple subjects, it is preferable to train multiple machine learning programs separately. Unlike the training at step 234, the training at step 236 is not task-specific. This can be achieved by labeling the segments non-specifically relative to the task. Thus, according to some embodiments of the invention, the training at step 236 includes labeling segments corresponding to the task of interest and segments corresponding to the background task using the same label. Segments corresponding to time intervals when the subject is not engaged in any task (or, equivalently, engaged in activities representing inattention) are labeled with a label different from the label assigned to the segment corresponding to the task. Therefore, the training process adjusts the parameters of the machine learning program, wherein the goal of the adjustment is that when the parameters are applied to a spectrum, the output of the machine learning program is as close as possible to the label associated with the spectrum.

[0140] The method can determine that the subject is likely focused when the output of the trained process in step 236 is close to the label assigned to a segment corresponding to a task (task of interest or background task). The method can set the output of the process as a score that defines the probability.

[0141] Then, at step 237, the trained machine learning programs can be stored in a computer-readable medium and can be used later without retraining them, as further detailed above.

[0142] Method 230 ends at step 238.

[0143] It is understandable that while method 230 described in the text identifies task-specific attention and attention or inattention, this is not always the case. For some applications, it may be necessary to identify task-specific attention rather than attention, and for others, attention rather than task-specific attention. In the former case (identifying only task-specific attention), steps 234 and 236 can be skipped. In the latter case (identifying only attention), steps 233 and 235 can be skipped.

[0144] Now for reference Figure 24A and Figure 24B This is a flowchart describing a method for estimating the state of consciousness of the brain according to some embodiments of the present invention. Figure 24A The flowchart in the diagram can be used when you want to determine whether a single subject's brain is in a specific state of consciousness, and Figure 24B The flowchart in the diagram can be used when it is desirable to determine whether the brain of a particular subject in a subject group is in a particular state of consciousness. The particular state of consciousness can be any state of consciousness that the brain may be in, including but not limited to a state of fatigue, a state of concentration, a state of inattention, a state of absent-mindedness, a state of blankness of thought, a state of wakefulness, and a state of drowsiness.

[0145] refer to Figure 24A The method begins at step 240 and continues to step 241, where EEG data is received, as further detailed above. The EEG data corresponds to signals collected from the brain of a subject engaged in brain activity.

[0146] The method proceeds to step 242, where the EEG data is segmented into multiple segments, preferably according to a segmentation protocol. Preferably, the segmentation protocol is predetermined, and more preferably, it is predetermined and independent of the subject's activity, as further detailed above. In some embodiments, the segmentation protocol employs a sliding window, as further detailed above, and in some embodiments, the segmentation protocol is based solely on the EEG data, as further detailed above. Preferably, but not necessarily, the segments are defined based on energy bursts within the EEG data. This can be achieved, for example, by applying a Hilbert transform to each channel of the EEG data to obtain a band envelope for that channel, and by applying a threshold to the band envelope to identify time intervals in which energy exceeds a predetermined threshold (energy burst). Multiple segments can then be defined based on the identified time intervals.

[0147] The method can proceed to step 243, in which each of the plurality of segments is assigned a label. The label is selected based on the task the subject is required to perform during the time interval overlapping with the corresponding segment and the desired state of consciousness. In various exemplary embodiments of the invention, the labels are binary. As a representative example, consider the possibility that the subject's brain is in a state of fatigue. Further consider that during the period of collecting the EEG signals, there are time intervals requiring the subject to perform tasks requiring attention (e.g., data input, reading, image viewing, driving, etc.), and time intervals requiring the subject not to perform any such tasks and to simulate a state of fatigue (e.g., closing their eyes). In this case, the segment overlapping with the time interval of the subject performing the attention-requiring task is assigned a label (e.g., "0"), and the segment overlapping with the time interval of the subject simulating a state of fatigue is assigned a different label (e.g., "1").

[0148] The method proceeds to step 244, where classification features are extracted from each segment. The classification features are optionally and preferably based at least on the frequency of the EEG data within the segment. For example, the method can (e.g., using Fourier transform) determine brainwave bands within the segment (e.g., alpha, beta, delta, theta, and gamma bands) and extract one or more features for each brainwave band. A representative example of features that can be extracted is the energy content of each brainwave band. These embodiments are particularly useful when a sliding window is used for segmentation in step 242. When the segmentation is based on energy bursts, the features may include at least one of the following: the peak amplitude of the burst in each band, the area under the envelope curve in each band, and the duration of the burst in each band.

[0149] The number of features extracted for each segment is represented by D, so each segment is assigned a one-D dimension feature vector at step 244.

[0150] The method continues to step 245, where a clustering procedure is applied to the features extracted in step 244 to initialize each cluster at the seed. This embodiment considers any clustering procedure, such as, but not limited to, Unsupervised Optimal Fuzzy Clustering (UOFC). Preferably, the clustering is performed to provide a predetermined number L of clusters. The initial cluster seed in the clustering procedure can be random, or more preferably, the initial cluster seed can be an input to the method (e.g., read from a computer-readable medium). A representative example of a technique for calculating cluster seeds is provided below.

[0151] The method optionally and preferably continues to step 246, where the clusters are ranked according to the subject's state of consciousness. The ranking can be based on membership levels of multiple segments of the EEG data for the multiple clusters. Specifically, for each cluster, the membership levels of all segments labeled with a tag identifying a state of consciousness of interest can be combined to provide a ranking score for the cluster, and the cluster producing the highest ranking score can be defined as a cluster representing a state of consciousness of interest. In conjunction with the example above of estimating the probability that the subject's brain is in a state of fatigue, the ranking score for each cluster can be calculated by combining the membership levels of all segments labeled "1", and the cluster producing the highest ranking score can be defined as a cluster representing a state of fatigue. The membership level is optionally and preferably in the range [0, 1]. The membership level can be defined as 1 / d i,j Proportional, where d i,j It is the distance from the j-th segment feature to the i-th cluster. Conveniently, a membership matrix representing the membership level of each segment in a given cluster can be constructed and used for ranking.

[0152] The method ends at step 247.

[0153] The parameters of the plurality of clusters obtained by method 240 may optionally and preferably be stored in a computer-readable medium for future use. For example, in some embodiments of the invention, the coordinates in the feature space of the center of one or more of the plurality of clusters may be stored in a computer-readable medium for future use. Preferably, at least the coordinates of the center of the cluster representing the state of interest are stored.

[0154] Stored cluster parameters can be used to assign a state of consciousness score to unlabeled data segments from the same subject. Such unlabeled data segments are typically obtained by collecting EEG signals from the same subject's brain during a later session, digitizing the EEG signals to form EEG data, and segmenting the EEG data using a segmentation protocol, such as a predetermined protocol, more preferably a predetermined protocol independent of the subject's activity. In conjunction with the above exemplary case, it is desirable to estimate the probability that a subject's brain is in a state of fatigue. The membership level of a given unlabeled data segment in a stored cluster previously characterized as a state of fatigue can be calculated (e.g., by calculating the distance in feature space between the feature vector of the segment and the center of the cluster), and this membership level can be used to estimate the probability that the brain is in a state of fatigue during time intervals overlapping with the given unlabeled data segment. In embodiments of the invention, the membership level is in the range [0, 1], and the probability can be the membership level itself. Alternatively, the probability can be defined by normalizing the membership level.

[0155] refer to Figure 24B The method begins at step 250 and continues to step 251, where EEG data is received for each of a plurality of subjects in a subject group. The EEG data corresponds to signals collected from the brains of the respective subjects involved in brain activity. Optionally and preferably, the EEG data for each subject is segmented and labeled, as further detailed above. The method continues to step 252, where categorical features are extracted from the EEG data collected for each subject, as further detailed above. In step 253, the categorical features are clustered for each subject, optionally and preferably using a random initialization seed. Preferably, clustering is performed to provide a predetermined number L clusters. Each of the obtained clusters is represented by a one-D dimension center vector of the feature, such that step 253 provides a plurality of L groups of center vectors, one L group of center vectors for each subject.

[0156] In this article, "group L" refers to a group containing L elements.

[0157] The method continues to step 254, where the D-dimensional centroid vectors are clustered among the subject groups. Clustering can be performed using any clustering procedure, including but not limited to a UOFC procedure. Preferably, clustering is performed to provide the same number L clusters as in step 253. Each of the clusters provided at step 254 also has a centroid, and the method, optionally and preferably, at step 255, extracts centroids from each of the clusters provided in step 254, resulting in a total of L new cluster centroids. In some embodiments of the invention, the method proceeds to step 256, where the characteristics of a specific subject in the group are re-clustered, except that the seed used for the clustering operation is the L new cluster centroids provided in step 255.

[0158] Optionally and preferably, prior to the re-clustering in step 256, the set of classification features extracted in step 252 is supplemented by the cluster centers extracted in step 255, such that the set of classification features applied to the re-clustering in step 256 is larger than the set of classification features applied to the clustering in step 253. The inventors have found that this expansion of the set stabilizes the performance of the method.

[0159] In step 257, the method ranks the clusters according to the subjects’ state of consciousness, as further detailed above, and in step 258, the method ends.

[0160] The parameters of one or more clusters obtained by method 250 may optionally and preferably be stored in a computer-readable medium for future use, as further detailed above. The stored cluster parameters can be used to assign a state of consciousness score to a subject in an unlabeled data segment, which may be the same subject to whom the clustering process was applied by method 250, or alternatively, a different subject. In other words, once the cluster parameters are stored, they can be considered generic and can be used for any subject.

[0161] Figure 25 This is a flowchart describing a method for determining a state of inattention or distraction according to some embodiments of the present invention. The method begins at step 300 and continues to step 301, where EEG data is received, as further detailed above. The EEG data corresponds to signals collected from the brain of a subject engaged in brain activity over a time period, wherein the time period includes a time interval during which the subject performs a prohibited task.

[0162] A prohibited task is a task that requires a participant to respond to a situation unless the situation meets some criteria, in which case the participant is required not to respond. A series of numbers may be presented to the participant, and they may be required to respond to the currently presented number (e.g., by typing the number), unless the number meets some criteria (e.g., the number is 3), in which case the participant is required not to respond.

[0163] The method may continue to step 302, where the EEG data is segmented at step 30. The segmentation preferably ensures that the start of the prohibited task (in the example above, the time instance showing the number "3") is entirely outside the segment. In other words, the segmentation is such that each segment is surrounded by a time interval in which no prohibited task begins. Preferably, the end of each segment is t milliseconds (ms) prior to any start of the prohibited task, where t is at least 50, at least 100, at least 150, or at least 200.

[0164] In step 303, a label is assigned to each segment based on the subject's misclassification error relative to the beginning immediately following the segment. Specifically, when the subject responds to the beginning immediately following the segment (a misclassification error), a first label, such as "1", is assigned to the segment, and when the subject does not respond to the beginning immediately following the segment (a correct rejection), a second label, such as "0", is assigned to the segment.

[0165] The method optionally and preferably continues to step 304, where the segment defined in step 304 and the label assigned in step 304 are used to train a machine learning program to estimate the probability of a segment corresponding to a time window in which the subject's brain is in a state of inattentiveness. The inventors have found that by keeping the start outside the segment and analyzing the EEG data containing the segment prior to the start, an inattentive state can be identified based on the label.

[0166] For example, consider a segment preceding a misclassification error. Since the subject made an error immediately following the start of that segment, it's likely the subject was in a state of inattention immediately before the start. The machine learning program captures the EEG data patterns of all such segments and attempts to find similarities among these patterns. On the other hand, consider a segment immediately preceding a correct rejection. Since the subject correctly identified that they should not react immediately to the start following that segment, it's likely the subject was not in a state of inattention before the start. The machine learning program also captures and attempts to find similarities between the EEG data patterns of these segments.

[0167] The trained machine learning process can then be stored in a computer-readable medium in step 305 and can be used later without retraining. During runtime, an unlabeled segment is provided to the trained machine learning process. The process determines which EEG pattern in the training data the unlabeled segment is most similar to and outputs an output accordingly.

[0168] The method ends at step 306.

[0169] Two or more of methods 10, 20, 230, 240, 250, and 300 can be combined to provide a combined method that provides a score for each of the aforementioned states. The methods can be executed sequentially or in parallel in any order.

[0170] As used in this article, the term "about" refers to 10%.

[0171] The terms “contains,” “includes,” “including,” “have,” and their variations mean “including but not limited to.”

[0172] The term "composed of" means "including and limited to".

[0173] The term "consistently made up of" means that a composition, method, or structure may include additional ingredients, steps, and / or portions, provided that the additional ingredients, steps, and / or portions do not materially alter the essential and novel features of the claimed composition, method, or structure.

[0174] As used herein, the singular forms “a,” “an,” and “the” include plural references unless the context clearly specifies otherwise. For example, the terms “a compound” or “at least one compound” can include a plurality of compounds, including mixtures thereof.

[0175] Throughout this application, various embodiments of the invention may be presented in a scope format. It should be understood that the scope format is merely for convenience and brevity and should not be construed as a rigid limitation on the scope of the invention. Therefore, the scope description should be considered as having specifically disclosed all possible sub-scopes and the individual values ​​within those scopes. For example, a description of a scope from 1 to 6 should be considered as having specifically disclosed sub-scopes, such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within those scopes, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the scope.

[0176] Whenever this document indicates a range of numbers, it is intended to include any referenced numbers (decimals or integers) within the indicated range. The phrases “range / range between” the first and second indicated numbers and “range / range from” the first indicated number to the second indicated number are used interchangeably in this document and are intended to include the first and second indicated numbers as well as all decimals and integers in between.

[0177] It should be understood that, for clarity, certain features of the invention described in the context of a single embodiment may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may also be provided individually or in any suitable sub-combination, or suitably provided in any other described embodiment of the invention. Certain features described in the context of various embodiments should not be considered as essential features of those embodiments unless the embodiment does not function without these elements.

[0178] The various embodiments and aspects of the invention described above and claimed in the claims are experimentally supported in the following examples.

[0179] Example

[0180] The following embodiments, together with the foregoing description, illustrate some embodiments of the invention in a non-limiting manner.

[0181] Example 1

[0182] Estimation of the experiment

[0183] method

[0184] EEG signals recorded from the brain were simultaneously presented to the subjects as a visual stimulus, consisting of a set of images. The data was segmented from -100 milliseconds (ms) to 900 milliseconds (ms) relative to the image start. Two sets of trimmed windows were extracted from these trials. A fixed-start window (“true trial”) was defined as ranging from -100 ms to 175 ms (window width 275 ms) relative to the image start, and a variable-start window (“sham trial”) was defined as including a random start point with the same width as the true trial.

[0185] The defined window is used to train a linear classifier and a non-linear classifier (a CNN in this example).

[0186] After training, the classifiers were fed EEG data of the same subjects, but during a different image review period. Each classifier produced a trial score array, which was smoothed using a moving average filter with a varying window size selected based on the desired accuracy and latency. In this example, window sizes ranging from 1 to 25 seconds were used.

[0187] Linear classifier

[0188] Each input segment consists of N EEG data samples across M channels.

[0189] For a data matrix X (data samples per channel, per segment), a weight matrix U (per channel of the data sample) is created using the FLD technique. The data matrix X is multiplied by the weight matrix U to amplify the difference between the experimental and non-experimental data. To reduce the data to K components, a projection matrix A (sampled K times per channel) is computed using temporal PCA, with each channel independent. The first K components of the PCA are retained. In this example, K is set to 6. The FLD is computed to select time points where the components and channels have greater weight.

[0190] CNN classifier

[0191] This example uses a CNN architecture for N=42 time points and M=19 channels. Figures 3A to 3B As shown in the image.

[0192] result

[0193] Single subject

[0194] The subjects performed three tasks: a focused task—finding an image containing the target, a distracted task—not looking at the image, and closing their eyes.

[0195] Figure 4 The test signal is shown, obtained from a test group and smoothed with a smoothing factor (window size) of 1 second (top panel), 2 seconds (second panel), 5 seconds (third panel), and 10 seconds (bottom panel). The attention threshold is marked by a thick black line. Blue corresponds to the time interval when the subject is focused on the image, red to the time interval when the subject is not focused on the image, and yellow to the time interval when the subject closes their eyes. Note that increasing the smoothing factor makes it easier to distinguish between focused and unfocused states. For example, in the bottom panel (smoothing factor of 10 seconds), all red dots are below the attention threshold, indicating that for this subject, the test score has a 100% success rate in detecting attention loss within 10 seconds.

[0196] 21 subjects

[0197] Twenty-one participants were asked to view a series of images of different categories and search for those images containing houses. These images were displayed on a computer screen at a frequency of 4 Hz. 2000 trials were used for training. To test the accuracy of the trials, the participants were again asked to search for houses (focused task, 800 trials), but also required to look away from the screen (look-away task, 400 trials), and while looking at the screen, to perform a distracted task (solving arithmetic problems), so they would be distracted by the images (focused task, 800 trials). Participants rested every 100 seconds.

[0198] Figure 5 A comparison of the accuracy between a linear classifier and a deep learning (CNN, in this example) classifier (see Methods) is shown. As illustrated, deep learning yielded a higher AUC for most subjects. For AUC calculation, data from the attentive task was assigned the label "1", while data from the inattentive task and the look-away task were assigned the label "0".

[0199] Figure 6 This demonstrates how performance accuracy increases with data accumulation. The positive decision rate for each condition is shown as a function of window size. The blue line represents the false positive rate (trials incorrectly detected as inattentive out of all genuine inattentive trials), while the yellow and red lines represent the true positive rates for eye-away and inattentiveness, respectively (trials correctly detected as inattentive out of all trials detected as inattentiveness). Moving along the time axis, an improvement in performance accuracy is observed as more data accumulates. For example, after 2 seconds, 95% of eye-away cases can be detected, but only one-third of inattentiveness cases.

[0200] Figure 7 The mean standardized trial scores of 21 subjects are shown before (t<0), after (t>0), and at rest (t=0). A series of t-tests were performed to test which time points attention shifted. In each t-test, all subjects' trials at a specific time point were compared to the median score (0.5). Significant time points (p<0.05) were observed at... Figure 5 The values ​​are highlighted (green indicates high test scores, red indicates low test scores). As shown in the figure, after the rest period, subjects exhibited higher test scores. This lasted for approximately 20 to 25 seconds. This is because subjects are generally more focused after the rest period. Figure 7 This demonstrates that the experimental measurements in this embodiment can be used as a measurement of attention.

[0201] This example demonstrates that the experimental measurement of this embodiment is effective in detecting obvious attentional shifts, where the subject looks away from the image or closes their eyes. This example also demonstrates that the experimental measurement of this embodiment is effective in detecting covert attentional shifts (when the subject looks at the images but does not pay attention to them) over an average period of approximately 15 seconds.

[0202] Example 2

[0203] Estimating attention from labeled EEG data

[0204] This example describes a time-domain and frequency-domain classifier trained on labeled EEG data. EEG signals were collected while subjects were instructed to gaze at the image without performing any task (covert loss of attention). Data was also collected with eyes closed (overt) and other covert and overt inattentive tasks. The classifier was then trained to distinguish between attentive and inattentive states. Both a time-domain classifier and a frequency-domain classifier were used.

[0205] method

[0206] EEG signals were recorded from the brain while a set of images was presented to the subject as a visual stimulus. The EEG signals were digitized to provide EEG data and preprocessed by applying a bandpass filter from 1 to 30 Hz and removing artifacts. The data was segmented from -100 ms to 900 ms relative to the start of the image. For a frequency domain classifier, a Fourier transform was applied to each segment, maintaining the frequency range of 1 Hz to 30 Hz.

[0207] The time-domain classifier is trained to distinguish between periods of focused attention and periods of inattentiveness, and the frequency-domain classifier is trained to distinguish between frequencies of focused attention and periods of inattentiveness.

[0208] After training, the time-domain and frequency-domain classifiers were input with EEG data obtained from the same subject, but during different image reviews.

[0209] Temporal classifier

[0210] Each input segment consists of N EEG data samples across M channels. The classifier in the example is a classifier with... Figures 3A to 3B The architecture shown is a CNN.

[0211] Frequency domain classifier

[0212] The input data for a single segment consists of K frequency bins across M channels. In this example, 30 frequency bins within the 1 to 30 Hz frequency range are used. The classifier in this example is... Figures 3A to 3B The architecture shown is a CNN.

[0213] result

[0214] 7 subjects

[0215] Seven participants were asked to perform four different tasks while a series of images of different categories were displayed on a computer screen at a rate of 4 Hz. In the first task, participants searched for images containing houses (a focused task). In the second task, participants were asked to look away from the screen (an overtly inattentive task). In the third task, participants were asked to look at the screen without paying attention to the displayed images (a covertly inattentive task). In the fourth task, participants were asked to close their eyes (an overtly inattentive task).

[0216] Figure 8 The test scores (blue bars) are shown as a comparison with the scores generated by the time-domain (red bars) and frequency-domain (orange bars) CNNs trained using the labeled EEG data. The AUC results are displayed for a two-second timeframe (eight images), for fixation inattention (top panel), gaze-away inattention (middle panel), and eye-closed inattention (bottom panel), detected by each of the three classifiers.

[0217] Figure 8 The results indicate that, for most subjects, the trial score is effective in detecting obvious inattention (eyes closed and fixation) with an AUC greater than 0.9. However, for occult inattention (fixation), some subjects (subject numbers 2, 3, 6, and 7) benefited from using the aforementioned time-domain or frequency-domain classifier.

[0218] Example 3

[0219] Combinatorial fractions

[0220] method

[0221] To combine different classifiers (trial, time-domain, and frequency-domain in this example), all three classifiers are used to classify the validation data, and the AUC of each classifier is calculated. For each subject, classifiers with an AUC less than 0.1 compared to the best classifier are discarded by assigning them zero weights. For the remaining classifiers, the weights are calculated using the following formula:

[0222]

[0223] AUC i It is the AUC value of the i-th classifier out of a total of n classifiers.

[0224] refer to Figure 8 In the top panel, the AUC values ​​for Subject 1's trial, time-domain, and frequency-domain classifiers are 0.733, 0.725, and 0.492, respectively. The weight of the third classifier is therefore set to zero because it is more than 0.1 smaller than the maximum AUC. The weights of Subject 1's first two classifiers are 0.509 and 0.491. The scores of the three classifiers are then normalized to values ​​between 0 and 1, multiplied by their respective weights, and summed. The resulting array of scores, one score for each trial, is used as an indicator of the likelihood that the subject's brain is in a state of concentration.

[0225] The combined classifier was tested in a cohort of 25 participants. Participants were asked to perform a series of tasks on three different dates.

[0226] Day 1

[0227] (i) Close your eyes for 5 minutes ("Close A").

[0228] (ii) Stare at the black screen for 5 minutes ("Open A").

[0229] (iii) Detect images of houses in seven other categories, which are displayed on a computer screen at a frequency of 4 Hz for 10 minutes.

[0230] (iv) Detecting an image with pixelated regions in a regular image, the image with pixelated regions being displayed on a computer screen at a frequency of 4 Hz (“pixel A”) for 10 minutes.

[0231] (v) Close your eyes for 5 minutes ("Close B").

[0232] the next day

[0233] (i) Detecting an image with pixelated regions in a regular image, the image with pixelated regions being displayed on a computer screen at a frequency of 4 Hz (“pixel B”) for 10 minutes.

[0234] (ii) Stare at the black screen for 5 minutes ("Open B").

[0235] (iii) Detecting an image with pixelated regions in a regular image, the image with pixelated regions being displayed on a computer screen at a frequency of 4 Hz (“pixel C”) for 10 minutes.

[0236] (iii) Gaze at the screen, where the image is displayed at a rate of 4 Hz (“gazing”) for 5 minutes.

[0237] Day 3

[0238] (i) Perform the 30-minute Uchida-Kraepelin Test (“UKTest”), which is a pen-and-paper task (adding numbers in a long line).

[0239] Attentive states were defined as tasks requiring subjects to detect targets (“house”, “pixel A”, “pixel B”, “pixel C”), while all other tasks were defined as inattentive states. Attentive and inattentive states were detected by classifying the collected data using a trial classifier, a time-domain classifier, a frequency-domain classifier, and a combined classifier.

[0240] result

[0241] Figure 9 The AUC performance of four classification methods for detecting attentive states is shown. As illustrated, the combined classifier achieved the highest AUC for 18 out of 25 subjects. For the other subjects, the other classifiers achieved the maximum AUC.

[0242] Figure 10 The attention index is shown, which is defined as the score obtained for each subject using a classifier, which provides the highest AUC for that subject, averaging over 25 subjects. Figure 10 This demonstrates the ability of the attention index to distinguish between focused and unfocused states. This can be accomplished by setting a threshold, where the brain is in a focused state when the attention index is above a predetermined threshold, and in an unfocused state when the attention index is below the predetermined threshold. In this example, the predetermined threshold could be approximately 0.76.

[0243] Example 4

[0244] Estimate the "experiments" used for auditory stimulation.

[0245] Four medical students were asked to listen to recordings of pathological stethoscope sounds (crackling sounds). The data were processed in the same manner as in Section 3 of Example 1, except that the fixed start window (“true trial”) was defined as -100 ms to 185 ms relative to the start of the auditory stimulus (window width 285 ms). A trial classifier was trained and tested separately for each subject. In addition, another classifier was trained for all combinations of data.

[0246] Figures 11A to 11D The evoked response potentials (ERPs) of the four subjects are shown, and Figure 12 The AUC of the trial classifier is shown. The numbers on the bars represent the number of trials used to train the classifier.

[0247] For the three subjects (Subject A, Subject B, and Subject D), the performance was sufficiently high (0.59 to 0.76). A classifier trained on combined data produced similar results (0.78). This example demonstrates the ability of the experimental measurements in this embodiment to estimate the probability that the brain is in a state of concentration, even when the stimulus is auditory.

[0248] Example 5

[0249] Estimate attention that is not synchronized with the stimulus

[0250] This example describes a technique for estimating attention in a case where EEG data is out of sync with stimuli. The technique can be used to estimate the likelihood of the brain being in a state of focused attention while performing an interest task that is not driven by stimuli. For example, the interest task can be performed at random time intervals or at time intervals chosen by the subject.

[0251] The described technique is a logistic regression-based machine learning procedure. The training of the machine learning procedure is specific to the subject and also specific to the task of interest requiring attention estimation. For a given type of task of interest (e.g., a visual processing task, an auditory processing task, a working memory task, a long-term memory task, a language processing task, multitasking, etc.), two sets of training tasks are selected. A first set includes attentive training tasks of the same type as the task of interest, and a second set includes inattentive training tasks of a different type. The training tasks in the first set simulate the task of interest, and the training tasks in the second set simulate attention loss while performing the task of interest.

[0252] This example illustrates procedures for two types of tasks of interest: data input-related tasks and image annotation-related tasks. To perform the data input-related task, participants are asked to locate specific data items and type them into a table. To perform the image annotation-related task, participants are asked to label bounding boxes around specific types of subjects in an image.

[0253] method

[0254] Task

[0255] In this example, the following task is used to generate training data for logistic regression.

[0256] Data input

[0257] Participants were shown an image containing various numerical data items (price, review score, number of reviewers for different products). In another session, participants were shown a table containing other types of data items (date, name, salary). Participants were asked to enter specific data values ​​into specific data fields in a table.

[0258] game

[0259] Subjects were shown an animation of descending numbers on a screen and asked to type in the numbers before they reached the bottom of the screen.

[0260] Distracted

[0261] The task was the same as the game above, but while observing the falling numbers, participants had to imagine their next holiday or last weekend.

[0262] read

[0263] Participants were presented with a passage on a randomly selected reading topic and were asked to rate their level of interest in the topic.

[0264] Sustained Attention Response Task (SART)

[0265] A series of numbers was displayed to the participants on a screen, and they were asked to press the corresponding number key on a keyboard after each number was displayed, except when the number was 3. The task was intentionally boring, and the chosen task was difficult to maintain focus on. Error was measured.

[0266] Image annotation

[0267] A series of images are shown to the subjects on a screen, and the subjects are asked to draw bounding boxes around specific objects in the images (e.g., large vehicles, bottles) on the screen.

[0268] Open your eyes

[0269] The subjects were asked to keep their eyes open and rest.

[0270] Close your eyes

[0271] The participants were asked to close their eyes and rest.

[0272] protocol

[0273] Nineteen participants took part in the experiment. Participants participated twice. In the first visit, participants were asked to perform data input, games, mindfulness, reading, SART (Self-Assisted Learning), open-eyes, closed-eyes, and image annotation tasks. In the second visit, participants were asked to perform reading, data input, closed-eyes, open-eyes, and image annotation tasks.

[0274] Data collection and labeling

[0275] EEG data was collected using a 1 / 3-second sliding window with a 5 / 6-second overlap between windows and was segmented into 2-second segments. The input data for classification consisted of 2D data segments at N time points across M channels for each data segment.

[0276] The data collected during the first visit was defined as the training data set, and the data collected during the second visit was defined as the validation data set.

[0277] The segment labels are either "0" or "1", depending on the task performed within each segment and the task of interest. Specifically, when the task of interest is data input, segments in which the subject performs the data input task are labeled "1", and segments in which the subject performs any other task are labeled "0". When the task of interest is image annotation, segments in which the subject performs the image annotation task are labeled "1", and segments in which the subject performs any other task are labeled "0".

[0278] Data Analysis

[0279] In this example, the machine learning program is trained to provide a score that estimates the likelihood of a particular subject's brain focusing on a specific task of interest, while defining all other activities the subject might engage in as background tasks. This score is referred to herein as "task-specific attention." In this example, the value of task-specific attention is in the range [0, 1].

[0280] The machine learning program is trained for each subject and separately for each task of interest.

[0281] The segmented EEG data was filtered through a bandpass filter ranging from 1 to 45 Hz. A vector of classification features was extracted for each data segment. The number of features was calculated depending on the number of electrodes, as some features were channel-specific, while others were inter-channel features. For example, for a 7-electrode EEG system, there were 723 classification features and one label.

[0282] The classification features used in this example are summarized in Table 5.1 below, where M is the number of channels (in this example, M=7).

[0283] Table 5.1

[0284]

[0285] Based on the distribution of feature scores in the training data, convert these feature vectors into Z-scores. Save the conversion procedure for use on the test data.

[0286] A logistic regression procedure is trained on the Z-score of the training set using labels assigned to each segment, thereby providing a trained logistic regression function defined by a set of learning coefficients, each corresponding to a set of features that form each of the feature vectors. Task-specific attention for a given segment of a particular subject's validation data set is computed by applying the trained logistic regression function (including coefficients learned for a specific subject) to the feature vector of the given segment.

[0287] result

[0288] Figure 13 Thirty-three features were shown that were found to affect the logistic regression function for a pool of 18 subjects. Figure 13 The following abbreviations were used:

[0289] std: Standard deviation of the signal

[0290] bpm: blink rate per minute

[0291] vpm: Vertical eye movement per minute

[0292] covM: (covariance of 2 channels)

[0293] eigenval: eigenvalues ​​of the covariance matrix

[0294] max: The maximum value of the signal

[0295] {Feature}_X: X indicates the index of the relevant electrode (channel).

[0296] {Feature}_X_Y: Features used to determine the interaction between the two electrodes at indices X and Y.

[0297] The trained logistic regression function obtained for each subject is applied to a fragment of the validation data set, and the correct detection of the state is then evaluated based on the assigned labels.

[0298] Figure 14A and Figure 14B This shows that for 19 subjects, when the task of interest was defined as data input ( Figure 14A ) and image annotation ( Figure 14B The AUC value for attention on a specific task is shown. An average AUC value obtained by averaging across all subjects is also provided. As shown in the figure, on average, the AUC of all classifiers exceeds 0.9.

[0299] Example 6

[0300] Estimation of attention

[0301] The inventors discovered that typical EEG patterns of general attention can be distinguished from typical EEG patterns of a specific task. This example describes a classifier that can be trained to detect whether a subject is paying attention, regardless of the specific task the subject is performing.

[0302] method

[0303] The task and protocol are the same as in Example 5.

[0304] Data collection and labeling

[0305] Collect EEG data and segment the EEG data into 2-second segments (stride = 0.5 seconds, 75% overlap).

[0306] The tags used in this example are summarized in Table 6.1 below.

[0307] Table 6.1

[0308]

[0309] Therefore, for all tasks where subjects were asked to provide an input positively correlated with the task objective (and thus indicating the subject's level of attention), a segment was non-specifically labeled "1". All other tasks were treated as background. Note that SART was treated as a background task because the count was the number of errors.

[0310] The data collected during the first visit is defined as the training data set, and the data collected during the second visit is defined as the validation data set (see Example 5: Protocol).

[0311] Data Analysis

[0312] For classification, a CNN was used. In this example, the CNN architecture is similar to... Figure 3A and Figure 3B The same as shown in the diagram. Then an intermediate value filter is applied to the classification score generated by the CNN.

[0313] During training, each segment is labeled according to the task performed in the segment, forming a vector of length N (the number of segments), denoted as Y_train.

[0314] Each segment undergoes preprocessing, including detrending, applying Fourier transform (n=300) to convert the spectrum to absolute values, and cropping at 45Hz. This provides a dataset matrix X_train of dimensions N x M x K, where M is the number of channels and K is the number of frequency bins.

[0315] CNN uses gradient descent (Adam Optimizer, learning rate 10). -4 (To conduct training)

[0316] result

[0317] The segments of the validation data set are fed into a trained CNN, such as the CNN obtained for each subject, and the scores provided by the CNN are evaluated based on the assigned labels to correctly detect the state.

[0318] Figure 15 The AUC values ​​of the scores obtained from the 19 subjects are shown. An average AUC value obtained by averaging across all subjects is also provided. As shown, on average, all classifiers achieve an AUC greater than 0.9, indicating that the procedure of this embodiment can estimate the probability of a subject's attention being focused, regardless of the specific task the subject is performing.

[0319] Example 7

[0320] Estimation of state of consciousness

[0321] The inventors discovered that typical EEG patterns for a conscious state of the brain can be distinguished from other EEG patterns through clustering. This example describes a clustering procedure that can detect whether the subject's brain is in a conscious state.

[0322] Given N ongoing EEG matrices ,in It is the first The number of samples per subject This refers to the number of electrodes, which triggers a clustering procedure. Now refer to... Figure 16 Describe the program.

[0323] The data matrix for each subject was preprocessed by applying a bandpass filter and removing blinks and artifacts. Segmentation was then applied to the data matrix for each subject. In this example, two types of segmentation were used.

[0324] In a first-type segmentation, the matrix is ​​segmented into 2-second windows, overlapping by 1 second, thus resulting in the... Each theme Excerpt.

[0325] In a second type of segmentation, referred to in this paper as burst analysis, a Hilbert transform is applied to each channel of the matrix to obtain a band envelope for that channel. Energy levels above a predetermined threshold are considered "bursts," and segments are defined based on the detected bursts.

[0326] Then, multiple features are extracted from each segment and each channel. When using the first type of segmentation, the multiple features are the energies in the Alpha, Beta, Delta, Theta, and Gamma bands. These features are extracted using a Fast Fourier Transform (FFT). When using the second type of segmentation, for each of the Alpha, Beta, Delta, Theta, and Gamma bands, the features are the peak amplitude of the burst in the respective band, the area under the envelope curve in the respective band, and the duration of the burst in the respective band. The number of features extracted for each segment is denoted by D, therefore each segment is assigned a D-dimensional feature vector.

[0327] Then, a first unsupervised optimal fuzzy clustering (UOFC) procedure applies multiple features of each subject to provide L clusters for each subject, and a total of N such L clusters (N is the number of subjects in this example). The cluster centers are randomly initialized. The D-dimensional center feature vector of the i-th cluster obtained by UOFC for the n-th subject is represented as C. n,i .

[0328] An additional UOFC procedure is applied to the D-dimensional center C. n,i (n=1, ..., N, i=1, ..., L), thereby providing an L cluster group of the D-dimensional center, denoted as {COC}.

[0329] Then, a further UOFC procedure is applied to multiple features for each subject, again providing L clusters for each subject, and a total of N said L clusters, except that in the further UOFC, the individual elements of the group {COC} are used as an initializer for each cluster center, instead of the random initializer used in the first UOFC procedure. Furthermore, the L cluster centers can also be added as features to the original feature group for the further UOFC re-clustering procedure.

[0330] The further output of UOFC is a membership matrix for each subject, where the membership matrix represents the membership (0-1) of a segment of a given cluster. The membership value is defined as 1 / d. i,j Proportional, where d i,j It is the distance from the j-th fragment feature to the i-th cluster. In this example, the exponential metric (e^(-d)) i,j 2) Used to measure the distance.

[0331] For each subject, the mean membership of the i-th cluster associated with tasks related to high fatigue or inattention is calculated, and the cluster that produces the highest mean membership value is defined as the "fatigue cluster". Note that, due to the COC, the selected cluster is also influenced by the eye-closing characteristics of other subjects.

[0332] Figure 17 Cluster membership of segments associated with energy in the alpha band is shown. The segment with eyes closed, representing a state of brain fatigue, has the highest membership, demonstrating that the clustering procedure of this embodiment is capable of detecting segments during a state of brain fatigue.

[0333] Figure 18 The diagram illustrates a representative example of a GUI that presents the output of a clustering procedure. The upper left region 181 shows cluster membership as a function of time. In this example, four clusters are used, each shown in a different color (yellow, blue, green, red). The upper right region 184 shows the cluster center of each cluster. The bottom region 186 shows the raw data for all channels (seven channels in this example) and the detected features (the envelope of the alpha band in this example). Several controllers can be provided on the GUI. One controller 188 allows the operator to select a band, a filter, and an envelope; another control 190 allows the operator to select subjects; and yet another control 192 allows the operator to select the number of clusters.

[0334] The clustering procedure described in this example was evaluated on the dataset of 19 subjects presented in Examples 5 and 6 above. The tasks were labeled such that closing the eyes represented a state of fatigue, simulating drowsiness. Therefore, segments with eyes closed were labeled "1". Segments with eyes open during a rest period after a long work task when a person is not focused were also labeled "1". Segments during other tasks were labeled "0".

[0335] Figure 19 The AUC values ​​obtained from 19 subjects are shown. An average AUC value obtained by averaging across all subjects is also provided. As shown, on average, the AUC value is greater than 0.9, indicating that the clustering procedure of this embodiment is able to estimate the state of consciousness of a subject's brain.

[0336] Example 8

[0337] Distracted

[0338] The inventors discovered that typical EEG patterns for inattentive states can be distinguished from other EEG patterns. This example describes a machine learning program that can detect whether the subject's brain is in an inattentive state.

[0339] EEG signals were collected from 10 subjects while they performed a SART task (see Example 5, Methods).

[0340] The EEG signal is preprocessed as detailed above, and then filtered into typical EEG bands (alpha, beta, gamma, and theta). The envelope signal of each standard band is extracted.

[0341] A segment of the EEG signal is collected starting at each prohibition trigger (initiated by the appearance of the number "3" on the screen). The segment lasts for 4 seconds, so the end of the segment is 200ms before the start. This 200ms offset ensures no leakage of the EEG signal after it begins entering the segment. Segments from the filtered signal and the envelope signal are collected in a similar manner and used as additional channels.

[0342] The 4s segment is considered a trial, and if the subject fails in the prohibited task, i.e., responds to the start (represented as "misclassification error"), it is labeled "1", and if the subject succeeds in the prohibited task, i.e., does not respond to the start (represented as "correct rejection"), it is labeled "0". Trials are collected from multiple subjects and mixed together to form the X_train matrix and the Y_train vector containing the labels.

[0343] The X_train matrix and Y_train vector are used to train a neural network using gradient descent (Adam optimizer, learning rate 10⁻⁵). The model is fine-tuned using the subjects' individual data. For this purpose, the neural network is trained with a low learning rate on a small dataset consisting only of trials from specific subjects, and the two bottom layers of the network are frozen simultaneously.

[0344] An ensemble of five neural networks was formed, with each network distinct from the others by excluding different subject groups from the training group. The subjects excluded from the training group served as a validation group for evaluation and early termination. The neural networks that achieved an AUC score exceeding 0.65 on the validation group, obtained only from trials with specific subjects, formed the final ensemble.

[0345] For prediction, the EEG signal is segmented into 4-second segments (sliding windows with a stride of 0.5 seconds, i.e., 75% overlap). Each segment is fed forward into each neural network that makes up the whole, resulting in a set of scores, one for each neural network. The average of the score sets is defined as the score of the segment. The scores are aligned such that the score at time t corresponds to the 4-second window ending at time t. This procedure produces a distraction score signal, sampled at 2Hz, with the first 7 values ​​being zero. The first non-zero value (at the 8th index) corresponds to the time window t = [0...4] seconds. The distraction score signal is then smoothed with a Gaussian filter (std = 3, n_samples = 10), and the time intervals during which the distraction score signal is above a predetermined threshold (0.7 in this example) are defined as distraction periods.

[0346] Figure 20 The image shows a representative example of a distraction signal from subject number 2.

[0347] Figure 21 The AUC for the misclassification error prediction calculated for each of the 10 subjects is shown. As illustrated, on average, the AUC value is close to 0.8, indicating that the procedure in this embodiment is able to estimate the probability that the subject's brain is in a state of inattention.

[0348] Example 9

[0349] Exemplary combined output

[0350] An exemplary combined output for estimating brain state is in Figure 22A and Figure 22B As shown, for data input tasks ( Figure 22A ) and image annotation tasks ( Figure 22BThe timeline also shows other tasks, including reading, data input, eyes-closed, eyes-open, and image annotation. For descriptions of these tasks, please refer to the methods in Example 5. Figure 22A and Figure 22B The estimated brain states are attention (top), task-specific attention (middle), and fatigue (bottom), as described in Examples 5, 6, and 7, which illustrate the procedures used to estimate these states. As shown, attention scores are higher during reading, data input, and image annotation, and lower during inattentive tasks with eyes open and closed. Task-specific attention is high during segments of user engagement with tasks of interest and low otherwise.

[0351] Although the invention has been described in conjunction with specific embodiments thereof, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, it is intended to cover all such alternatives, modifications, and variations falling within the spirit and broad scope of the appended claims.

[0352] The applicant intends that all publications, patents, and patent applications mentioned in this specification be incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application were specifically and individually attributed to be incorporated herein by reference. Furthermore, any reference or identification of any reference in this application should not be construed as an admission that such reference is used as prior art of the invention. The use of section headings should not be construed as necessarily limiting. Additionally, any priority documents of this application are incorporated herein by reference in their entirety.

Claims

1. A method for quantifying attention, comprising: Receive electroencephalogram (EEG) data corresponding to signals collected from a subject's brain, said signals being synchronized with stimuli applied to said subject, said EEG data being segmented into multiple segments, each segment corresponding to a single stimulus; Each segment is divided into a first time window and a second time window. The first time window has a fixed starting point, and the second time window has a varying starting point, which varies depending on the segment. The varying starting point is a random starting point. The fixed starting point and the varying starting point are related to their respective stimuli. The first time window includes the beginning of the stimulus, and the second time window is not related to the brain's response to the stimulus. The first time window and the second time window are processed to determine a probability of a given segment, the probability describing a state of concentration in the brain.

2. The method for quantifying attention as described in claim 1, further comprising: Additional EEG data collected from the subject's brain is received while the subject is intentionally not focused on a portion of the stimulus. The additional EEG data is also segmented into multiple segments, each corresponding to a single stimulus. The fragments of the additional EEG data are processed to determine an additional probability of the given fragment, the additional probability describing the state of concentration of the brain; as well as Combine the stated possibility with the stated additional possibility.

3. The method for quantifying attention as described in claim 2, further comprising: Each segment of the additional EEG data is represented as a time-domain data matrix, wherein the processing includes processing the time-domain data matrix.

4. The method for quantifying attention as described in claim 2, further comprising: Each segment of the additional EEG data is represented as a frequency domain data matrix, wherein the processing includes processing the frequency domain data matrix.

5. The method for quantifying attention as described in claim 2, further comprising: Each segment of the additional EEG data is represented as a time-domain data matrix and a frequency-domain data matrix, wherein the processing includes processing the time-domain data matrix and the frequency-domain data matrix separately to provide two separate scores describing the additional possibilities, and wherein the combination includes combining a score describing the possibilities with the two separate scores describing the additional possibilities.

6. The method for quantifying attention as described in any one of claims 1 to 5, further comprising: Receive additional physiological data and process the additional physiological data, wherein the probability is also based on the processed additional physiological data.

7. The method for quantifying attention as described in claim 6, characterized in that: The additional physiological data involves at least one physiological parameter selected from a group consisting of the number and timing distribution of blinks, the duration of blinks, pupil size, muscle activity, movement, and heart rate.

8. The method for quantifying attention as described in any one of claims 1 to 5, further comprising: Spatial-temporal frequency features are extracted from the multiple fragments, and the spatial-temporal frequency features are clustered into multiple clusters of different states of consciousness.

9. The method for quantifying attention as described in claim 8, characterized in that: The state of consciousness includes at least one of the following groups: a state of fatigue, a state of concentration, a state of inattentiveness, a state of distraction, a state of blank thinking, a state of wakefulness, and a state of drowsiness.

10. The method for quantifying attention as described in claim 1, characterized in that: The first time window has a fixed width.

11. The method for quantifying attention as described in any one of claims 1 or 10, characterized in that: The second time window has a fixed width.

12. The method for quantifying attention as described in claim 1, characterized in that: Each of the first time window and the second time window has the same fixed width.

13. The method for quantifying attention as described in any one of claims 1 to 5, characterized in that: The second time window has a variable width.

14. The method for quantifying attention as described in any one of claims 1 to 5, characterized in that: The process includes applying a linear classifier.

15. The method for quantifying attention as described in any one of claims 1 to 5, characterized in that: The process involves applying a nonlinear classifier.

16. The method for quantifying attention as described in claim 15, characterized in that: The nonlinear classifier includes a machine learning program.

17. A computer software product comprising a computer-readable medium storing a plurality of program instructions, wherein when the plurality of program instructions are read by a data processor, the data processor performs the method of quantizing attention as described in any one of claims 1 to 5.

Images