Systems and methods for cognitive enhancement based on electroencephalogram (EEG) signals

Abstract

The present invention provides a system and method for modifying the presentation of digital content on a display based on a user's electroencephalogram (EEG) signals. The method includes receiving raw EEG signals and providing the received raw EEG signals as input to a trained neural network and a trained multilayer perceptron (MLP) network; generating a cognitive score using the trained neural network, the neural network being trained using EEG signals associated with sustained attention and inattention cognitive states; generating a physiological score using the trained MLP network, the MLP network being trained using EEG signal features associated with the alpha and beta bands; receiving the cognitive score and the physiological score using a display controller module; and modifying the presentation of the digital content on the display based on the cognitive score and the physiological score using the display controller module. Dynamic manipulation of the digital content may enhance a user's cognitive abilities.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to Singapore Application No. 10202260087R, filed November 14, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] SYSTEMS AND METHODS FOR MODIFYING THE PRESENTATION OF DIGITAL CONTENT ON A DISPLAY BASED ON A USER'S ELECTROCELERATED ELECTROCELERATED (EEG) SIGNALS FIELD OF THE INVENTION The present application relates to systems and methods for modifying the presentation of digital content on a display based on a user's electroencephalogram (EEG) signals. Dynamic manipulation of digital content may enhance a user's cognitive abilities. [Background technology]

[0003] Human-computer interaction is typically achieved through peripheral devices such as a keyboard, mouse, and display. This type of interaction inevitably involves the user's nervous system. Instead of such peripheral devices, it has been proposed to use brain-computer interface (BCI) systems to establish direct information exchange between the human brain and computer systems. BCI systems allow users to bypass neuromuscular communication and instead allow users to control the system based on the user's brain activity.

[0004] The human brain produces biological signals, including electrical patterns known as brain waves, which can be detected and recorded via an electroencephalogram (EEG). These brain waves can be quantified using EEG devices, which typically capture them in analog form. The EEG data can then be analyzed in their original analog form or converted to a digital representation for further analysis. Most existing BCI systems employ techniques that utilize recorded EEG signals because it is an easier and relatively inexpensive brain signal acquisition technique compared to other non-invasive methods such as functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), and magnetoencephalography (MEG).

[0005] Through interaction with a BCI system, an individual can auto-regulate their brain signals in response to real-time stimuli. Thus, by training their brain, individuals may be able to rewire their underlying neural circuits through the process of neuroplasticity. Therefore, those skilled in the art are constantly searching for ways to augment or enhance a person's cognitive abilities by using their brain activity to control BCI systems. Summary of the Invention [Means for solving the problem]

[0006] In one aspect, the present application discloses a computer system configured to modify the presentation of digital content on a display based on acquired raw electroencephalogram (EEG) signals. The computer system includes a computer processor and a non-transitory medium readable by a processing unit, the non-transitory medium storing instructions that, when executed by the processing unit, cause the processing unit to: provide the raw EEG signals as inputs to a trained neural network and a trained multilayer perceptron (MLP) network. The processing unit is then instructed to: generate a cognitive score using the trained neural network, the neural network being trained using EEG signals associated with a sustained attention cognitive state and an inattention cognitive state; and generate a physiological score for each time window of the raw EEG signals using the trained MLP network, the MLP network being trained using EEG signal features associated with the alpha and beta bands. The processing unit is then further instructed to provide the cognitive score and the physiological score to a display controller module, and to use the display controller module to modify the presentation of the digital content on the display based on the cognitive score and the physiological score.

[0007] In another embodiment of this aspect, the present application discloses that the raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to the computer system, and the instructions to modify the presentation of the digital content on the display based on the cognitive score and the physiological score then further include instructions to direct the processing unit to: use the display controller module to display spatiotemporal activity on the display; and use the mobile biosensor device to instruct the user to continue engaging with the displayed spatiotemporal activity until it is determined that an average cognitive score over a cluster of time windows of the raw EEG signals has achieved a predetermined goal.

[0008] In yet another embodiment of this aspect, the present application discloses that the raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to the computer system, and the instructions to modify the presentation of the digital content on the display based on the cognitive score and the physiological score then further include instructions to direct the processing unit to: use the display controller module to display spatiotemporal activity on the display; and use the mobile biosensor device to instruct the user to continue engaging with the displayed spatiotemporal activity until it is determined that an average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score.

[0009] In another aspect, the present application discloses a method for modifying a presentation of digital content on a display based on acquired raw electroencephalogram (EEG) signals, the method including: receiving the raw EEG signals using a machine learning module and providing the received raw EEG signals as inputs to a trained neural network and a trained multilayer perceptron (MLP) network; generating a cognitive score using the trained neural network, the neural network being trained using EEG signals associated with a sustained attention cognitive state and an inattention cognitive state; and generating a physiological score for each time window of the raw EEG signals using the trained MLP network, the MLP network being trained using EEG signal features associated with the alpha and beta bands. The method then includes receiving the cognitive score and the physiological score using a display controller module; and modifying the presentation of the digital content on the display based on the cognitive score and the physiological score using the display controller module. [Brief explanation of the drawings]

[0010] Various embodiments of the present disclosure are described below with reference to the following drawings: [Figure 1] FIG. 1 is a block diagram of a computer system and a mobile device that may be used to implement an EEG-based cognitive enhancement system, according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a processing system for implementing embodiments of the present disclosure. [Figure 3a] FIG. 1 illustrates a set of stimuli for generating sustained attention EEG signals for training a neural network, according to an embodiment of the present disclosure. [Figure 3b]FIG. 1 illustrates a timing protocol for calibration of a set of training data sets used to train a neural network, according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a process for enhancing a user's cognitive performance based on the user's EEG signals, according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram illustrating four main parts of a multi-scale convolutional neural network, according to an embodiment of the present disclosure. [Figure 6] FIG. 10 illustrates a position map of a cap having 32 channels, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 shows histogram plots illustrating the classification performance of trained neural networks. [Figure 8] FIG. 1 illustrates a multi-layer perceptron neural network structure according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart of a process for modifying the presentation of digital content on a display based on raw EEG signals, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following detailed description is made with reference to the accompanying drawings, which illustrate, by way of example, details and embodiments of the present disclosure. Features described in the context of an embodiment may be applicable to the same or similar features of other embodiments, even if not explicitly described therein. Additional and / or combinations and / or alternatives to features described in the context of an embodiment may be applicable to the same or similar features of other embodiments.

[0012] In the context of various embodiments, the articles "a," "an," and "the" as used in reference to features or elements include a reference to one or more of the feature or element.

[0013] In the context of various embodiments, the term "about" or "approximately" as applied to a numerical value encompasses the exact value and a reasonable variance as commonly understood in the relevant art, e.g., within 10% of the specified value.

[0014] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0015] As used herein, "comprising" means including, but not limited to, what follows the word "comprising." Thus, use of the word "comprising" indicates that the listed elements are required or mandatory, but other elements are optional and may or may not be present.

[0016] As used herein, "consisting of" means including, and limited to, what follows the phrase "consisting of." Thus, use of the phrase "consisting of" indicates that the listed elements are required or mandatory, and that no other elements may be present.

[0017] As used herein, the term "EEG signal" or "EEG data" generally refers to electroencephalography (EEG) obtained from one or more subjects, where these signals correspond to the subjects' brain activity.

[0018] Additionally, those skilled in the art will recognize that throughout this specification, certain functional units in this description are labeled as modules. Those skilled in the art will also recognize that modules may be implemented as circuits, logic chips, or any type of discrete component. Furthermore, those skilled in the art will also recognize that modules may be implemented in software, which may then be executed by various processor architectures. In embodiments of the present disclosure, modules may also include computer instructions or executable code that may instruct a computer processor to perform a sequence of events based on received instructions. The choice of module implementation is left to those skilled in the art as a design choice and does not in any way limit the scope of the claimed subject matter.

[0019] Furthermore, those skilled in the art will also recognize that the detailed operation, internal structure, techniques, and training datasets of the machine learning models, neural networks, and anomaly detection models referenced in this disclosure have not been disclosed or shown in detail as such information is known to those skilled in the art and, therefore, has been omitted for the sake of brevity.

[0020] Electroencephalography (EEG) is a commonly used method for directly measuring human brain activity. It involves placing multiple electrodes on the surface of the human head to collect EEG signals. Due to its high temporal resolution, EEG can capture dynamic changes in brain activity at the subsecond level. Leveraging machine learning and signal processing techniques, brain-computer interface (BCI) systems have the ability to identify human emotions using EEG data.

[0021] Existing BCI systems for cognitive enhancement typically use repetitive, non-generalized, and exhausting training environments. As a subject's mental abilities decline due to fatigue or low energy levels, this degrades the quality of the EEG signal, thereby reducing the effectiveness of the cognitive enhancement system. Furthermore, because EEG signals vary between subjects due to individual differences, results obtained from signal processing and traditional machine learning techniques (used to extract temporal, spectral, or spatial features from EEG signals) degrade when these techniques are applied to different subjects.

[0022] 1 shows a block diagram of a computer system and a mobile device that can be used to implement an EEG-based cognitive enhancement system according to an embodiment of the present disclosure. The computer system 110 continuously monitors and receives EEG signals and human-computer interaction (HCI) signals 108 from a mobile device 102 that is provided to a user or subject of the computer system 110. The mobile device 102, including a biosensor device 104 and a human-computer interaction (HCI) device 106, is configured to acquire the EEG and HCI signals and subsequently communicate these signals to the computer system 110. The communication of the signals 108 by the mobile device 102 to the computer system 110 may be via wired or wireless data transmission means.

[0023] In particular, biosensor device 104 is typically a non-invasive device designed to capture and measure the electrical activity of a user's brain. In embodiments of the present disclosure, biosensor device 104 is often embedded in a headpiece, such as a headset, and may include multiple electrodes / channels strategically positioned on the user's scalp to pick up electrical signals generated by neuronal activity. These electrodes are typically attached to a processing system that processes the electrical signals into EEG signals, which may then be transmitted to computer system 110. In further embodiments of the present disclosure, dry electrodes may be used to record EEG signals from a subject, with electrodes positioned near the subject's frontal lobe (to collect EEG signals from the frontal region) because this is the region that contains attention-related information. Biosensor device 104 may then be connected to computer system 110 via Bluetooth, allowing data to be transmitted wirelessly between these two components.

[0024] HCI device 106 comprises a wide range of tools, interfaces, and systems that enable a user to interact with a computing system / device. HCI device 106 may include, but is not limited to, a graphical user interface (GUI), a touchscreen, a keyboard or touchpad, a voice recognition system, a gesture control system, an augmented and virtual reality interface, an eye-tracking system, and any other device that facilitates data entry, information retrieval, and communication between a user and a computing system, and allows items in control system 110 to be controlled by the user. Signals acquired by HCI device 106 may then be provided to computer system 110. In a further embodiment of the present disclosure, a subject's gaze direction may be estimated using an eye-tracking device mounted on display 130 such that the eye-tracking device can capture the subject's gaze and communicate this information to computer system 110. The eye-tracking device may include a projector for creating a near-infrared light pattern at the subject's eye, a camera for recording the pattern projected at the eye and the subject's gaze, and an algorithm for generating coordinates of the subject's gaze to determine the direction of the subject's eye orientation. All this information may then be provided to the display controller module 120 to control the presentation of the information on the display unit 130 .

[0025] 1 , computer system 110 includes machine learning module 111, display controller module 120, and display unit 130, where machine learning module 111 includes a neural network module including neural network 112 and a multi-layer perceptron (MLP) network module including MLP network 114. In embodiments of the present disclosure, display controller module 120 may be configured to modify and / or control digital content shown on display unit 130 based on data received from machine learning module 111 and / or mobile device 102. In embodiments of the present disclosure, display controller module 120 may modify the presentation of digital content shown on display unit 130 by directly utilizing HCI signals obtained from mobile device 102.

[0026] The neural network 112 may include, but is not limited to, a multi-scale convolutional neural network with various convolutional layers and branches that simultaneously extract features and information from different scales of input data, and the MLP network 114 may include, but is not limited to, a type of neural network with an input layer, one or more hidden layers, and an output layer in which artificial neurons are all interconnected, where each neuron in the network is linked to all neurons in adjacent layers using weighted connections. The neural network 112 may be trained using EEG signals associated with physiological states of sustained attention and inattention, and the MLP network 114 may be trained using EEG signal features associated with the alpha and beta bands.

[0027] In an embodiment of the present disclosure, the raw EEG signals may be preprocessed by the machine learning module 111 before they are processed by the trained neural network 112 and the trained MLP network 114. During preprocessing of the raw EEG signals, a bandpass filter (0.3-40 Hz) is applied to remove low and high frequency noise from the raw EEG signals. Artifacts are then removed from the raw EEG signals by subtracting a smoothed baseline obtained using a moving averaging technique with multiple moving windows from the EEG signals.

[0028] TIFF2025539717000002.tif73170

[0029] Once trained, the trained neural network 112 and the trained MLP network 114 may then generate cognitive and physiological scores, respectively, based on the EEG signals obtained from the mobile device 102. Note that the computation of the cognitive score 116 and the physiological score 118 by each network occurs simultaneously. The cognitive score 116 and the physiological score 118 are then provided to the display controller module 120, which is further configured to modify the data presented on the display unit 130 based on the received cognitive score 116 and physiological score 118.

[0030] 2 shows a block diagram illustrating components of a processing system 200 that may be provided within computer system 110, mobile device 102, or any of the modules provided within system 110 and / or mobile device 102 to perform any of the functions described above, according to an embodiment of the present disclosure. Those skilled in the art will recognize that the exact configuration of each processing system provided within these modules may differ, the exact configuration of processing system 200 may vary, and the arrangement shown in FIG. 2 is provided by way of example only.

[0031] In an embodiment of the present disclosure, processing system 200 may include controller 201 and user interface 202. User interface 202 is configured to allow manual interaction between a user and the computing modules, if desired, and to this end includes the input / output components necessary for a user to input instructions for providing updates to each of these modules. Those skilled in the art will recognize that the components of user interface 202 may vary depending on the embodiment, but typically include one or more of a display 240, a keyboard 235, and an optical device 236.

[0032] The controller 201 is in data communication with the user interface 202 via a bus 215 and includes a memory 220, a processor 205 mounted on a circuit board for processing instructions and data to implement the methods of the present embodiment, an operating system 206, an input / output (I / O) interface 230 for communicating with the user interface 202, and a communications interface in the form of a network card 250 in this embodiment. The network card 250 may be used, for example, to transmit data from these modules to other processing devices over a wired or wireless network, or to receive data over a wired or wireless network. Wireless networks that may be utilized by the network card 250 include, but are not limited to, Wireless Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunications networks, wide area networks (WANs), etc.

[0033] Memory 220 and operating system 206 are in data communication with CPU 205 via bus 210. The memory components include both volatile and nonvolatile memory, and two or more of each type of memory, including random access memory (RAM) 223, read-only memory (ROM) 225, and mass storage device 245, the latter of which may include one or more solid-state drives (SSDs). Those skilled in the art will recognize that the memory components described above include non-transitory computer-readable media and should be interpreted to include all computer-readable media except for transitory propagating signals. Typically, instructions are stored in the memory components as program code, but may also be hardwired. Memory 220 may include kernels and / or programming modules, such as software applications, which may be stored in either volatile or non-volatile memory.

[0034] As used herein, the term "processor" is used generically to refer to any device or component capable of processing such instructions, which may include a microprocessor, microcontroller, programmable logic device, or other computing device. That is, processor 205 may be provided by any suitable logic circuitry for receiving inputs, processing them according to instructions stored in memory, and generating output (e.g., to a memory component or on display 240). In this embodiment, processor 205 may be a single-core or multi-core processor having a memory-addressable space. In one example, processor 205 may be multi-core, including, for example, an eight-core CPU. In another example, the processor may be a cluster of CPU cores operating in parallel to accelerate computations.

[0035] In one embodiment of the present disclosure, a brain-computer interface (BCI) system comprising a computer system 110 and a mobile device 102 as previously shown in FIG. 1 may be used to enhance a subject's cognitive abilities through an attention training process based on real-time feedback from the subject. In the proposed training system, a highly immersive attention training environment, where the environment is based on a galaxy theme, is displayed on a display unit 130. The spatiotemporal environment shown on the display 130 includes an engaging multi-dimensional, stereoscopic training environment to foster cognitive agility, spatial perception, and / or engagement.

[0036] In the displayed environment, the user may control the navigation of a virtual object, i.e., a spaceship, using HCI signals from the user, such as the user's gaze direction and the user's attention level (i.e., cognitive score), where the trained neural network 112 is used to calculate the user's attention level based on the user's EEG signals. Note that the training system in this embodiment explicitly implements relaxation routines and / or programs incorporated into the attention training process, thus ensuring that the user is less fatigued and more motivated, thereby allowing the user to better engage with the content on the display for longer periods of time. The engaging multidimensional galaxy environment shown on the display 130 also offers multiple levels of difficulty and multiplayer settings to accommodate a wide range of training scenarios. Furthermore, the training system includes specific objectives aimed at enhancing sustained focus, selective attention, and the ability to resist distractions, cultivating a more engaging training experience, where training intensity is easily adjustable based on task performance and duration, providing a flexible and personalized approach. The autonomous models used in the system are also subject-independent, eliminating the need for individual calibration training.

[0037] In general, the proposed training system can be divided into two parts. In the initial stage, a neural network 112 containing a comprehensive brain-computer interface (BCI) classification model is trained. During the training process, subjects are exposed to stimuli specifically designed to elicit either heightened attention or relaxation, while their EEG data is collected. To establish a subject-independent model that can be applied to any subject or user, two separate cognitive tasks related to sustained attention and inattention are used. To elicit attention, a flanker test using emoji-based stimuli is utilized, as shown in Figure 3a. The first two rows 302 represent a congruent stimulus set (all emojis are sad or happy), while the other two rows 304 represent an incongruent stimulus set (the central emoji is different from the surrounding emojis). The subject may respond to the stimuli using the left and right arrows on the keyboard, depending on whether the stimulus is congruent or incongruent, respectively, requiring focused attention. The subjects' EEG signals are collected simultaneously and used to generate a training dataset of EEG signals associated with sustained attention cognitive states. For the inattention task, subjects are presented with a blank black screen and instructed to slowly gaze around a boundary to avoid fixing their attention on a specific point. The subjects' EEG signals are collected simultaneously and used to generate a training dataset of EEG signals associated with inattention cognitive states.

[0038] In an embodiment of the present disclosure, the calibration data includes EEG data from multiple sessions, e.g., 10 sessions consisting of six blocks each, each lasting approximately three minutes and encompassing both training data sets (of EEG signals associated with sustained attention and inattention cognitive states) to generate multiple data recording trials. This is shown in FIG. 3b. The neural network 112 is then trained using the recorded EEG signals from the two separate training data sets so that the trained neural network 112 can be used to detect a user's attention level. In an initial stage, the MLP network 114 can be trained using EEG signal features associated with the alpha and beta bands. The trained MLP network 114 can then be used to detect a subject's fatigue level based on the acquired EEG signals.

[0039] In the final stage of the proposed training system, the subject is presented with an immersive virtual display including a visual pathway embedded with spacecraft controls within a virtual solar system environment and is asked to control the spacecraft to reach a specified objective (visit selected planets in a predetermined order). Figure 4 illustrates the key processes implemented by the system in the final stage of the proposed training system. In this embodiment, the speed and direction of the spacecraft's movement can be controlled by the cognitive score or the subject's attention level decoded from the subject's EEG signal, and by using the subject's eye-tracking device output. In an embodiment of the present disclosure, the duration of one session can be predetermined, e.g., 20 minutes, and a complete training cycle can consist of multiple sessions. The system begins in step 402 by acquiring EEG and HCI signals from the subject.

[0040] Then, in step 404, the subject's cognitive score or the subject's attention level may be predicted by the trained neural network 112 based on the captured EEG signals. In an embodiment of the present disclosure, the cognitive score may be calculated at a predetermined period, e.g., every 200 milliseconds, for each EEG time segment, e.g., 4 seconds. The score generated by the trained neural network 112 varies between "0" and "1," with a score of "0" representing a highly unfocused cognitive state and a score of "1" representing a highly focused cognitive state. In another embodiment of the present disclosure, the score generated by the trained neural network 112 is a probability score S that represents the probability of a cognitive attention state. attention where S atttention = f(seg, φ), where seg denotes the processed EEG segment and φ are the universal model parameters learned from the attention training process.

[0041] Simultaneously, in step 404, the subject's physiological score or fatigue level is continuously assessed using the trained MLP network 114. In particular, the trained MLP network 114 is configured to predict the subject's physiological score or fatigue level based on the captured EEG signals. In a virtual display, the calculated physiological score or fatigue level is monitored and mapped to the spacecraft's fuel level.

[0042] Once the subject's physiological and cognitive scores have been calculated, the system then determines whether the subject is fatigued based on the calculated physiological scores in step 406. If the subject is determined to be fatigued, the subject undergoes a series of relaxation exercises in step 410. Conversely, if the subject is determined not to be fatigued, the subject is then guided through an attention training process in step 408.

[0043] The stimuli in the attention training process are designed to induce attention in an immersive environment and typically include a high-dimensional spatial environment in which the subject interacts with the environment via the subject's cognitive scores (calculated based on the subject's EEG signals), eye-tracking device signals, and / or computer keyboard input (HCI signals). Four core attention skills can be trained during the attention training process: the first attention skill involves the subject's ability to remain focused on an important goal without distractions (sustained attention); the second attention skill involves the subject's ability to select and focus specific input for further processing while simultaneously suppressing irrelevant or distracting information (selective attention); the third attention skill involves the subject's ability to pay attention to two things at once (divided attention); and the fourth attention skill involves the subject's ability to block potentially distracting information from their focus of attention or control their response when conflicts arise (executive function). In an embodiment of the present disclosure, the objective of the immersive environment can be to visit multiple planets consecutively to distribute predetermined targets as quickly as possible while avoiding all obstacles and replenishing fuel as needed. The subject's performance can then be scored based on the time it takes to achieve the goal, the strategies employed to accept / avoid the object, and the subject's efficiency in obtaining the reward and avoiding the obstacle. Additionally, the subject must navigate the spacecraft to a fuel station to refuel, which further initiates the subject's relaxation process.

[0044] In an embodiment of the present disclosure, attention level (speed) and planet location details (such as altitude and relative distance from other targets) may be displayed to help the subject gain a deeper perspective of the training environment. The subject may then select a training difficulty level, such as basic, hybrid, or intelligent. In this embodiment, the basic level refers to a training sequence that uses eye-tracking and keyboard strokes to reach a few target locations without any interruptions along the way; the hybrid level refers to a training sequence that requires the subject to use EEG signals, a keyboard, and a tracking device, with the goal of reaching N subsequent planets with intermediate levels of hurdles in between; and the intelligent level refers to a training sequence that integrates input from EEG signals and eye-tracking, incorporates reward points and penalties, and with the goal of reaching more than N target locations while avoiding obstacles and enemies.

[0045] In the exemplary training sequence, N is defined as 4 and may be as high as 9 at higher difficulty levels. Once the number of planets is selected based on the difficulty level, the order in which successive goals are completed can then be determined by the subject. The training environment is designed in a training framework such that all planets are attached to entry checkpoints. The subject must then pass through the checkpoints to touch the associated planets. Initially, before launching the spacecraft, the subject is presented with the time remaining at each checkpoint closure and must use their gaze input to select the order of interplanetary travel. The four goal locations may be selected based on the subject's attention level estimated by the trained neural network 112, and the subject's gaze and eye movements may be extracted from an eye-tracking device. As described above, the digital display includes a dynamically changing spatiotemporal display, where the complexity of the spatiotemporal display can be said to dynamically change based on EEG signals to maintain a particular attention level or cognitive score or to prompt the subject to achieve a predetermined target attention level or cognitive score.

[0046] Further, during the attention training process, it should be noted that the trained MLP network 114 continuously predicts the subject's physiological score or fatigue level based on the captured EEG signals. In an embodiment of the present disclosure, the trained MLP network 114 generates physiological score values ​​as indicators of the subject's fatigue level, and these values ​​may range from "0" to "1," where a value of "0" indicates the subject is not fatigued and a score of "1" indicates the subject is very fatigued. These scores are calculated every 200 milliseconds for EEG signals captured over a 4-second time window. Those skilled in the art will recognize that these periods may be varied and are left as a design choice to those skilled in the art. Based on the subject's fatigue level, the system determines whether to trigger a relaxation process.

[0047] If the relaxation process is triggered, the system then proceeds to step 410. In this step, the subject undergoes a series of relaxation exercises to restore the subject's mental endurance and strength. In embodiments of the present disclosure, the subject may be guided through several tasks, such as conscious relaxation with audiovisual-based deep breathing exercises, and may be exposed to calming music and beautiful / serene images, all aimed at refreshing the subject's cognitive skills and beneficial strategies, so that once the subject undergoes the attention training process again, the subject can then navigate through the process more quickly. The length of the relaxation exercises may be fixed, for example, three minutes, or may be determined by the system. Furthermore, the subject may select a specific exercise or combination of exercises based on their own requirements.

[0048] After the relaxation exercises are completed, the system repeats steps 402, 404, and 406 to determine whether the subject is fatigued. If it is determined that the subject is not fatigued, the subject then undergoes the attention training process in step 408, as discussed above. Otherwise, if it is determined that the subject is still fatigued in step 406, the subject then undergoes another set of relaxation exercises in step 410. In an embodiment of the present disclosure, regardless of whether the subject is determined to be "fatigued," after multiple consecutive repetitions of step 410, for example, two consecutive repetitions, the subject must undergo the attention training process in step 408.

[0049] Steps 402 through 410 are continuously repeated until a preset objective, such as navigating a fixed number of planets while overcoming several obstacles, is achieved, after which the process shown in FIG. 4 ends at step 412. In the proposed system, goal selection is dynamically set based on the performance metrics achieved by the subject over the ongoing session. For example, if the subject is taking significantly longer than the allotted time to complete a particular objective (each objective is associated with a predetermined time interval, e.g., three minutes for basic level training to reach two goal locations), the system recommends that the subject play the same level again. Also, if the subject completes a particular level objective within a time frame significantly shorter than the predetermined time, the subject is given the option to skip the very next level. This type of dynamic goal allocation based on the subject's performance metrics is ideal because it optimizes the subject's training and avoids unnecessary training. Factors such as the sustainability of attention scores, fluctuations in fatigue scores, and the frequency of requests to activate relaxation mode can also control goal selection to make cognitive training more effective. [Neural Networks]

[0050] In an embodiment of the present disclosure, the neural network 114 may include a multi-scale convolutional neural network having a dynamic temporal layer configured to learn dynamic and frequency representations of all channels of the EEG signal, an asymmetric spatial layer configured to calculate global and hemispherical spatial kernels, and a higher-level fusion layer configured to fuse information from the global and hemispherical spatial kernels to form a fully connected layer driven by a softmax function. The detailed operation of this multi-scale convolutional neural network is described in this section.

[0051] EEG data may be viewed as a 2D time series, with one dimension representing EEG channels (electrodes) and the other representing time, reflecting changes in brain activity over time. The spatial dimension can indicate patterns of brain activation in different functional regions (based on electrode placement). As shown in Figure 5, the neural network utilizes modules including a dynamic temporal layer 502, an asymmetric spatial layer 504, a high-level fusion layer 506 for identifying distinctive time-frequency-channel-specific EEG features associated with the user's emotional state, and a classifier layer 508.

[0052] The dynamic temporal layer 502 does so by learning dynamic time / frequency representations from the EEG data for each channel using multi-scale 1D convolution kernels to capture more informative time-frequency representations. The asymmetric spatial layer 504 then introduces hemispheric kernels to model these asymmetries, leveraging neuroscience findings showing that brain activity in the left and right hemispheres is asymmetrically related to emotion or focused attention. The asymmetric spatial layer 504 obtains learned representations for each channel and applies them to learn a global spatial representation and asymmetric patterns of emotion or attention using different scale convolution kernels. The high-level fusion layer 506 then combines the learned representations from the hemispheres with the global kernel or representation obtained from the asymmetric spatial layer. The fused representations are then passed to a classifier layer 508, which has a fully connected layer with softmax as its activation function. dynamic time layer

[0053] TIFF2025539717000003.tif83170

[0054] TIFF2025539717000004.tif71170

[0055] TIFF2025539717000005.tif78170

[0056] TIFF2025539717000006.tif53170

[0057] The asymmetric spatial layer has multi-scale 1D convolution kernels whose size is related to the location of the EEG channels. There are two types of spatial kernels: global kernels and hemispherical kernels.

[0058] TIFF2025539717000007.tif80170

[0059] TIFF2025539717000008.tif83170

[0060] TIFF2025539717000009.tif76170

[0061] TIFF2025539717000010.tif58170

[0062] TIFF2025539717000011.tif56170

[0063] TIFF2025539717000012.tif159170

[0064] In Table 1, LK-ReLU stands for Leaky-ReLU activation function, AP stands for average pooling operation, BN stands for batch normalization, GAP stands for global average pooling, and "-1" in tensor size means the number of samples in one mini-batch. Furthermore, unless otherwise noted, the stride of the CNN is (1,1), and that for the pooling layer is the same as the pooling step.

[0065] As described in the previous section, when a neural network was trained using EEG signals associated with sustained attention and inattention cognitive states, it was determined that the trained neural network was capable of distinguishing between a subject's attentional and inattentional mental states, resulting in an average classification accuracy of 80% across 50 healthy subjects, as shown in the histogram plotted in Figure 7. [Multilayer Perceptron (MLP) network]

[0066] 8 illustrates a multi-layer perceptron neural network structure according to an embodiment of the present disclosure. MLP networks represent a generalization of single-layer perceptrons, where single-layer perceptrons form half-plane decision regions, while multi-layer perceptrons are capable of forming arbitrarily complex decision regions and can separate a variety of input patterns.

[0067] Generally, an MLP network comprises an input layer, a hidden layer, an output layer, and a bias layer. The input layer contains a layer of neurons that receives information from an external source and passes this information to the network for processing. The hidden layer contains a layer of neurons that receives information from the input layer and processes it in a hidden manner, where the hidden layer does not directly connect to other layers in the system. The output layer contains a layer of neurons that receives the processed information and sends an output signal from the system. The bias layer acts on the neurons like an offset, where the function of the bias is to provide a threshold for neuron activation, and bias inputs are connected to each hidden neuron and output neuron in the network. The connection weights in the MLP network are then learned from available training patterns, and its performance is gradually improved over time by iteratively updating the weights in the network. In an embodiment of the present disclosure, the MLP network was trained using EEG signal features related to the alpha and beta bands. The training dataset used to train the MLP network may be obtained from Z. Cao, C.-H. Chuang, J.-K. King, and C.-T. Lin, "Multi-channel EEG recordings during a sustained-attention driving task," Sci. Data, vol. 6, no. 1, pp. 1-8, April 2019.

[0068] A process for modifying the presentation of digital content on a display based on raw EEG signals according to an embodiment of the present disclosure is shown in FIG. 9, where process 900 may be performed by modules included within computer system 110.

[0069] Process 900 begins in step 902 with the system receiving EEG and HCI signals. Process 900 then utilizes a trained neural network and a trained MLP network to generate cognitive and physiological scores, respectively. The generated scores are then provided to a display controller module in step 906. Process 900 then utilizes the cognitive and / or physiological scores to modify the presentation of digital content on a display. Process 900 then ends.

[0070] In an embodiment of the present disclosure, during modification of the presentation of digital content on the display based on the cognitive and physiological scores, the process 900 may use a display controller module to display first digital content on the display in response to determining that the average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score. In a further embodiment, the first digital content may include a relaxation program.

[0071] In an embodiment of the present disclosure, during modification of the presentation of the digital content on the display based on the cognitive and physiological scores, the process 900 may use the display controller module to display second digital content on the display in response to determining that the first period has elapsed. In a further embodiment, the second digital content includes an attention training program.

[0072] In an embodiment of the present disclosure, before generating a cognitive score using the trained neural network, process 900 may filter the raw EEG signals provided to the trained neural network to remove low-frequency noise and high-frequency noise, and may remove artifacts from the filtered EEG signals by subtracting a smoothed reference from the filtered EEG signals, where the smoothed reference is obtained using a moving averaging technique.

[0073] In an embodiment of the present disclosure, raw EEG signals may be acquired from a user using a mobile biosensor device communicatively connected to a computer system.

[0074] In an embodiment of the present disclosure, during modification of the presentation of digital content on the display based on the cognitive and physiological scores, the process 900 may use a display controller module to display a dynamically changing spatiotemporal display on the display, where the complexity of the spatiotemporal display is dynamically changed to maintain the cognitive score at a constant level.

[0075] In an embodiment of the present disclosure, raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to a computer system, and during modification of the presentation of digital content on the display based on the cognitive and physiological scores, process 900 may use a display controller module to display spatiotemporal activity on the display and use the mobile biosensor device to instruct the user to continue engaging with the displayed spatiotemporal activity until it is determined that the average cognitive score over a cluster of time windows of the raw EEG signals has achieved a predetermined goal.

[0076] In an embodiment of the present disclosure, raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to a computer system, and during modification of the presentation of digital content on the display based on the cognitive and physiological scores, process 900 may use a display controller module to display spatiotemporal activity on the display and use the mobile biosensor device to instruct the user to continue engaging with the displayed spatiotemporal activity until it is determined that the average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score.

[0077] In an embodiment of the present disclosure, process 900 may use a mobile biosensor device to obtain human-computer interaction (HCI) signals from a user, provide the HCI signals to a display controller module, and use the display controller module to modify the presentation of digital content on a display based on the cognitive and physiological scores and the HCI signals. [Example of an attention training system]

[0078] An exemplary cognitive training environment / program / game is disclosed in this section. In this example, the program includes a third-person camera view 3D scene. The speed of the ship in this program is calculated using an acceleration curve, where the higher the attention score, the higher the curvilinear acceleration curve of the spacecraft. Real-time 4-channel EEG signal segments are converted to corresponding attention scores predicted by a trained neural network, with scores ranging from 0 to 1, representing very unfocused to very focused states, respectively.

[0079] The heading of the ship depends on the point of regard, which is the coordinate point the user is looking at. If the spacecraft must turn left, right, up, or down, the user's line of sight must shift left, right, up, and down respectively from the center point.

[0080] The ship also features a fuel bar that indicates the fuel level, which changes accordingly. As the ship progresses, its fuel decreases. To provide the user with additional fuel, the ship must navigate through a green ring-shaped stone structure. When the user enters this ring, the active scene pauses and a cutscene of instructions is displayed, governed by the entire relaxation portion of the training. The platform is designed to allow for relaxation instructions to help reduce the user's mental fatigue. During the relaxation exercise, the subject performs deep breathing exercises, which activates the subject's brain function for further program experiences.

[0081] The training theme presents distractions such as asteroid fields, rocks, and fog that exist in the visual space of the player's spacecraft. If the player's spacecraft accidentally collides with an asteroid or immobile space object, the spacecraft will be destroyed; therefore, great care must be taken to avoid them as efficiently as possible. The training program is designed so that the player must use their sustained, selective, and divided attention to dismiss the distractions as quickly as possible.

[0082] For example, if a rock appears on the path of the ship, the player must improve their focused attention so that the ship moves faster and escapes line of sight. If two asteroids are moving toward the ship at different speeds, the player must decide which asteroid to destroy first based on its speed. In other words, the fastest-moving asteroid should be eliminated first, followed by the other. The action sequence and amount of time taken by the player are evaluated by the program to track the player's level of divided attention. If the program determines that the player's sustained attention level is significantly above a certain threshold α, the player's spaceship gains better shields that can directly destroy any asteroids / rocks, thereby improving the spaceship's power play.

[0083] During gameplay, a counter is used to display the amount of time for the entire mission, e.g., the counter displays the minutes and seconds of game time. However, this counter is deactivated when the player activates relaxation mode, and the counter restarts when the player returns to the training program. Data for each mission is obtained from the counter and stored in a separate output file. During goal-directed navigation, basic skills related to executive function include adaptive thinking, planning, self-monitoring, self-control, and time management proficiency, and organization is also indirectly trained. 2D maps for planning interplanetary missions and checking progress, and high-dimensional perspective views for training, may also be made available for target information.

[0084] Many other changes, substitutions, variations, and modifications may be ascertained by those skilled in the art, and this application is intended to include all such changes, substitutions, variations, and modifications that fall within the scope of the appended claims.

Claims

1. 1. A computer system configured to modify a presentation of digital content on a display based on acquired raw electroencephalogram (EEG) signals, comprising: A computer processor; A non-transitory medium readable by a processing unit, which, when executed by the processing unit, causes the processing unit to: providing the raw EEG signals as inputs to a trained neural network and a trained multi-layer perceptron (MLP) network; generating a cognitive score using the trained neural network, wherein the neural network is trained using EEG signals associated with a sustained attention cognitive state and an inattention cognitive state; generating a physiological score for each time window of the raw EEG signal using the trained MLP network, wherein the MLP network is trained using EEG signal features associated with the alpha and beta bands; providing the cognitive score and the physiological score to a display controller module; and using the display controller module to modify the presentation of the digital content on the display based on the cognitive score and the physiological score; a non-transitory medium storing instructions to cause A computer system comprising:

2. The instructions for modifying the presentation of the digital content on the display based on the cognitive score and the physiological score may include: displaying, using the display controller module, first digital content on the display in response to determining that an average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score; 10. The computer system of claim 1, further comprising instructions to:

3. The computer system of claim 2 , wherein the first digital content includes a relaxation program.

4. The instructions for modifying the presentation of the digital content on the display based on the cognitive score and the physiological score may include: displaying, using the display controller module, second digital content on the display in response to determining that a first period of time has elapsed; The computer system of any one of claims 1 to 3, further comprising instructions to:

5. The computer system of claim 4 , wherein the second digital content comprises an attention training program.

6. Prior to the instructions to generate a cognitive score using the trained neural network, the non-transitory medium may instruct the processing unit to: filtering the raw EEG signals provided to the trained neural network to remove low-frequency and high-frequency noise; and removing artifacts from the filtered EEG signal by subtracting a smoothed reference from the filtered EEG signal, the smoothed reference being obtained using a moving averaging technique; 10. The computer system of claim 1, further comprising instructions to:

7. The computer system of claim 1 , wherein the raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to the computer system.

8. The instructions to modify the presentation of the digital content on the display based on the cognitive score and the physiological score may include instructions to the processing unit to: using the display controller module to display a dynamically changing spatiotemporal display on the display, wherein the complexity of the spatiotemporal display is dynamically changed to maintain the cognitive score at a constant level; The computer system of any one of claims 1 to 7, further comprising instructions to:

9. The raw EEG signals are obtained from the user using a mobile biosensor device communicatively connected to the computer system, and the instructions to modify the presentation of the digital content on the display based on the cognitive score and the physiological score include: displaying spatiotemporal activity on the display using the display controller module; and instructing the user, using the mobile biosensor device, to continue engaging with the displayed spatiotemporal activity until it is determined that an average cognitive score over a cluster of time windows of the raw EEG signals has achieved a predetermined goal; 10. The computer system of claim 1, further comprising instructions to:

10. the raw EEG signals are acquired from a user using a mobile biosensor device communicatively connected to the computer system; The instructions to modify the presentation of the digital content on the display based on the cognitive score and the physiological score may include instructions to the processing unit to: displaying spatiotemporal activity on the display using the display controller module; and instructing the user, using the mobile biosensor device, to continuously engage with the displayed spatiotemporal activity until it is determined that an average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score; 10. The computer system of claim 1, further comprising instructions to:

11. The non-transitory medium may include a processor acquiring a human-computer interaction (HCI) signal from the user using the mobile biosensor device; providing the HCI signal to the display controller module; and using the display controller module to modify the presentation of the digital content on the display based on the cognitive scores, the physiological scores, and the HCI signal; 8. The computer system of claim 7, further comprising instructions to:

12. 2. The computer system of claim 1, wherein the neural network comprises a multi-scale convolutional neural network including a dynamic temporal layer configured to learn dynamic and frequency representations of all channels of an EEG signal, an asymmetric spatial layer configured to compute global and hemispherical spatial kernels, and a higher-level fusion layer configured to fuse information from the global and hemispherical spatial kernels to form a fully connected layer driven by a softmax function.

13. 1. A method for modifying presentation of digital content on a display based on acquired raw electroencephalogram (EEG) signals, comprising: using a machine learning module to receive the raw EEG signals and provide the received raw EEG signals as inputs to a trained neural network and a trained multi-layer perceptron (MLP) network; generating a cognitive score using the trained neural network, wherein the neural network is trained using EEG signals associated with a sustained attention cognitive state and an inattention cognitive state; generating a physiological score for each time window of the raw EEG signal using the trained MLP network, wherein the MLP network is trained using EEG signal features associated with the alpha and beta bands; receiving the cognitive scores and the physiological scores using a display controller module; and using the display controller module to modify the presentation of the digital content on the display based on the cognitive score and the physiological score; A method comprising:

14. Modifying the presentation of the digital content on the display based on the cognitive score and the physiological score includes: using the display controller module to display first digital content on the display in response to determining that an average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score; 14. The method of claim 13, further comprising:

15. The method of claim 14 , wherein the first digital content comprises a relaxation program.

16. Modifying the presentation of the digital content on the display based on the cognitive score and the physiological score includes: displaying, using the display controller module, second digital content on the display in response to determining that a first period of time has elapsed. The method of any one of claims 13 to 15, further comprising:

17. The method of claim 16 , wherein the second digital content comprises an attention training program.

18. Prior to the step of generating the cognitive score using the trained neural network, the method further comprises: using the machine learning module to filter the raw EEG signals provided to the trained neural network to remove low frequency and high frequency noise; and removing artifacts from the filtered EEG signal by subtracting a smoothed reference from the filtered EEG signal, the smoothed reference being obtained using a moving averaging technique; 14. The method of claim 13, further comprising:

19. 14. The method of claim 13, wherein the raw EEG signals are obtained from a user using a mobile biosensor device communicatively connected to a computer system.

20. Modifying the presentation of the digital content on the display based on the cognitive score and the physiological score includes: using the display controller module to display a dynamically changing spatiotemporal display on the display, wherein the complexity of the spatiotemporal display changes dynamically to maintain the cognitive score at a constant level; The method of any one of claims 13 to 19, further comprising:

21. the raw EEG signals are obtained from the user using a mobile biosensor device communicatively connected to a computer system, and modifying the presentation of the digital content on the display based on the cognitive score and the physiological score comprises: displaying spatiotemporal activity on the display using the display controller module; and instructing the user, using the mobile biosensor device, to continue engaging with the displayed spatiotemporal activity until it is determined that an average cognitive score over a cluster of time windows of the raw EEG signals achieves a predetermined goal; 14. The method of claim 13, further comprising:

22. the raw EEG signals are acquired from a user using a mobile biosensor device communicatively connected to the computer system; Modifying the presentation of the digital content on the display based on the cognitive score and the physiological score includes: displaying spatiotemporal activity on the display using the display controller module; and instructing the user, using the mobile biosensor device, to continuously engage with the displayed spatiotemporal activity until it is determined that an average physiological score over a cluster of time windows of the raw EEG signals exceeds a predetermined threshold score; 14. The method of claim 13, further comprising:

23. The method comprises: acquiring a human-computer interaction (HCI) signal from the user using the mobile biosensor device; providing the HCI signal to the display controller module; and modifying the presentation of the digital content on the display based on the cognitive scores, the physiological scores, and the HCI signal using the display controller module; 20. The method of claim 19, further comprising:

24. 14. The method of claim 13, wherein the neural network comprises a multi-scale convolutional neural network including a dynamic temporal layer configured to learn dynamic and frequency representations of all channels of an EEG signal, an asymmetric spatial layer configured to compute global and hemispherical spatial kernels, and a higher-level fusion layer configured to fuse information from the global and hemispherical spatial kernels to form a fully connected layer driven by a softmax function.

Images