Wireless Soft Scalp Electronics and Virtual Reality Systems for Brain-Machine Interfaces

Abstract

Exemplary wireless soft scalp electronic systems and methods are disclosed that can actuate commands for a brain-machine interface (BMI) or brain-computer interface (BCI) by performing real-time, continuous classification of motor imagery (MI) brain signals or of steady-state visual evoked potential (SSVEP) signals, for example, via a trained neural network. In some embodiments, the exemplary system is configured as a thin, portable system including microneedle electrodes that can acquire EEG signals for a brain-machine interface controller. The microneedle electrodes can be configured as soft, imperceptible, gel-less, epidermal-penetrating microneedle electrodes that can provide improved contact surface area and reduced electrode impedance density, for example, to improve EEG signal and signal classification accuracy. The microneedle electrodes can be further integrated with soft electronics that can be locally worn in close proximity to the electrodes to reduce messy wiring and improve signal acquisition quality.

Description

[Technical field]

[0001] Statement of Government Interest This invention was made with Government support under Grant No. R21AG064309 awarded by the National Institutes of Health. The Government has certain rights in this invention.

[0002] Related Applications This PCT application claims priority to and the benefit of U.S. Provisional Application No. 63 / 194,111, entitled "WIRELESS SOFT SCALP ELECTRONICS AND VIRTUAL REALITY SYSTEM FOR MOTOR IMAGERY-BASED BRAIN-MACHINE INTERFACES," filed on May 27, 2021, and U.S. Provisional Application No. 63 / 311,628, entitled "VIRTUAL REALITY (VR)-ENABLED BRAIN -COMPUTER INTERFACES VIA WIRELESS SOFT BIOELECTRON ICS," filed on February 18, 2022, each of which is incorporated by reference in its entirety herein. [Background technology]

[0003] Motor imagery electroencephalography (MI) refers to the mental simulation of body movements by consciously accessing aspects of body movements to provide a mechanism for brain-machine interface. Conventional electroencephalography (EEG) for motor imagery typically involves extensive setup time and uses multiple wired electrodes and hair caps with gel that are uncomfortable to use. Although the latest EEG designs are trending towards wireless wearable EEG for everyday mobile EEG monitoring, they nevertheless continue to use rigid bulky circuits and gel-based skin-contact electrodes that are intrusive in nature, provide low information throughput due to noise-prone brain signal detection, and have limited recording channels.

[0004] Similar EEG hardware can also be used to acquire steady-state visual evoked potentials (SSVEPs), which are brain signals that are natural responses to visual stimuli of specific frequencies. When the retina is excited by a visual stimulus, for example in the range 3.5 Hz to 75 Hz, the brain can generate electrical activity at the same frequency (or a multiple of) the visual stimulus.

[0005] Improvements in brain-machine interface (BMI) hardware and BMI applications would be beneficial. Summary of the Invention

[0006] Exemplary wireless soft scalp electronic systems and methods are disclosed that can actuate commands for a Brain-Machine Interface (BMI) or Brain-Computer Interface (BCI), for example, by performing real-time, continuous classification of Motor Imagery (MI) brain signals or of Steady State Visual Evoked Potential (SSVEP) signals via a trained neural network.

[0007] In some embodiments, the exemplary system is configured as a thin portable system including a microneedle electrode capable of acquiring EEG signals for a brain-machine interface controller. The microneedle electrode may be configured as a soft, imperceptible, gel-less, epidermal-penetrating microneedle electrode that can provide an improved contact surface area and reduced electrode impedance density to improve EEG signal and signal classification accuracy, for example. The microneedle electrode may be further integrated with soft electronics that can be locally attached in close proximity to the electrode to reduce messy wiring and improve signal acquisition quality.

[0008] The exemplary wireless soft scalp electronic system and method may operate in combination with a virtual reality (VR) / augmented reality (AR) training system with a VR / AR environment controller to provide clear and consistent visuals and instantaneous biofeedback to a user in MI or SSVEP applications. In some embodiments, the VR / AR environment controller may use the acquired and classified EEG signals to actuate commands to render object VR / AR scenes associated with motion images (e.g., one or more body objects capable of performing aspects of body movements) that are viewed by the user as feedback to the user during MI training. The VR / AR hardware and brain-machine interface hardware may be used to provide and acquire visual stimuli for acquisition of steady-state visual evoked potentials. The VR / AR hardware and associated training may reduce the variability of detectable EEG responses, for example, in MI and SSVEP applications. In the study reported herein, it was observed that the scalp electronic system and associated training provided high classification accuracy (93.22±1.33% for four classes) for motion imagery applications, enabling wireless real-time control of virtual reality games.

[0009] In one aspect, a system is disclosed that includes an electroencephalogram-based (EEG) brain-machine interface, the system including a set of thin EEG sensors, each comprising an array of flexible, epidermal-penetrating microneedle electrodes fabricated on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit, and a brain-machine interface operably connected to the set of thin EEG sensors, the brain-machine interface comprising a processor and a memory operably connected to the processor, the memory having instructions stored therein, execution of the instructions by the processor causing the processor to receive EEG signals acquired from the thin EEG sensors, continuously classify brain signals from the acquired EEG signals as control signals via a trained neural network, and output the control signals to a virtual reality environment controller to actuate commands (e.g., for training) within a VR scene generated by the virtual reality environment controller that is viewed by the subject.

[0010] In some embodiments, the commands cause a set of limb movements in a VR scene, and the trained neural network is configured to classify brain signals for the set of movements.

[0011] In some embodiments, a set of low-profile EEG sensors is connected to a brain-machine interface via a set of stretchable flexible connectors.

[0012] In some embodiments, the microneedle electrodes have an enlarged contact surface area and reduced electrode impedance density.

[0013] In some embodiments, the system further includes a wearable soft headset comprising a low modulus elastomeric band.

[0014] In some embodiments, the trained neural network comprises a spatial convolutional neural network.

[0015] In some embodiments, a set of thin EEG sensors are placed along the scalp for motion imaging.

[0016] In some embodiments, a set of thin EEG sensors is placed along the scalp for steady-state visual evoked potential (SSVEP) measurements.

[0017] In some embodiments, the virtual reality environment controller is configured to generate split-eye asynchronous stimuli (SEAS) within the virtual scene for a real-time text speller interface.

[0018] In some embodiments, execution of the instructions by the processor further causes the processor to transmit the acquired EEG signals to a remote or cloud computing device that performs a retraining operation of the trained neural network, and to receive an updated trained neural network from the remote or cloud computing device during runtime operation of the virtual reality environment controller.

[0019] In some embodiments, each of the multiple flexible epidermal-penetrating microneedle electrodes of the array is at least 500 μm in height (e.g., 800 μm) for attachment to a hairy scalp, has a base width of about 350 μm, and has an area of ​​about 36 mm2.

[0020] In another aspect, a method is disclosed that may include providing a set of thin EEG sensors placed on a scalp of a user, each set of thin EEG sensors including an array of flexible epidermal-penetrating microneedle electrodes fabricated on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit, receiving acquired EEG signals from the thin EEG sensors by a processor or brain-machine interface operably connected to the set of thin EEG sensors, continuously classifying brain signals from the acquired EEG signals as control signals by the processor via a trained neural network, and outputting the control signals by the processor to a virtual reality environment controller to actuate commands within a VR scene generated by the virtual reality environment controller viewed by the subject.

[0021] In some embodiments, a set of low-profile EEG sensors is placed directly on the scalp without any conductive gel or paste.

[0022] In some embodiments, the set of low profile EEG sensors includes i) a reference array of flexible epidermal-penetrating microneedle electrodes placed at a vertex position on the scalp, and ii) six arrays of flexible epidermal-penetrating microneedle electrodes removably attached to low modulus elastomer bands at a first frontal position, a second dorsal position, and four lateral positions for motion imaging measurements.

[0023] In some embodiments, the set of low profile EEG sensors includes i) a reference array of flexible epidermal-penetrating microneedle electrodes placed in a dorsal position on the scalp, and ii) four arrays of flexible epidermal-penetrating microneedle electrodes removably attached to a low modulus elastomer band at the dorsal region of the scalp for steady-state visual evoked potential (SSVEP) measurements.

[0024] In some embodiments, the method may further include transmitting, by the processor, the acquired EEG signals to a remote or cloud computing device that performs a retraining operation of the trained neural network, and receiving, by the processor, an updated trained neural network from the remote or cloud computing device during runtime operation of the virtual reality environment controller.

[0025] In another aspect, a non-transitory computer readable medium is disclosed. A non-transitory computer readable medium may have instructions stored thereon, the execution of the instructions by a processor of a brain-machine interface controller causing the processor to receive EEG signals acquired from a set of thin EEG sensors placed on a scalp of a user, the set of thin EEG sensors each including an array of flexible epidermal-penetrating microneedle electrodes fabricated on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit, the set of thin EEG sensors being placed directly on the scalp without conductive gel or paste, continuously classifying brain signals from the acquired EEG signals as control signals via a trained neural network, and outputting the control signals to a virtual reality environment controller to actuate commands within a VR scene generated by the virtual reality environment controller that is viewed by the subject.

[0026] In some embodiments, the set of low profile EEG sensors includes i) a reference array of flexible epidermal-penetrating microneedle electrodes placed at a vertex position on the scalp, and ii) six arrays of flexible epidermal-penetrating microneedle electrodes removably attached to low modulus elastomer bands at a first frontal position, a second dorsal position, and four lateral positions for motion imaging measurements.

[0027] In some embodiments, the set of low profile EEG sensors includes i) a reference array of flexible epidermal-penetrating microneedle electrodes placed in a dorsal position on the scalp, and ii) four arrays of flexible epidermal-penetrating microneedle electrodes removably attached to a low modulus elastomer band at the dorsal region of the scalp for steady-state visual evoked potential (SSVEP) measurements.

[0028] In some embodiments, execution of the instructions further causes the processor to transmit the acquired EEG signals to a remote or cloud computing device that performs a retraining operation of the trained neural network, and to receive an updated trained neural network from the remote or cloud computing device during runtime operation of the virtual reality environment controller.

[0029] Those skilled in the art will understand that the drawings, described below, are for illustration purposes only. [Brief description of the drawings]

[0030] [Figure 1] 1 illustrates an exemplary EEG-based brain-machine interface system, according to an exemplary embodiment. [Diagram 2] 1 illustrates an exemplary EEG brain-machine interface system configured as a thin EEG sensor-based soft scalp electronic device for motion imagery (MI) training or operation, according to one exemplary embodiment. [Figure 3A] 1A-1C each illustrate aspects of an exemplary EEG brain-machine interface system configured as a thin EEG sensor-based soft scalp electronic device for SSVEP training or operation, according to one exemplary embodiment. [Figure 3B] 1A-1C each illustrate aspects of an exemplary EEG brain-machine interface system configured as a thin EEG sensor-based soft scalp electronic device for SSVEP training or operation, according to one exemplary embodiment. [Figure 3C]1A-1C each illustrate aspects of an exemplary EEG brain-machine interface system configured as a thin EEG sensor-based soft scalp electronic device for SSVEP training or operation, according to one exemplary embodiment. [Figure 4A] 1 illustrates a method of operating an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 4B] 1 illustrates an exemplary method of operation for configuring and reconfiguring an exemplary EEG brain-machine interface system during runtime operation, according to an exemplary embodiment. [Figure 5A] 1A-1D each illustrate an exemplary method of manufacturing a component of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 5B] 1A-1D each illustrate an exemplary method of manufacturing a component of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 5C] 1A-1D each illustrate an exemplary method of manufacturing a component of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 5D] 1A-1D each illustrate an exemplary method of manufacturing a component of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 5E] 1A-1D each illustrate an exemplary method of manufacturing a component of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 6A] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 6B] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 6C]1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 6D] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 6E] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 7A] 1 illustrates mechanical characterization results of components of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 7B] 1 illustrates mechanical characterization results of components of an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8A] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8B] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8C] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8D] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8E]1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8F] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. [Figure 8G] 1A and 1B each illustrate aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Several references, which may include various patents, patent applications, and publications, are cited in the reference list and discussed in the disclosure provided herein. Citation and / or discussion of such references is provided only to clarify the description of the disclosed technology and is not an admission that such references are "prior art" to any aspect of the disclosed technology described herein. For notational purposes, [n] corresponds to the nth reference in the list. For example, [1] refers to the first reference in the list. All references cited and discussed herein are incorporated by reference in their entirety to the same extent as if each reference was individually incorporated by reference.

[0032] Exemplary System FIG. 1 illustrates an exemplary electroencephalogram-based (EEG) brain-machine interface system 100 ("EEG BMI" system 100) according to one exemplary embodiment. The system 100 includes a set of low-profile EEG sensors 102 (shown as 102a, 102b, 102c, 102d), each comprising an array 104 of flexible, skin-penetrating microneedle electrodes 106 fabricated on a flexible circuit board 108. In the example shown in FIG. 1, a first EEG sensor 102a is shown as a reference electrode connected via a flexible connector 109 to a flexible front-end electronics assembly 110 that interfaces with a BMI control system 112 that classifies the signals via a neural network 114 (shown as "spatial CNN" 114) to generate control signals to a computing device or machine 116. The computing device or machine 116 may include a VR / AR training system 118 and / or a machine computer system 119. The VR / AR training system 118 and / or the machine / computer system 119 may then be configured to execute a VR / AR application 121. The virtual reality application 121 may include a BMI rendering and UI module 122 and a module including game environment parameters 124. The term "VR / AR" refers to a system that can provide a virtual reality system, an augmented reality system, or both.

[0033] Other EEG sensors (denoted as "sensor arrays" 102b, 102c, 102d) are measured via hardware or software in relation to the reference sensor 102a and, in the example of Figure 1, are connected through the reference EEG sensor assembly 102a via flexible cabling 126. The system 100 may use more than one reference sensor assembly (e.g., 102a).

[0034] 1, the flexible cabling 126 includes a set of laser machined, stretchable, flexible interconnects 128. The interconnects 128 may have conductors formed in a meandering or serpentine pattern 130 that allows the interconnects 128 to be stretched or bent. A flexible connector 109 connects the flexible assembly of the reference sensor 102a to the flexible front-end electronics assembly 110.

[0035] The flexible front-end electronics assembly 110 may include one or more analog-to-digital converters 132 operably connected to the array 104 of needle electrodes 102b, 102c, 102d through the flexible cable 126. The ADC 132 may convert analog signals from the reference array of needle electrodes 102a and from the sensor arrays (e.g., 102b, 102c, 102d) to digital signals. The digital signals may be transmitted by a network interface 134 to a network interface 135 within the BMI control system 112. Additionally, the flexible front-end electronics assembly 110 may include a controller 136 that may be configured to control the operation of the energy storage device 138, the ADC 132, and the network interface 134.

[0036] The BMI control system 112 is configured to continuously classify brain signals from the acquired EEG signals, via a trained neural network, as control signals. The BMI control system 112 can provide the control signals to a machine 119, for example, to operate a vehicle (e.g., a powered wheelchair) or a robotic prosthetic limb, etc.

[0037] The BMI control system 112 may include a trained neural network 114, a network interface 135, a controller 137, a filter module 140, and a scaling module 142. The trained neural network 114 is configured to classify acquired EEG signals and generate control signals to a computing device or machine 116. In the example shown in Figure 1, the trained neural network 114 is configured as a spatial CNN. The trained neural network may be configured as other CNNs and AI systems, for example, as described or referenced herein.

[0038] In some embodiments, the BMI control system 112 is configured to be reconfigured during runtime operation. In the example shown in FIG. 1, the BMI control system is shown connected to a cloud system 144 configured with a neural network training system 146. The cloud system 144 is configured to receive EEG signals acquired from the BMI control system 112 and retrain a local version of the neural network 114. The neural network training system 146 determines whether the retrained neural network 148 improves upon the previous neural network 114. Once such a determination is made, the neural network training system 146 provides the retrained neural network 148 to the BMI control system 112, which replaces the neural network 114 with the updated version (e.g., via its controller 137).

[0039] To provide the EEG signals for classification, the BMI control system 112 includes a network interface 135 for communicating and receiving from the network interface 134 of the flexible front-end electronics assembly 110. The filter module 140 and the scaling module 142 are configured to pre-process the acquired EEG signals prior to the classification operation. In some embodiments, the filter module 140 is configured to filter the acquired EEG data using, for example, a Butterworth bandpass filter, and the scaling module 142 is configured to upscale the filtered EEG data using, for example, a linear upscaling operator.

[0040] In the example shown in FIG. 1, the BMI control system 112 may be configured to operate with a VR / AR training system 118 that includes a VR / AR environment controller (not shown) that can use the classified control signals to actuate a set of commands in a VR scene displayed to a user. The VR / AR environment may be implemented using a VR / AR headset and VR / AR software. The VR / AR environment may operate with the VR software (e.g., Unity) to configure a computing device to display VR / AR graphics in the VR / AR headset. The VR / AR headset (e.g., Samsung Gear VR) may be connected to a smartphone. In one exemplary implementation, the VR software may render 3D models (Maya) of hands and feet or other geometric objects to facilitate visualization of the MI.

[0041] It should be understood that the animation software, VR / AR software, VR / AR headsets, and various computing devices described with reference to this exemplary implementation are all intended as non-limiting examples, and that the present disclosure may be implemented using any suitable animation software, smartphone (or other computing device), VR (or AR) headset, and / or any AR or VR software package. Similarly, it should be understood that the games described are non-limiting examples, and that embodiments of the present disclosure may be used to control and receive output from any computing device.

[0042] A computing device may include a processing unit, which may be a standard programmable processor that performs arithmetic and logical operations necessary for the operation of the computing device. Multiple processors may be used. As used herein, processing unit and processor refer to physical hardware devices that execute coded instructions to perform functions on inputs and produce outputs, including, for example, but not limited to, microprocessors (MCUs), microcontrollers, graphical processing units (GPUs), and application specific circuits (ASICs). Thus, while instructions may be discussed as being executed by a processor, the instructions may be executed simultaneously, serially, or otherwise by one or more processors. A computing device may also include a bus or other communication mechanism for communicating information between various components of the computing device.

[0043] It should be understood that the logical operations described above may be implemented (1) as a sequence of computer-implemented operations or program modules operating on a computing system, and / or (2) as interconnected machine logic circuits or circuit modules within a computing system. The implementation is a matter of choice dependent on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are variously referred to as state operations, operations, or modules. These operations, operations, and / or modules may be implemented in software, firmware, special purpose digital logic, hardware, and any combination thereof. It should also be understood that more or fewer operations may be performed than illustrated in the figures and described herein. These operations may also be performed in orders different from those described herein.

[0044] One or more programs may implement or use the processes described in connection with the subject matter disclosed herein, for example through the use of an application programming interface (API), reusable controls, etc. Such programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. Alternatively, the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.

[0045] Example #2 Motor imagery-based brain-machine interface 2 illustrates an exemplary EEG BMI system 100 (shown as 100a) including a flexible front-end electronics assembly 110 (shown as 110a) configured as a thin EEG sensor-based soft scalp electronics (SSE) device for motion imaging (MI) training or operation that interfaces with a VR / AR headset 202 (shown as 202a) according to one exemplary embodiment. The SSE device 110a can be placed along the scalp of a user and includes (i) fully portable signal acquisition electronics on a flexible substrate and (ii) a stretchable interconnect 128 that connects to a set of flexible microneedle arrays 104 (shown as 104a, 104b, 104c, 104d, 104e, and 104f).

[0046] The soft scalp electronics system 110a may be configured for MI brain signal detection for continuous BMI by continuously recording brain signals via a head-worn strap 206. The SSE system 110a is configured to provide acquired EEG signals via a wireless connection (or via a wired connection) to an external computing device, which then classifies the acquired EEG signals as signals, for example, for MI applications or for immersive visualization training.

[0047] In the example shown in FIG. 2, the BMI system 100a includes a reduced number of EEG electrodes that are easier to set up and reduce the complexity of setup use, without sacrificing classification performance, e.g., compared to traditional EEG applications.

[0048] In one embodiment, in the example shown in FIG. 2, the SSE system 110a includes an array of integrated stretchable interconnects 128 bonded to flexible microneedle electrodes (FMNEs) (e.g., 104). The soft scalp electronics system 110a can be fabricated using flexible membrane circuits to have high mechanical compliance. The flexible membrane circuits can be integrated with electronic chips (e.g., front-end acquisition ICs and network interface ICs) and encapsulated to maintain mechanical compliance.

[0049] In the example shown in Figure 2, each of the arrays of FMNEs (e.g., 104) includes a set of high aspect ratio needles, e.g., greater than 2 (e.g., 800 μm high with a base width of 350 μm). Other base-to-height ratios of the needles may be used, e.g., 1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0. In some embodiments, the base-to-height ratio of the needles may be greater than 2.

[0050] In the example shown in FIG. 2, the soft scalp electronics system 110a is attached or secured to a wearable head strap 206, which may be integrated with a set of low modulus elastomeric bands 208 that may be molded together to secure multiple FMNEs at MI locations on the user's scalp. The primary bands 206 may wrap around the user's head around an axial plane 210 to secure five FMNEs on the temporal lobe (reference FMNE 104a ("Cz"), first axial FMNE 104b ("C2"), second axial FMNE 104c ("C3"), third axial FMNE 104d ("C4"), and fourth axial FMNE 104e ("C5")). The primary band 208 connects to the inion FMNE 104f ("Fz") and nasion FMNE 104g ("POz") via flexible interconnects to provide six channels of EEC measurement relative to the reference electrode array. A schematic diagram of the same is shown in plot 212. Plot 212 also shows the FMNE in relation to a standard EEG cap having 20 or more electrodes. The primary band 208 is also connected to a ground electrode 214 configured to be placed behind the ear. Other numbers of electrode arrays may be used, including 7, 8, 9, 10, 11, 12, etc. In some embodiments, the number of electrode arrays may be greater than 12.

[0051] The electrode array (e.g., 104) is connected to the ADC front-end circuitry (including ADC 132, shown as 132a) of the soft scalp electronics system 110a. The soft scalp electronics system includes a network interface 134 (shown as "Bluetooth controller" 134a) that can communicate acquired EEC signals to the BMI control system 112 (shown as "tablet" 112a). In the example shown in FIG. 2, the BMI control system 112a is configured to process sequences from the EEG recording machine learning classification algorithm 114 (shown as "convolutional neural network" 114a) to generate MI classifications that can be used as control signals to control VR / AR targets within the VR / AR system environment. The machine computer system 120a includes a filter operation 140 (shown as 140a) and a rescaling operation 142 (shown as 142a). The machine computer system 112a is configured to optimally capture event-related synchronization and desynchronization for, for example, separate hands and feet, as well as global alpha rhythm activity. The ML model can resolve spatial features from multiple bipolar sources in the motor cortex. The output of the classification can be sent as commands 220 to a target, shown as VR target 222.

[0052] Example #3 SSVEP-based brain-machine interface 3A illustrates an exemplary EEG BMI system 100 (shown as 100b) including a flexible front-end electronics assembly 110 (shown as 110b) configured as a thin EEG sensor-based soft scalp electronics (SSE) device for SSVEP training or operation that interfaces with a VR / AR headset 202 (shown as 202b) according to an exemplary embodiment. The SSE device 110b can also be placed along the scalp of a user and includes (i) fully portable signal acquisition electronics on a flexible substrate and (ii) a stretchable interconnect 128 that connects to a set of flexible microneedle arrays 104 (104', e.g., shown as 104a', 104b', 104c', 104d', 104e', see FIG. 3B).

[0053] To improve the throughput of brain signal recording, in the example shown in FIG. 3A, the EEG BMI system 100b is used to obtain SSVEP signals from different eye-specific stimuli being presented to each eye via a split-eye asynchronous stimulation (SEAS) application. Separate eye stimulation can generate unique asynchronous stimulation patterns that can provide more encoded channels to improve the throughput of brain signal recording. The EEG BMI system 100b can be used to provide real-time monitoring of steady-state visual evoked potentials (SSVEPs) for portable BCI with more than 30 channels, for example, for text spelling. In the example shown in FIG. 3A, a user interface panel 302 is presented, for example, in a VR / AR environment, with text elements 304, and the text elements 304 (highlighted portion) include different steady-state blinking stimulation patterns. That is, each text element 304 (32 elements) can be encoded with a unique steady-state blinking stimulation. In SEAS, the steady-state blinking stimulation patterns can now be different on the left and right displays (shown as 308 and 310, respectively).

[0054] FIG. 3C shows exemplary stimulus frequencies (312) and targets (shown as “Target Text” 314). In the example shown in FIG. 3C, the stimulus frequencies 312 are presented either as the same frequency to both eyes (lines “1” and “3” shown as 316 and 318) or as different frequencies to the left and right eyes (lines “2” and “4” shown as 320, 322). In lines “2” and “4” (320, 324), each of the first numbers (326) represents a frequency seen by the left eye and each of the second numbers (328) represents a frequency seen by the right eye. Other configurations may be used. For example, the output of the same frequency by lines “1” and “3” provides a common reference for the eye to track to what extent different frequencies by lines “2” and “4”, for example, can be presented asynchronously and detected consistently. In the case of asynchronous operation, unique frequencies should be used between the left and right eyes due to possible mixing within the subject that may affect classification.

[0055] It should be noted that although the example shown in FIG. 3C shows the frequencies arranged in a particular order or increment, the ordering of the unique frequencies may be varied (and still provide similar performance).

[0056] In conjunction with the deep learning algorithms and soft electronics hardware described herein, the EEG BMI system 100b can provide real-time data processing and classification for, for example, 33 classes of SSVEP input. In the studies reported herein, it was observed that the EEG BMI system 100b can provide 33 discriminative classes with an accuracy of 78.93% within a 0.8 second acquisition window and 91.73% within a 2 second acquisition window.

[0057] 3A, the SSE system 110b may also include an array of integrated stretchable interconnects bonded to the flexible microneedle electrodes (FMNEs) 104'. The soft scalp electronics system 110b may be fabricated using flexible membrane circuits to have high mechanical compliance. The flexible membrane circuits may be integrated with electronic chips (e.g., front-end acquisition ICs and network interface ICs) and encapsulated to maintain mechanical compliance.

[0058] In the example shown in Figure 3A, each of the arrays of FMNEs 104' includes a set of high aspect ratio needles, e.g., 800 μm high with a base width of 350 μm. Other base-to-height ratios of needles may be used, e.g., 1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0. In some embodiments, the base-to-height ratio of the needles may be greater than 2.

[0059] In the example shown in FIG. 3A, the soft scalp electronics system 110b is connected to an array of FMNEs 104' attached or secured to a wearable head strap 206 (shown as 206').

[0060] The electrode array 104' is connected to an ADC front-end circuit (e.g., including ADC 132) of the soft scalp electronics system 110b. The soft scalp electronics system 110b includes a network interface (e.g., 134) that can communicate acquired EEC signals to a BMI control system (e.g., 112). In the example shown in FIG. 3A, the BMI control system 112a is configured to process sequences from the EEG recordings machine learning classification algorithm (e.g., 114) to generate SSVEP classifications that can be used as control signals for controlling VR / AR targets within the VR / AR system environment.

[0061] To address the complexity of the stimuli and maintain a high level of classification performance, machine learning operations may be performed on a per-session basis. Figure 4B illustrates an exemplary method 330 of operations for configuring and reconfiguring a BMI control system (e.g., 112) during runtime operation.

[0062] In the example shown in FIG. 4B, the method 430 includes acquiring (432) an EEG signal, for example, via the soft scalp electronic system 110b. The EEG signal may be acquired during a calibration operation. Multiple training trials may be performed to acquire sufficient data to train the machine learning model with minimal bias. The calibration operation may be performed before each session. The method 430 then includes transmitting (434) the acquired signal to a training system (e.g., a cloud infrastructure). The training system may preprocess (436) the acquired signal via segmentation (438) (e.g., segmenting the data in the range of 0.8 seconds to 2 seconds), filtering (440) (e.g., using a bidirectional third order high-pass Butterworth filter with a corner frequency of 2.0 Hz), and a rescaling operation (441) (e.g., linearly rescaling between the range of -0.5 to 0.5).

[0063] The training system may perform classification operations (438) by training variations of the spatial CNN model with hyperparameter tuning (e.g., filter size, number of filters, and number of convolution steps). The training system then determines (440) whether performance is improved. If so, the training system then sends (442) the model parameters to a runtime system (e.g., the BMI control system 112).

[0064] In some embodiments, the EEG BMI system 100b for SSVEP may be used in combination with the EEG BMI system 100a for MI.

[0065] Exemplary Methods of Operation 4A illustrates a method 400 of operating an embodiment of a system disclosed herein to output a control signal, comprising providing 402 a thin EEG sensor, such as those described in connection with FIG. 1, 2A, 3A, or other BMI configurations described herein.

[0066] Next, the method 400 includes receiving, by a processor or brain-machine interface operatively connected to the set of thin EEG sensors, the acquired EEG signals from the receiving thin EEG sensors (404). Examples of data acquisition are provided in connection with Figures 1, 2A, and 3A.

[0067] The method 400 then includes continually classifying (406), by the processor, via the trained neural network, brain signals from the acquired EEG signals as control signals.

[0068] The method 400 then includes outputting (408), by the processor, a control signal to the virtual reality environment controller to actuate a command within a VR scene generated by the virtual reality environment controller that is viewed by the subject.

[0069] Example #1: Method for manufacturing flexible microneedle arrays 5A illustrates a method 500 for fabricating an array of electrodes on a substrate according to an exemplary embodiment of the present disclosure. In the example of FIG. 5A, the area of ​​each electrode array is approximately 36 mm 2 , thereby eliminating the need for conventional large cup electrodes (100 mm 2 ) can improve EEG spatial resolution over other array sizes, e.g., about 10 mm 2 , about 15mm 2 , about 20mm 2 , about 25mm 2 , about 30mm 2 , about 35mm 2 , approx. 40mm 2 , about 45mm 2 , about 50 mm 2can be used, where "about" refers to a range of ±1 mm, ±2 mm, or ±2.5 mm. In some embodiments, the array size is 50 mm 2 More than 100mm 2 A typical EEG with conductive gel is about 100 mm in size. 2 It is.

[0070] In FIG. 5A, process 500 includes providing a master negative PDMS (polydimethylsiloxane) array mold (502). The PDMS array mold can be cleaned using IPA and dried at 60° C. for 10 minutes. Process 500 includes using the negative PDMS mold to create an epoxy positive mold (504). The positive mold can be formed by epoxy resin (e.g., EpoxAcast resin manufactured by Smooth On, Inc.). Process 500 includes transferring the epoxy positive mold to a glass slide (506). As an adhesive layer for PDMS, an adhesive can be used to bond the PDMS negative mold to the glass slide. Image 506′ shows the positive epoxy mold. Processes 502-506 can be repeated to form additional copies of the epoxy positive mold. In some embodiments, the PDMS negative mold can be treated with ambient air plasma (e.g., for 2 minutes).

[0071] Process 500 includes positioning copies of the epoxy positive mold in an array in a tray (508). Process 500 includes adding additional PDMS to the tray to form a new set of negative molds (510). The molds shown in step 508 are a 4×2 array, but it should be understood that any number of epoxy molds created in steps 502-506 may be used. Process 500 includes peeling 512 the PDMS negative mold from the tray used in steps 508 and 510. Image 512′ shows the negative silicone mold formed from PDMS. Image 522′ shows the final coated electrodes. The mold components are formed in steps 502-512.

[0072] To fabricate the needles, the process 500 first includes adding a diluted polyimide (PI) solution to the PDMS negative mold (514), which may then be soft baked (e.g., 80° C. for 10 minutes). The process 500 includes adding a second diluted polyimide solution to the mold at 516, which may then be hard baked (e.g., 200° C. for 1 hour). In the example shown in FIG. 5A, the first diluted polyimide solution is a 3:1 ratio solution and the second diluted polyimide solution is a 2:1 solution. Other ratios and processing times can be used. The process 500 includes removing the PI needles from the mold (518), for example, peeling the polyimide microneedle array (PI MNA) from the PDMS mold.

[0073] The PI needle can then be coated with a conductive layer. The process 500 can include placing the PDMS needle on the PDMS coated slide (520). A thin layer of polyimide (PI) (e.g., PI sold by HD Microsystems under the trademark PI 2610) can be applied to the negative PDMS mold by rubbing with a razor blade before soft baking. The PI can be spin-coated on the mold, for example, at 800 RPM for 60 seconds. The process 500 then includes coating the PI needle using sputter deposition (522), for example, via a Cr coating and then an Au coating, where the CR and AU depths are 5 nm and 200 nm, respectively. The sputtering can be performed in multiple steps. For example, the top or bottom surface of the PI needle can be sputter-coated in one step, and then the remaining surface can be sputter-coated in the next step.

[0074] Example #2: Method for manufacturing flexible microneedle arrays FIG. 5B illustrates another method 530 of fabricating an array of electrodes on a substrate according to an exemplary embodiment of the present disclosure. The method 530 includes creating a PDMS negative mold (512), for example, as described in connection with FIG. 5A. Then, the process 530 further includes adding (532) a thin layer of epoxy (e.g., EP4CL-80MED, Master Bond Inc.) to form a needle 533. The epoxy can be a one-part epoxy with high tensile and compressive strength, as well as its biocompatibility. The one-part epoxy can be used without a solvent, which can prevent the epoxy from bubbling, does not require a mixing step, and avoids mixing to introduce air into the epoxy. Then, the process 530 includes adding (534) a perforated polyimide sheet (535) to the needle. Image 534" shows an exemplary design of a perforated polyimide sheet, and image 534' shows an exemplary fabricated perforated polyimide sheet with high compliance and flexibility. The process 530 then includes performing a low temperature cure (536) (e.g., 100°C for 1 hour). The low temperature cure may be used to allow the mold to be repeatedly used for more replica molding cycles. In contrast, in some embodiments, a high temperature PI cure may destroy the PDMS mold after as few as three fabrication cycles.

[0075] The process 530 then includes peeling the needle structure from the mold and placing the needle structure on a PDMS coated slide (538). The needle may be coated by sputtering Cr / AU on both sides of the needle structure (540). Image 538' shows the needle assembly before Cr / AU coating, and image 540' shows the needle assembly after Cr / AU coating.

[0076] Exemplary Methods for Manufacturing Flexible Main Circuits 5C and 5D show an exemplary method 550 of manufacturing a flexible main circuit, according to one exemplary embodiment. The flexible main circuit may include a polyimide substrate, which may be thin enough to allow the curvature of the electrodes to conform to the scalp surface.

[0077] Method 550 includes spin-coating PDMS (552) (e.g., 3000 rpm for 30 seconds) onto a cleaned silicon wafer. The PDMS coated wafer may be cured (e.g., on a hotplate at 150° C. for 5 minutes). An oxygen plasma treatment may be performed to make the PDMS hydrophilic. Image 552' shows the spin-coated PDMS. Method 550 then includes spin-coating polyimide (553) (e.g., 4000 rpm for 1 minute) and baking in a vacuum oven (e.g., 250° C. for 3 hours with a ramping step). Image 553' shows the first polyimide spin-coated wafer.

[0078] The method 550 then includes sputtering copper (e.g., 500 nm copper) for a first metal layer 554. Image 554' shows the first copper-deposited wafer.

[0079] The method 550 then includes patterning (555) the wafer by spin-coating (e.g., 3000 rpm for 30 seconds) and baking (e.g., on a hotplate at 110° C. for 1 minute) a photoresist (e.g., SC1827), exposing it to UV using a mask aligner (e.g., MA6) with a first metal pattern (which has been polished), and developing the exposed photoresist with a developer (e.g., MF-319).

[0080] The method 550 then includes etching 556 the exposed copper with a copper etchant (APS-100, diluted 1:1 with DI water) and stripping the photoresist. Image 556' shows the bottom Cu etch circuit where the Cu etch has been performed. Next, the method 550 includes spin coating 557 polyimide (e.g., 850 rpm for 1 minute) and baking it in a vacuum oven (e.g., 250° C. for 3 hours with a ramping step). Image 557' shows the second polyimide spin coated wafer.

[0081] The method 550 then includes patterning the wafer (558) by spin-coating (e.g., 1000 rpm for 30 seconds) and baking (e.g., on a hotplate at 95° C. for 4 minutes) a photoresist (e.g., AZP4620) and developing the exposed photoresist with a developer (e.g., AZ400K, diluted with 4 parts DI water). The method 550 then includes exposing the PI to an oxygen plasma etch (559) using a reactive ion etch (Plasma-Therm) and removing the photoresist. Figure 559' shows the etched polyimide circuit with vias.

[0082] The method 550 then includes depositing 560 a second Cu layer by sputtering (eg, 1.5 μm of copper for the second metal layer). Image 560′ shows the second deposition wafer.

[0083] The method 550 then includes patterning the wafer (561) by spin-coating a photoresist (e.g., AZP462) at, for example, 3000 rpm for 30 seconds and baking it (e.g., on a hot plate at 95° C. for 4 minutes), exposing it to UV using a mask aligner (e.g., MA6) with a second metal pattern, and developing the exposed photoresist with a developer (e.g., AZ400K, diluted with 4 parts DI water). The method 550 then includes etching (562) the exposed copper with a copper etchant (APS-100, diluted 1:1 with DI water) and removing the photoresist therefrom. Image 562′ shows the top Cu etched circuit.

[0084] The method 550 then includes spin coating 563 polyimide (e.g., 4000 rpm for 1 minute) and baking in a vacuum oven (e.g., 250° C. for 3 hours with a ramping step). Image 563′ shows the third polyimide spin-coated wafer.

[0085] The method 550 then includes patterning the wafer (564) by spin-coating and baking a photoresist (AZP462), for example at 2000 rpm for 30 seconds (e.g., on a hot plate at 95° C. for 4 minutes), and developing the exposed photoresist with a developer (AZ400K, diluted with 4 parts DI water). The method 550 then includes performing an oxygen plasma etch exposure PI (565) using a reactive ion etch (Plasma-Therm) and stripping the photoresist to produce the final flexible circuit. Image 565′ shows the polyimide etched top circuit with the Cu deposition layer exposed.

[0086] The method 550 then includes incorporating the IC on the flexible circuit (566) by transferring the circuit to a glass slide (see image 566'), reflowing solder onto the chip components to incorporate the IC (see image 566"), and encapsulating the entire circuit (e.g., 110) in an elastomer (see image 566"'). FIG. 5D shows the final fabricated flexible circuit bent onto a glass slide. Descriptions of alternative methods are given in Mahmood et al. 2021, Mahmood et al. 2019, Zavanelli et al. 2021.

[0087] Example #1 Method for Manufacturing Stretchable Interconnects 5E illustrates a method 570 for fabricating an exemplary flexible interconnect (e.g., 128) according to one exemplary embodiment. To improve throughput and reduce manufacturing code (e.g., compared to conventional microfabrication in a clean room), the method 570 may use a femtosecond laser cutter (WS-Flex USP, OPTEC) to fabricate the stretchable interconnect (e.g., 128) using a microfabrication process. The method 570 may include three main fabrication processes: PET substrate preparation for polyimide film, sputtering Cr / Au on the polyimide film, and laser cut patterning.

[0088] To prepare a PET substrate for the PI film, the method 570 may include spin-coating PDMS (Sylgard 184, Dow Corning) (572) onto a PET sheet (e.g., at 700 rpm for 60 seconds), curing it (e.g., at 70° C. for 30 minutes), and then attaching the prepared Cr / Au on PI sheet onto the PDMS (e.g., ensuring that it adheres completely and is free of air bubbles or ripples). The method 570 then includes spin-coating (573) excess PI 2610 (e.g., 3000 rpm for 1 minute) and performing a first bake step on a hotplate (e.g., 70° C. for 30 minutes), removing the PI film from the PDMS / PET substrate after the first bake step and taping (576) the PI film directly to a clean hotplate to prevent thermal curling and shrinkage), and then proceeding with a second bake operation (e.g., 100° C. for 15 minutes, then 150° C. for 15 minutes, then 200° C. for 15 minutes, then 250° C. for 15 minutes) (578).

[0089] To sputter Cr / Au onto the PI film, method 570 may first include taking a 0.5 mil sheet of PI film (Kapton HN Film, 0.5 mil, DuPont) and thoroughly washing it, for example, first with IPA and then with acetone, drying after each wash. Method 570 may then include cutting the PI film into a sheet sized 6 inches by 4 inches to fit within a sputter machine. Method 570 may then include sputtering Cr / Au (10 nm / 200 nm) onto the PI film (574).

[0090] To pattern with a laser cutter (580), the method 570 includes re-attaching the PI film sandwich onto the PDMS on the PET substrate and using a femtosecond laser cutter (WS-Flex USP, OPTEC) to secure the material to the stage using a vacuum. The method 570 can then include aligning the material with the stage and adjusting the material by zeroing the laser head so that the masked areas align with the interconnect ends of the design. The pattern can be cut, for example, with IFOV mode, 60 kHz pulses, a travel speed of 60, a jump speed of 60, 12% power, and 2 iterations (3 passes total). The method 570 can then include peeling (e.g., manually peeling) (582) the final cut interconnects from the PDMS substrate using fine-tipped tweezers. Image 582' shows the patterned interconnect before it is peeled. Image 584 shows the patterned interconnect as it is being peeled. Images 584 show exemplary extensibility characteristics of a flexible interconnect (e.g., 128) at 0%, 50%, and 100% stretch (584a, 584b, and 584c, respectively). Plot 586 shows mechanical testing results of the flexible interconnect (e.g., 128) over a set of cycles, and plot 588 shows the strain / resistance curve of the flexible interconnect (e.g., 128). Testing shows mechanical failure after 250% tensile stretch.

[0091] Example #2 Method for Manufacturing Stretchable Interconnects The substrate for the interconnects is prepared by e-beam evaporation of Cr / Au (5 nm / 200 nm) on a 2 mil PI film (200 HPP-ST, DuPont). The metal-coated PI film can then be laminated onto the PDMS-coated PET film to retain the material during the laser cutting process. Once an array of stretchable interconnects is patterned onto the metal-coated PI film, excess material other than the patterned interconnects can be manually peeled off from the PDMS-coated PET film. Using water-soluble tape, the interconnects are transfer printed onto a soft elastomeric substrate (Ecoflex 00-30, Smooth-On, Inc.) and the areas other than their contact pads are encapsulated with an additional layer of elastomer. The interconnects are electrically connected to the electrodes and sensors with silver paint (Fast Drying Silver Paint, Ted Pella).

[0092] Experimental Results and Example #1 Two studies were conducted to develop and evaluate a virtual reality brain-machine interface, the first using a motion imagery BMI and the second using steady-state visual evoked potentials (SSVEPs).

[0093] Overview of the motor imagery study. Figures 6A, 6B, 6C, 6D, and 6E each show aspects of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment. Specifically, Figure 6A shows an overview of a study to develop a virtual reality (VR) implementation for motor imagery training and real-time control of a video game demonstration. The study evaluated a fully portable, wireless, soft scalp electronic device that includes at least three main components: 1) multiple flexible microneedle electrodes for attachment to a hairy scalp, 2) laser-machined stretchable and flexible interconnects, and 3) a thin flexible circuit. The study also included a virtual reality (VR) component that may enable a convenient immersive training environment to aid in motor visualization. These components were used in the study as a monolithic EEG system optimized to minimize motion artifacts and maximize signal quality. A transepidermal electrode was used to provide optimal impedance density on the scalp, improve the signal-to-noise ratio, and improve spatial resolution for MI recording. Overall, this study showed that the exemplary device and system embodiments provide a feasible approach to a high-performance BMI system that can work well with powerful machine learning algorithms and virtual reality environments. In addition, this study showed that the embodiments of the present disclosure can use an imperceptible hair-wearable system with only six EEG channels and achieve a high accuracy of 93.22±1.33% for four classes with a peak information transfer rate of 23.02 bits / min in four human subjects.

[0094] Methodology. In this study, we developed a customized Android application configured to provide real-time, continuous motion image classification of six-channel MI signals. The Android application was used to evaluate the VR training and testing process. In the training system, the system presented the subject with a modified view (630) of VR visuals accompanied by text and animation prompts designed for MI response testing. In the example shown in FIG. 6A, VR screen 632 is an exemplary VR scene including hands and feet detached from the torso. VR screen 633 is an exemplary VR scene including distinct color-coded visual cues and text prompts that can be activated by the user via motion imagery. The developed application showed the VR scene 635 with associated EEG signals 637 observed by the measurement equipment. Plot 619 shows the EEG signals 637 obtained from one of the interfaces of the Android application.

[0095] Results of an exemplary embodiment of the present disclosure demonstrate the superior performance of the VR environment as a training tool (2240 ​​samples from 4 subjects, 560 samples per subject, window length w=4 seconds). Further accuracy improvements were also observed with the FMNE and VR setup. The improved accuracy can be attributed to the immersive VR training program, where the hands and feet, detached from the torso, are presented in the subject's field of view in approximately the same position as the subject's real limbs (632). In this study, subjects were allowed to gently rotate and tilt their heads to visualize their hands and feet within the VR application.

[0096] Neural Network Training. FIG. 6B shows the neural network training of the MI classification system used in this study. In the example shown in FIG. 6B, training for a spatial convolutional neural network for motor image classification is shown. The system acquired six EEG channels (618) and decomposed them into spatial features from multiple bipolar sources of the motor cortex. The input (618) included six EEG data with a predefined sample size (shown in this example as 1000 samples). The spatial convolutional neural network used in the study included several hidden layers (634) (shown as "2D Convolution" 634a and "2D Spatial Convolution" 634b, 634c, 634d, 634e). In step 637, a flattening step can be performed to generate a dense output layer.

[0097] Classification Results. FIG. 6C shows the classification results (N=4 subjects) using the spatial CNN of FIG. 6B in this study. FIG. 6C includes a comparison plot 640 of spatial CNN classification accuracy between four analysis bases, including raw data, high-pass filtered data (HPF), bandpass filtered data (Bandpass), and power spectral density analysis (PSDA). The analysis was performed and is presented over multiple window lengths (1, 2, and 4 seconds). Error bars indicate standard error from four subjects.

[0098] 6C also includes a second comparison plot 642 of spatial CNN classification accuracy between those using conventional Ag / AgCl gel electrodes and the exemplary FMNE. The analysis was also performed over multiple window lengths (1, 2, and 4 seconds), with error bars showing standard error from four subjects.

[0099] 6C also shows two confusion matrices (644, 646) showing results from a real-time accuracy test of motor imaging brain data acquired with conventional Ag / AgCl electrodes and with the exemplary FMNE. In matrix 644, the results for the conventional Ag / AgCl electrodes show an overall accuracy of 89.65% (N=2,240 samples, window length=4 seconds, and four human subjects). In matrix 646, the results for the exemplary FMNE show an overall accuracy of 93.22% (N=2,240 samples, window length=4 seconds, and four human subjects).

[0100] FIG. 6C also shows two additional confusion matrices (648, 650) showing results from a real-time accuracy test of the motor imagery brain data obtained using a spatial CNN classifier and using a standard CNN classifier. In matrix 648, the performance results of the spatial CNN classifier show an overall accuracy of 93.3% for the FMNE dataset (n=2240 samples from 4 subjects, 560 samples per subject, window length w=4 seconds). In matrix 650, the performance results of the standard CNN classifier for the FMNE dataset show an overall accuracy of 64% (n=2240 samples from 4 subjects, 560 samples per subject, window length w=4 seconds).

[0101] FIG. 6E shows a table showing a comparison of the exemplary embodiment with other devices as reported in the literature [15, 20, 25, 26, 27, 21]. Indeed, it can be observed that the exemplary BMI sensor device and system can provide accurate control for, for example, virtual reality games using the MI paradigm. The table shows that the exemplary BMI sensor device and system can provide an ITR of about 23 bits / min using only six electrodes, in one implementation, providing an accuracy of over 93%. Other performance results for other configurations are also reported herein.

[0102] Analysis of channel selection. FIG. 6D shows the results of a preliminary analysis carried out in this study to evaluate the optimized number and selection of channels. For this analysis, 44 channels of data from the conventional 128 channels from 13 healthy normal subjects, as carried out in a previous study (High-Gamma Dataset) [1], were considered. From the 44 channels that were most significant in the analysis, six optimal channels were determined. These six channels were then implemented as a sensory array set, for example, as shown and described in relation to FIG. 2. Other channels may be similarly selected depending on the patient or subject being evaluated, including symptomatic subjects.

[0103] The example shown in Figure 6D shows the entire data set (652) used in the study. The data was pre-processed and divided into 4 second windows using a third order Butterworth bandpass filter with corner frequencies of 4 Hz and 30 Hz.

[0104] The data was used to train a convolutional neural network (CNN) with a standard convolution on the first layer, followed by four spatial convolutional layers (654), with a filter size of (10,1), to generate the trained network.

[0105] The data (652) was also evaluated (656) using a generator that periodically cycled through the data channels (while eliminating the remaining channels) to calculate output perturbations on selected channels. The resulting data was fed to the trained network (generated from 654). The output perturbations were compared to the true expected outputs to generate a relative perturbation for that channel (658). These relative perturbations were summed (662) across classes to generate a final perturbation value for each of the channels.

[0106] The results are then compared and ranked 664. The bar graph shows the relative perturbation of each channel with the top six channels labeled. In this study, a reduced number of electrodes (i.e., six) was used, which reduced the complexity of the setup without significantly compromising the classification performance.

[0107] Indeed, it has been observed that,for analysis, using the exemplary FMNE, a smaller number of,channels can be used than in a conventional EEG setup.,Approximate locations of electrodes corresponding to a standard,10-10 electrode placement system are discussed in

[16] .

[0108] Buckling Force Evaluation of Microneedle Electrodes. FIG. 7A shows the results of quantitative mechanical testing conducted during a study of the buckling force performance of an exemplary microneedle electrode, fabricated using the process described in connection with FIG. 5B, for example. Specifically, FIG. 7A shows SEM observations (710) of a microneedle electrode evaluated under a motorized force gauge applied via axial force on a single microneedle. Plot (712) shows the force versus displacement curves from the buckling force evaluation of five electrodes. It was observed that the five fabricated FMNEs were able to withstand an average applied force of at least up to 626 mN, which is well above the skin insertion force of a single microneedle (20-167 mN)

[17] .

[0109] Flexibility evaluation of microneedle electrodes. FIG. 7B shows results from a cyclic bending mechanical test to evaluate the flexibility of an exemplary FMNE, for example, to evaluate mechanical robustness in tissue insertion. The exemplary FMNE was configured as a gold-based electrode attached to the skin surface, which is safe to use due to its excellent biocompatibility. FIG. 709 shows a schematic diagram of a cross section of a test sample. During testing, the needle electrode was bent continuously up to 100 times with a radius of curvature of 5 mm while measuring the change in electrical resistance. The results show a negligible resistance shift of less than 0.6 Ω.

[0110] Impedance density characterization of microneedles. Table 1 shows the results of a comparative study of impedance and impedance density of microneedle (MN) electrodes. Different microneedle designs of various heights were evaluated in this study. The designs included a fixed base width of 200 μm and a pitch of 500 μm in a 14×14 array. [Table 1]

[0111] The impedance density (ID, kΩ·cm2) can be calculated by multiplying the measured impedance (Z, kΩ) by the measured electrode contact area (A, cm2): ID=Z*A.

[0112] Considerations for motor imagery. Brain-machine interfaces (BMIs) offer a possible solution for individuals with physical disabilities such as paralysis, or brain injuries that result in similar motor impairments. Among them, a notably challenging disability is locked-in syndrome, where the individual is conscious but unable to move or communicate [1]. Here, BMIs may be able to restore some function to these individuals, providing greater capacity for movement and communication and improving quality of life [1, 2]. Electroencephalography (EEG) is the best non-invasive method for acquiring electrical activity of the brain [3-5], where electrodes attached to the scalp surface record the sum of postsynaptic potentials in the superficial cortex. Traditional research-grade and medical-grade EEGs measure signals at the scalp using hair caps or large headgear with multiple rigid electrodes. These heavy and bulky systems are uncomfortable to wear and often require large rigid electronics that are either attached to the system or isolated using long leads [3]. These devices rely heavily on consistent skin-electrode impedance and typically suffer from significant motion artifacts and electromagnetic interference [3, 4]. Typically, electrodes are combined with conductive gels or pastes to improve surface contact and reduce impedance. These interface materials are a source of noise due to changes in impedance at these locations due to motion artifacts or material degradation. Overall, conventional systems require significant set-up time and are inconvenient and uncomfortable to use.

[0113] Improved signal acquisition can be performed with lightweight, flexible electronics and dry electrodes [6, 6', 6"]. Modern EEG designs show a trend towards wireless wearable EEGs. These may be suitable for everyday mobile EEG monitoring with compact battery-powered designs that are superior to conventional amplifiers and hair-cap EEGs. For mobile systems, dry electrodes are preferred due to their short setup time, lack of skin irritation, and superior long-term performance [7, 8]. Furthermore, dry electrodes often outperform gel-based EEG sensors while offering long-term wearability without signal quality degradation [4, 7, 9]. The recent development of skin-interfaced electrodes for the demonstration of biopotential acquisition demonstrates new strategies and solutions for on-skin bioelectronics

[10] . Examples include the use of screen-printed highly conductive composites with excellent stretchability and interfacial conduction properties

[11] and nanowire-based networks fabricated via interfacial hydrogen bonding in solution

[12] . Non-invasive EEG There are several strategies for BMI paradigms

[13] . Steady-state visual evoked potentials (SSVEP) can be studied [3], and the subject can operate the machine interface by shifting his gaze between flashing stimuli of different frequencies. However, SSVEP recordings have limited practical applications due to the requirement of an array of visual stimuli that obstruct the operator's view. Also, bright flashing stimuli can cause eye strain and fatigue if used for long periods of time. Alternatively, motor imagery (MI) is a very advantageous paradigm for sustained BMI, since it does not require the use of external stimuli, and its class is based on imagined motor activities such as opening and closing a hand or moving a foot [14, 15]. With MI, a specified motor imagery task can lead to fluctuations of sensorimotor rhythms within the corresponding motor cortical areas, which are measurable with EEG.

[0114] Experimental Results and Example #2 Overview of Steady State Visual Evoked Potential (SSVEP) Study. A second study was conducted to evaluate a wireless soft bioelectronic system and a VR-based SSVEP detection platform. Figures 8A, 8B, 8C, 8D, 8E, 8F, and 8G each show aspects of a study to develop a virtual reality (VR) implementation for SSVEP training and real-time control using an exemplary EEG brain-machine interface system, according to an exemplary embodiment.

[0115] In this study, the platform was configured for split-eye asynchronous stimulation operation and evaluated for information throughput as a portable brain-computer interface (BCI). Among other things, this study confirmed that a VR interface with 33 stimulation classes could be implemented with real-time wireless recording of SSVEP for text spelling. The soft wearable platform included flexible circuits, stretchable interconnects, and dry needle electrodes, which operated together with a VR headset to provide a fully portable wireless BCI. This study also demonstrated that skin-compatible electrodes provide a biocompatible and consistent skin-electrode contact impedance for high-quality recording of SSVEP. Compared to a conventional tethered EEG system, the exemplary wireless soft electronic system showed superior performance in SSVEP recording. A spatial CNN classification method integrated with the soft electronics provided real-time data processing and classification, showing accuracy rates ranging from 78.93% in 0.8 seconds to 91.73% in 2 seconds for 33 classes from nine human subjects. In addition, the bioelectronic system with only four EEG recording channels demonstrated high ITR performance (243.6 ± 12.5 bits / min) compared to previous studies, enabling the successful demonstration of VR text spelling and navigation in a real-world environment.

[0116] This study also showed that excellent signal reproduction with minimal artifacts could be afforded by the monolithic and compliant nature of flexible circuits for SSE. In conventional systems with rigid electronics and non-flexible wiring, movement can cause stress concentrations at the skin-electrode interface. These stresses, when combined with conventional gel-based electrodes, cause significant skin-electrode impedance fluctuations, resulting in movement artifacts. With dangling wires, the effects of gravity compound these issues. The FMNE studied for SSE applications was observed to provide improved SNR by penetrating the most superficial skin layers, which consist of dry and dead skin cells. By penetrating these layers and placing the conductive portion of the electrode well inside the dermis, the system allows for smaller electrodes than conventional setups, significantly reducing impedance density while improving spatial resolution for MI detection. When directly compared to a representative Ag / AgCl gel electrode, the FMNE achieved superior SNR.

[0117] Methodology: This study used a soft bioelectronic system with multiple components, including a VR headset, dry needle electrodes (e.g., 102), stretchable interconnects (e.g., 109), and wireless flexible circuits (e.g., 110). In this study, the mechanical reliability of the various components was performed. This study also evaluated the performance of different electrodes for SSVEP stimulation setup.

[0118] The training setup involved subjects wearing a VR head-mounted display (HMD) with straps placed over the electrodes. Along with the VR HMD, subjects could wear soft electronics with dry needle electrodes (hairy sites) and wireless circuitry (neck) secured by a headband to record brain signals.

[0119] Data were sampled at 500Hz using a custom-built EEG system for multiple datasets. Once the VR headset was placed on the subject, the application was remotely controlled from a data acquisition Android device. To avoid bias, stimuli were presented in a grid simultaneously facing the subject (Xiao et al. 2021). The subject started each trial with eyes closed and recorded the alpha rhythm for 8 seconds. At the end of this period, a short beep was sounded and the subject opened their eyes to look at the stimulus. The subject turned to the next stimulus every 2 seconds, as indicated by a loud clicking noise. The stimulus briefly stopped blinking for 0.6 seconds to allow the subject to shift their gaze to the next target. This process continued until all stimuli were viewed, and the subject was prompted to close their eyes to restart the cycle. In shorter time frames (less than 2 seconds), only the first 0.8, 1.0, or 1.2 seconds of the stimulus were used to classify the data. Thus, the number of samples will be the same for each time frame and may be calculated as N=S×40, where S is the number of stimuli. It should be understood that the durations and numbers of samples described herein are intended as non-limiting examples only, and that in some embodiments, different durations may be used.

[0120] 8A shows the VR text speller developed and evaluated during the study. Photograph 802 shows a subject wearing a virtual reality head mounted display (VR HMD) and soft electronics 804 for BCI demonstration. Plot 806 shows exemplary measured EEG data from four EEG channels transferred via Bluetooth (BLE) communication to a central processor (Android) where the signals are processed and classified in real time.

[0121] The computer-rendered output (808) shows the text-speller interface generated within the VR environment by the exemplary system used in this study. The flow diagram (810) shows the operational flow of the Android software developed for the BCI demonstration that was used to generate the text-speller interface (808). The system used the Unreal Engine program (discussed further below) to render the text-speller software and stimulus overlays to the user. The software included the operation of a pass-through camera that allowed navigation of the real-world environment via an augmented reality perspective using a powered wheelchair. In this study, the system was implemented on two sets of hardware: a VR-HMD viewport (812) and an augmented reality viewport (814). In the augmented reality viewport, SSVEP commands can be used for navigation control (816).

[0122] The split-eye asymmetric stimuli (SEAS) platform was generated using a widely used cross-platform software development engine (Unreal Engine 4.26, Epic Games Inc.) targeting VR hardware (Oculus Quest2, Facebook). Using the "material" property that can be applied to 2D or 3D objects in the engine, the study created materials that look different depending on the size of the VR panel being rendered. Materials can be animated using "sprites," which are animated raster graphics, with successive frames arranged in n × n "sheets." These frames were extracted and rendered using Unreal Engine's built-in texture animation features. These materials were used to animate most 2D or 3D objects and flat surfaces in the engine environment. The first step was to generate sheets with associated frames based on the frame rate. Here, a program was devised in MATLAB to generate a sine waveform, convert the sine waveform into tiles with brightness based on the amplitude of the sine waveform, and then place these tiles into a 10x10 "sheet" for Unreal Engine's texture rendering system. Figures 8F and 8G contain the MATLAB code and specific instructions for Unreal Engine to generate a VR interface. In Figure 8G, an example of stimulus tile generation is shown with the waveform and corresponding tile layout in a 10x10 sprite.

[0123] Cross-platform software (Unreal Engine, Epic Games) was used to develop animated textures that appeared differently on the left and right hand sides of the VR panel. For the first set of tests (called "Set 1"), 10 standard stimuli ranging from 10.0 to 17.2 Hz were generated to determine the separability of the SSVEP stimuli. Table 2 shows the frequency and phase for the left and right eyes.

[0124] All stimuli for the VR interface were generated programmatically based on the required frequency and phase offset. The 32 stimuli and their frequencies used in this study are shown and described in relation to Figure 3C. Two frequencies were used for 16 of the 32 stimuli: a first frequency for the left eye and a second frequency for the right eye. [Table 2]

[0125] Another test set ("Set 2") included a left eye frequency range of 10.0 to 17.7 Hz and a right eye frequency range of 16.9 to 9.2 Hz, respectively, as shown in Table 3. [Table 3]

[0126] Due to the complexity of the stimuli under study and to maintain a high level of classification performance, machine learning was performed on a session-by-session basis in this study. An example illustration is provided with respect to Figure 4B. During training, data were bidirectionally filtered using a third-order high-pass Butterworth filter with a corner frequency of 2.0 Hz for a period ranging from 0.8 to 2 seconds. Data were linearly rescaled to the range [-0.5, 0.5] across the training dataset, so the resulting values ​​are in the range [0, 1]. This type of preprocessing can cause problems with traditional Ag / AgCl electrodes, as the signal amplitude changes over time, reducing electrode performance. However, the exemplary dry needle electrode of the soft electronics may achieve consistent performance with improved skin contact. This simplified preprocessing while improving classification performance.

[0127] For studies using conventional Ag / AgCl electrodes, each subject's skin was cleaned by gentle scrubbing with an alcohol wipe and dead skin cells were removed using an abrasive gel (NuPrep, Weaver, and Co.) to maintain a contact impedance of less than 10 kΩ for all electrodes. The abrasive gel was removed using an alcohol wipe and the surface was dried using a clean paper towel. For FMNE, the only skin preparation performed was gently scrubbing the electrode locations with an alcohol wipe. EEG data were recorded using a custom application running on an Android Tablet (Samsung Galaxy Tab S4) using Bluetooth Low Energy wireless communication.

[0128] Preliminary SSVEP performance. Figure 8B shows the results of a preliminary performance evaluation for different electrode locations. Prior to running the text-speller setup with 32 stimuli, preliminary SSVEP and SEAS testing was performed to test the feasibility of using stimuli within a VR environment. Plot 818 shows the average classification accuracy of the two SSVEP stimulus sets across multiple time windows (0.8-2 seconds). Plot 820 shows the average information transfer rate (ITR) of the two SSVEP stimulus sets from eight subjects (N=440 for Set 1 and N=360 for Set 2). For the first set of tests (labeled "Set 1", according to Table 2), ten standard stimuli from 10.0-17.2 Hz were generated to determine the separability of the SSVEP stimuli (details in Table S1). Another test set ("Set 2", from Table 3) included left eye frequency ranges of 10.0 to 17.7 Hz and right eye frequency ranges of 16.9 to 9.2 Hz, respectively.

[0129] The results for Set "1" show high accuracy over short samples. For example, in a basic cue-guided task, the eight subjects in Set 1 demonstrate an accuracy of 91.25 ± 1.40% with a window length of only 0.8 seconds. The results indicate a high throughput ITR of 206.7 ± 7.3 bits / min. With increased window lengths, the overall accuracy increases significantly to 93.88 ± 1.11% at 1.0 seconds, 95.03 ± 0.97% at 1.2 seconds, and 98.50 ± 0.34% at 2.0 seconds.

[0130] FIG. 8B also shows the results of an evaluation of two configurations of electrode locations (Configuration "A" 824, Configuration "B" 826). Plot 822 shows the results of comparing the classification accuracy between the two electrode placements. From the plot (822), it can be observed that Configuration "A" demonstrated stronger performance than Configuration "B" for the subjects evaluated. The error bars in the graph represent the standard error of the mean. In Configuration "A" 824, two channels are each biased towards one respective hemisphere. In Configuration "B" 826, all channels share a central reference.

[0131] Based on this study, and learning from the "Set 2" setup, a new stimulus setup, "Set 3," was developed using 32 unique frequency combinations to minimize collisions and prevent inaccuracies. Details of this stimulus setup are described in relation to Figure 3C.

[0132] The information transfer rate (ITR), measured in bits per minute, can be used to evaluate BCI performance. The ITR is calculated based on the number of targets, the average time required to relay a command, and the classification accuracy according to Equation 1.

number

[0133] where N is the number of targets, A is the accuracy, and w is the total time required to execute the command, including data acquisition time plus processing, sorting, and execution latency.

[0134] CNN classification performance. To train the CNN, the test data was segmented the first time the stimuli were presented. For each time window (0.8-1.2 s), only the first period was used and the rest were discarded. After segmentation, the data was preprocessed using a third-order Butterworth high-pass filter with a corner frequency of 2.0 Hz. The data was not preprocessed before being used in training and classification. For sets “1” and “2”, the number of samples N was subdivided into 10 groups for cross-fold validation. For set “3”, for faster setup time, the number of samples N was subdivided into 4 groups for 4-fold cross-validation. Classification was performed using a convolutional neural network (CNN) with spatial convolutions (Bevilacqua et al. 2014, Mahmood et al. 2021, Ravi et al. 2020, Waytowich et al. 2018). For the CNN, a batch size of 128 samples was used and training was performed for 100 epochs or stopped early if 10 epochs did not improve classification of the validation data. The trained network with the best performing classification accuracy was saved for use in real-time classification.

[0135] FIG. 8C shows the classification performance results of the CNN classifier used in this study. The CNN classifier used spatial CNN classification. In FIG. 8C, an overview 828 of the spatial CNN model is shown, including the different hidden layers of the model and their features extracted from 1 second segments of the 4-channel EEG signal. In this study, a stimulation setup with a left stimulation frequency of 8.2 Hz and a right stimulation frequency of 13.2 Hz was used.

[0136] The study was conducted in two sets of experiments, a commercial setup with wired Ag / AgCl electrodes (Norton et al. 2015) and an exemplary wireless soft electronic system. Both setups used 4 EEG channels and 33 classes. The conventional setup used wired standard electrode leads and Ag / AgCl electrodes interfaced with conductive paste to record EEG signals on the scalp. The exemplary soft electronic system used dry needle electrodes with a headband without conductive gel, e.g., as described in connection with Figure 2.

[0137] Plots 830 and 832 show the performance results of the two sets of experiments, respectively. Plot 830 shows the classification accuracy and plot 832 shows the average ITR for each of the two sets of experiments. Using 4-fold cross-validation, the commercial setup showed an accuracy of 74.72±3.03% from 0.8 seconds of data (ITR: 222.4±15.0 bits / min) (via plots 830 and 832, respectively). It was observed that longer time lengths provided the expected slightly higher accuracy. In contrast, the exemplary soft electronic system showed a substantial increase in classification accuracy and ITR at 78.93±2.36% and 243.6±12.5 bits / min, respectively (via plots 830 and 832, respectively). Overall, this study demonstrates the unique advantages of using a wireless soft platform with dry electrodes over traditional tethered systems with required skin preparation and wired electrodes.

[0138] Characterization of Stimulus Frequency and Phase Shift. FIG. 8D shows the results of an evaluation of the effects of stimulation frequency and phase shift. The study was performed using a conventional setup and an exemplary soft electronics setup. Plot 834 shows the left and right eye frequency responses corresponding to the visualized successive stimuli, and plot 836 shows the corresponding left and right eye phase offsets.

[0139] Plot 838 shows the confusion matrix generated from the soft electronics results for 33 classes of SEAS stimuli (for 9 subjects). Plot 840 shows the same results under the same experimental conditions for a conventional setup. For single frequency stimuli, it can be observed that most of the confusion is from adjacent frequencies. In contrast, dual frequency stimuli have various mixtures with both single frequency stimuli and other dual frequency stimuli. The results showed that stimuli from one eye or the other eye are processed in both hemispheres of the visual cortex. In addition, the present study also demonstrated that there is significant hemisphere-related asynchrony and mixtures on which classification can be performed. The results show one of the highest ITRs at a high level with only four EEG channels compared to previous studies.

[0140] 8E shows a table of comparative performance between the exemplary flexible electronics and previous studies. As shown in the table, the exemplary flexible electronics can achieve an ITR of 243.5 bits / min for 33 classifications using four electrode channels, with an accuracy approaching 80%.

[0141] Discussion of SSVEP. Locked-in syndrome (LIS) describes a state of complete paralysis except for blinking and eye movements (Padfield et al. 2019). Here, brain activity and cognitive functions are usually unaffected, resulting in a state of pseudocoma in which subjects are unable to move or communicate but are aware of their consciousness and environment. Despite normal cerebral cortical activity, subjects are unable to control motor functions, typically due to damage to the lower brain and brainstem. There are several causes of LIS in humans, including but not limited to stroke, traumatic brain injury or hemorrhage of the brainstem, poisoning, or drug overdose. Brain activity analysis is typically used to diagnose LIS using instruments such as electroencephalography (EEG) to observe the sleep-wake patterns of affected individuals. With the advent of BCIs, subjects can bypass the requirement of motor functions by controlling machines such as computers or prosthetic devices by monitoring brain activity. BCIs offer a promising solution for subjects with LIS or severe physical disabilities such as quadriplegia, restoring movement and communication to these individuals and improving their quality of life. EEG designs for BCIs have been trending towards wearables with wireless capabilities since the standardization of common wireless protocols such as Bluetooth (Lin et al. 2010). Dry electrodes provide superior and consistent long-term performance compared to gel-based electrodes (Norton et al. 2015, Salvo et al. 2012, Stauffer et al. 2018) if the skin preparation, amplifiers, shielding, and electrode configuration are appropriate (Li et al. 2017, Salvo et al. 2012). Lightweight sensors with minimal cabling also significantly reduce dragging or movement artifacts in poorly configured conventional EEG (Tallgren et al. 2005).

[0142] Using SSVEP, up to 40 unique stimuli with various frequencies and phase offsets can be discriminated with reasonable accuracy (Nakanishi et al. 2017). Empirical evidence suggests a significant hemispheric asymmetry in SSVEP signals (Martens and Hubner 2013, Wu 2016). Recent studies have demonstrated asymmetric high-frequency dual-stimulus SSVEPs, where two stimuli flashed in alternative phases were used, demonstrating more efficient SSVEP encoding (Yue et al. 2020). Due to the asymmetry of connections between the ocular sensory receptors and the visual cortex, it can be speculated that different stimuli seen simultaneously by each eye give measurably different brain activity in either hemisphere (Richard et al. 2018).

[0143] The embodiments of the present disclosure include a portable VR-enabled BCI using a soft bioelectronic system and a SEAS platform using SSVEP. Using VR, asynchronous SSVEP stimuli-different frequencies can be presented simultaneously to each eye. Overall, using novel stimuli with VR along with a soft wearable wireless device enables 33 classes of high-throughput SSVEP BCI with high accuracy and low control latency. Using only four channels, it was observed that an accuracy of 78.93±1.05% was achieved for 0.8 seconds of data for a peak information rate of 243.6±12.5 bits / min. In high accuracy mode, the device achieves 91.73±0.68% for 2 seconds of data with a throughput of 126.6±3.7 bits / min. This performance is demonstrated using a real-time text speller interface using a full keyboard type setup.

[0144] Although exemplary embodiments of the present disclosure have been described in detail herein with certain examples, it should be understood that other embodiments are contemplated. Accordingly, it is not intended that the scope of the present disclosure be limited to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced or carried out in various ways.

[0145] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" or "approximately" one particular value and / or to "about" or "approximately" another particular value. When such a range is expressed, other exemplary embodiments include from the one particular value and / or to the other particular value.

[0146] "Comprising" or "containing" or "including" means that at least the named compound, element, particle, or method step is present in a composition, article, or method, but does not exclude the presence of other compounds, materials, particles, method steps, even if other such compounds, materials, particles, method steps have the same function as the one named.

[0147] In describing the exemplary embodiments, terms will be used for clarity. Each term is intended to contemplate its broadest meaning as understood by those skilled in the art and to include all technical equivalents that operate in a similar manner to achieve a similar purpose. It should also be understood that the reference to one or more steps of a method does not preclude the presence of additional method steps or intervening method steps between those steps explicitly identified. The steps of the method may be performed in a different order than described herein without departing from the scope of the present disclosure. Similarly, it should also be understood that the reference to one or more components in a device or system does not preclude the presence of additional components or intervening components between those components explicitly identified.

[0148] As discussed herein, a "subject" may be any suitable human, animal, or other organism, living or dead, or other biological or molecular structure, or chemical environment, and may relate to a particular component of the subject, e.g., a particular tissue or fluid of the subject (e.g., human tissue within a particular region of the body of a living subject), referred to herein as an "area of ​​interest" or "region of interest."

[0149] As discussed herein, it is understood that the subject may be a human or any animal. It is understood that the animal may be of any suitable type, including, but not limited to, a mammal, a veterinary animal, a livestock animal, or a pet-type animal. By way of example, the animal may be a laboratory animal (e.g., rats, dogs, pigs, monkeys) specifically selected to have certain characteristics similar to humans, and the like. It is understood that the subject may be, for example, any suitable human patient.

[0150] As used herein, the term "about" means approximately, in the region of, roughly, or around. When the term "about" is used in conjunction with a numerical range, it modifies that range by extending the boundaries above and below the stated numerical values. In general, the term "about" is used herein to modify numerical values ​​above and below the stated value by a variance of 10%. In one embodiment, the term "about" means ±10% of the numerical value of the number with which the term is used. Thus, about 50% means within a range of 45% to 55%. Numerical ranges recited herein by endpoints include all numerical values ​​and fractions subsumed within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, 4.24, and 5).

[0151] Similarly, numerical ranges recited herein by endpoints include subranges subsumed within that range (e.g., 1 to 5 includes 1 to 1.5, 1.5 to 2, 2 to 2.75, 2.75 to 3, 3 to 3.90, 3.90 to 4, 4 to 4.24, 4.24 to 5, 2 to 5, 3 to 5, 1 to 4, and 2 to 4). It is also to be understood that all numbers and fractions thereof are presumed to be modified by the term "about."

[0152] The following patents, applications, and publications, listed below and throughout the specification, are hereby incorporated by reference in their entireties. [1] N. Padfield, J. Zabalza, H. Zhao, V. Masero, J. Ren, Sensors 2019,19,1423. [2] A. Venkatakrishnan, GE Francisco, JLContreras-Vidal, Current physical medicine and rehabilitation reports 2014,2,93. [3]M.Mahmood,D.Mzurikwao,Y.-S.Kim,Y.Lee,S.Mishra,R.Herbert,A.Duarte,C.S.Ang,W.-H.Yeo,Nature Machine Intelligence 2019,1,412. [4]Norton et al,“Soft,curved electrode systems capable of integration on the auricle as a persistent brain-computer interface,” Proceedings of the National Academy of Sciences 2015,112,3920. [5]Herbert et al,“Soft Material-Enabled,Flexible Hybrid Electronics for Medicine,Healthcare,and Human-Machine Interfaces,” Advanced Materials 2020,32,1901924; W.-H.Yeo,Y.Lee,Journal of Nature and Science 2015,1,e132. [6]Lin et al,“Review of Wireless and Wearable Electroencephalogram Systems and Brain-Computer Interfaces - A Mini-Review,” Gerontology 2010,56,112. [6’]Tian et al,“Large-area MRI-compatible epidermal electronic interfaces for prosthetic control and cognitive monitoring,” Nature Biomedical Engineering 2019,3,194 [6”]Li et al,“Recent Developments on Graphene-Based Electrochemical Sensors toward Nitrite,” J.Wu,Y.Xia,Y.Wu,Y.Tian,J.Liu,D.Chen,Q.He,Journal of neural engineering 2020,17,026001. [7]P.Salvo,R.Raedt,E.Carrette,D.Schaubroeck,J.Vanfleteren,L.Cardon,Sensors and Actuators A: Physical 2012,174,96. [8]G.Li,S.Wang,YYDuan,Sensors and Actuators B: Chemical 2017,241,1244. [9]F.Stauffer,M.Thielen,C.Sauter,S.Chardonnens,S.Bachmann,K.Tybrandt,C.Peters,C.Hierold,J.Voros,Adv Healthc Mater 2018,7,e1700994; MALopez-Gordo,D.Sanchez-Morillo,FPValle,Sensors 2014,14,12847.

[10] D.Gao,K.Parida,PSLee,Advanced Functional Materials 2020,30,1907184; H.Wu,G.Yang,K.Zhu,S.Liu,W.Guo,Z.Jiang,Z.Li,Advanced Science 2021,8,2001938.

[11] W.Guo,P.Zheng,X.Huang,H.Zhuo,Y.Wu,Z.Yin,Z.Li,H.Wu,ACS applied materials & interfaces 2019,11,8567.

[12] Z.Jiang,MOGNayeem,K.Fukuda,S.Ding,H.Jin,T.Yokota,D.Inoue,D.Hashizume,T.Someya,Advanced Materials 2019,31,1903446.

[13] R. Abiri, S. Borhani, EWSellers, Y. Jiang, X. Zhao.

[14] G.Pfurtscheller,C.Neuper,D.Flotzinger,M.Pregenzer,Electroencephalography and Clinical Neurophysiology 1997,103,642 [14']X.Tang,W.Li,X.Li,W.Ma,X.Dang,Expert Systems with Applications 2020,149,1

[15] R.Zhang,Q.Zong,L.Dou,X.Zhao,Journal of neural engineering 2019,16,066004.

[16] V.Jurcak,D.Tsuzuki,I.Dan,Neuroimaging 2007,34,1600.

[17] O.Olatunji,A.Denloye,Journal of Polymers and the Environment 2019,27,1252; A. Romgens, D. Bader, J. Bouwstra, F. Baaijens, C. Oomens, Journal of the Mechanical Behavior of Biomedical Materials 2014,40,397.

[18] S.Russo,T.Ranzani,H.Liu,S.Nefti-Meziani,K.Althoefer,A.Menciassi,Soft robotics 2015,2,146 [18']WHYeo, YSKim, J. Lee, A. Ameen, L. Shi, M. Li, S. Wang, R. Ma, SHJin, Z. Kang, Advanced materials 2013, 25,2773.

[19] H.Yang,S.Sakhavi,KKAng,C.Guan,Paper presented at the 2015 Proceedings of the IEEE Engineering in Medicine and Biology Society(EMBC) “On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification”,2015; S. Chaudhary, S. Taran, V. Bajaj, A. Sengur, IEEE Sensors Journal 2019,19,4494.

[20] Z.Tang,C.Li,S.Sun,Optics 2017,130,11.

[21] RTSchirrmeister,JTSpringenberg,LDJFiederer,M.Glasstetter,K.Eggensperger,M.Tangermann,F.Hutter,W.Burgard,T.Ball,Human Brain Mapping 2017,38,5391.

[22] P.Welch,IEEE Transactions on Audio and Electroacoustics 1967,15,70.

[23] Y.-T.Kwon,H.Kim,M.Mahmood,Y.-S.Kim,C.Demolder,W.-H.Yeo,ACS Appl.Mater.Interfaces 2020,12,49398; Y.-T.Kwon,JJNorton,A.Cutrone,H.-R.Lim,S.Kwon,JJChoi,HSKim,YCJang,JRWolpaw,W.-H.Yeo,Biosens.Bioelectron.2020,165,112404.

[24] M.Abadi,P.Barham,J.Chen,Z.Chen,A.Davis,J.Dean,M.Devin,S.Ghe mawat,G.Irving,M.Isard,Design of the 12-year{USENIX}Creation Platform(operating). systems design and implementation) published in Automatic Dictionary ({OSDI} 16) and “Tensorflow: A System for Large-Scale Machine Learning”,2016.

[25] YRTabar,U.Halici,Journal of Neural Engineering 2016,14,016003.

[26] N.Lu,T.Li,X.Ren,H.Miao,IEEE Transactions on Neural Systems and Rehabilitation Engineering 2016,25,566.

[27] Z. Shi, F. Zheng, Z. Zhou, M. Li, Z. Fan, H. Ye, S. Zhang, T. Xiao, L. Chen, THTao, Y.-L. Sun, Y. Mao, Advanced Science 2019,6,1801617.

[28] Bevilacqua, V., Tattoli, G., Buongiorno, D., Loconsole, C., Leonardis, D., Barsotti, M., Frisoli, A., Bergamasco, M., 2014. A novel BCI-SSVEP based approach for control of walking in virtual environment using a convolutional neural network (IJCNN),pp.4121-4128.IEEE.

[29] Chen,X.,Wang,Y.,Nakanishi,M.,Gao,X.,Jung,T.P.,Gao,S.,2015.High-speed spelling with a noninvasive brain-computer interface.Proc Natl Acad Sci U S A 112(44),E6058-6067.

[30] Kwak,N.S.,Muller,K.R.,Lee,S.W.,2017.A convolutional neural network for steady state visual evoked potential classification under ambulatory environment.PLoS One 12(2),e0172578.

[31] Li,G.,Wang,S.,Duan,Y.Y.,2017.Towards gel-free electrodes: A systematic study of electrode-skin impedance.Sensors and Actuators B: Chemical 241,1244-1255.

[32] Lin,C.T.,Ko,L.W.,Chang,M.H.,Duann,J.R.,Chen,J.Y.,Su,T.P.,Jung,T.P.,2010.Review of wireless and wearable electroencephalogram systems and brain-computer interfaces--a mini-review.Gerontology 56(1),112-119.

[33] Mahmood,M.,Kwon,S.,Kim,Y.-S.,Siriaraya,P.,Choi,J.,Boris,O.,Kang,K.,Jun Yu,K.,Jang,Y.C.,Ang,C.S.,2021.Wireless Soft Scalp Electronics and Virtual Reality System for Motor Imagery-based Brain-Machine Interfaces.Advanced Science.

[34] Mahmood,M.,Mzurikwao,D.,Kim,Y.-S.,Lee,Y.,Mishra,S.,Herbert,R.,Duarte,A.,Ang,C.S.,Yeo,W.-H.,2019.Fully portable and wireless universal brain-machine interfaces enabled by flexible scalp electronics and deep learning algorithm.Nature Machine Intelligence 1(9),412-422.

[35] Martens,U.,Hubner,R.,2013.Functional hemispheric asymmetries of global / local processing mirrored by the steady-state visual evoked potential.Brain and cognition 81(2),161-166.

[36] Nakanishi,M.,Wang,Y.,Chen,X.,Wang,Y.-T.,Gao,X.,Jung,T.-P.,2017.Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis.IEEE Transactions on Biomedical Engineering 65(1),104-112.

[37] Norton,J.J.S.,Lee,D.S.,Lee,J.W.,Lee,W.,Kwon,O.,Won,P.,Jung,S.-Y.,Cheng,H.,Jeong,J.-W.,Akce,A.,Umunna,S.,Na,I.,Kwon,Y.H.,Wang,X.-Q.,Liu,Z.,Paik,U.,Huang,Y.,Bretl,T.,Yeo,W.-H.,Rogers,J.A.,2015.Soft,curved electrode systems capable of integration on the auricle as a persistent brain-computer interface.Proceedings of the National Academy of Sciences 112(13),3920-3925.

[38] Padfield,N.,Zabalza,J.,Zhao,H.,Masero,V.,Ren,J.,2019.EEG-based brain-computer interfaces using motor-imagery: Techniques and challenges.Sensors 19(6),1423.

[39] Ravi,A.,Beni,N.H.,Manuel,J.,Jiang,N.,2020.Comparing user-dependent and user-independent training of CNN for SSVEP BCI.Journal of neural engineering 17(2),026028.

[40] Richard,B.,Chadnova,E.,Baker,D.H.,2018.Binocular vision adaptively suppresses delayed monocular signals.NeuroImage 172,753-765.

[41] Rodeheaver,N.,Herbert,R.,Kim,Y.S.,Mahmood,M.,Kim,H.,Jeong,J.W.,Yeo,W.H.,2021.Strain-Isolating Materials and Interfacial Physics for Soft Wearable Bioelectronics and Wireless,Motion Artifact-Controlled Health Monitoring.Advanced Functional Materials 31(36),2104070.

[42] Rodeheaver,N.,Kim,H.,Herbert,R.,Seo,H.,Yeo,W.-H.,2022.Breathable,Wireless,Thin-Film Wearable Biopatch Using Noise-Reduction Mechanisms.ACS Applied Electronic Materials.

[43] Salvo,P.,Raedt,R.,Carrette,E.,Schaubroeck,D.,Vanfleteren,J.,Cardon,L.,2012.A 3D printed dry electrode for ECG / EEG recording.Sensors and Actuators A: Physical 174,96-102.

[44] Stauffer,F.,Thielen,M.,Sauter,C.,Chardonnens,S.,Bachmann,S.,Tybrandt,K.,Peters,C.,Hierold,C.,Voros,J.,2018.Skin Conformal Polymer Electrodes for Clinical ECG and EEG Recordings.Adv Healthc Mater 7(7),e1700994.

[45] Tallgren,P.,Vanhatalo,S.,Kaila,K.,Voipio,J.,2005.Evaluation of commercially available electrodes and gels for recording of slow EEG potentials.Clin Neurophysiol 116(4),799-806.

[46] Volosyak,I.,2011.SSVEP-based Bremen-BCI interface--boosting information transfer rates.J Neural Eng 8(3),036020.

[47] Wang,Y.,Wang,Y.T.,Jung,T.P.,2010.Visual stimulus design for high-rate SSVEP BCI.Electronics Letters 46(15).

[48] Waytowich,N.,Lawhern,V.J.,Garcia,J.O.,Cummings,J.,Faller,J.,Sajda,P.,Vettel,J.M.,2018.Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials.Journal of neural engineering 15(6),066031.

[49] Wu,Z.,2016.Physical connections between different SSVEP neural networks.Scientific reports 6(1),1-9.

[50] Xiao,C.,Chiang,K.-J.,Nakanishi,M.,Jung,T.-P.,2021.A Comparison Study of Single-and Multiple-Target Stimulation Methods for Eliciting Steady-State Visual Evoked Potentials.2021 10th International IEEE / EMBS Conference on Neural Engineering (NER),pp.698-701.IEEE.

[51] Xing,X.,Wang,Y.,Pei,W.,Guo,X.,Liu,Z.,Wang,F.,Ming,G.,Zhao,H.,Gui,Q.,Chen,H.,2018.A high-speed SSVEP-based BCI using dry EEG electrodes.Scientific reports 8(1),1-10.

[52] Yue,L.,Xiao,X.,Xu,M.,Chen,L.,Wang,Y.,Jung,T.-P.,Ming,D.,2020.A brain-computer interface based on high-frequency steady-state asymmetric visual evoked potentials.2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC),pp.3090-3093.IEEE.

[53] Zavanelli,N.,Kim,H.,Kim,J.,Herbert,R.,Mahmood,M.,Kim,Y.-S.,Kwon,S.,Bolus,N.B.,Torstrick,F.B.,Lee,C.S.,2021.At-home wireless monitoring of acute hemodynamic disturbances to detect sleep apnea and sleep stages via a soft sternal patch.Science advances 7(52),eabl4146.

Claims

1. A system comprising: A set of thin EEG sensors, each comprising an array of flexible epidermal penetrating micro-needle electrodes fabricated on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit; A brain machine interface operably connected to the set of thin EEG sensors; Wherein the brain machine interface comprises: A processor; A memory operably connected to the processor, the memory having instructions stored thereon, execution of the instructions by the processor causing the processor to: Receive an EEG signal acquired from the thin EEG sensor; Continuously classify brain signals as control signals from the acquired EEG signal via a trained neural network; Output the control signal to a virtual reality environment controller to actuate commands within a VR scene generated by the virtual reality environment controller viewed by a subject; A system that causes the above to be performed.

2. The system of claim 1, wherein the commands cause a set of limb movements within the VR scene and the trained neural network is configured to classify the brain signals for the set of movements.

3. The system of claim 1 or 2, wherein the set of thin EEG sensors is connected to the brain machine interface via a set of stretchable flexible connectors.

4. The system of claim 1, wherein the micro-needle electrodes have an enlarged contact surface area and a reduced electrode impedance density.

5. The system of claim 1, further comprising a wearable soft headgear comprising a low elastic modulus elastomer band.

6. The system of claim 1, wherein the trained neural network includes a spatial convolutional neural network.

7. The system of claim 1, wherein the set of thin EEG sensors is placed along the scalp for motion imaging.

8. The system of claim 1, wherein the set of thin EEG sensors is placed along the scalp for steady state visual evoked potential (SSVEP) measurement.

9. The system according to claim 8, wherein the virtual reality environment controller is configured to generate split-eye asynchronous stimuli (SEAS) in a virtual scene for a real-time text spelling interface.

10. The execution of the instructions by the processor causes the processor to transmit the acquired EEG signal to a remote or cloud computing device that performs a retraining operation of the trained neural network, receive an updated trained neural network from the remote or cloud computing device during a runtime operation of the virtual reality environment controller, and further perform the above steps. The system according to claim 1.

11. Each of the plurality of the flexible epidermis-penetrating micro-needle electrodes of the array has a height of at least 500 μm (e.g., 800 μm) so as to be mounted on the hairy scalp, has a base width of about 350 μm, and has an area of about 36 mm ２ The system according to claim 1, having an area of.

12. A method comprising: providing a set of thin-film EEG sensors disposed on a user's scalp, each of the set of thin-film EEG sensors comprising an array of flexible epidural penetrating micro-needle electrodes fabricated on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit; receiving, by a processor or a brain machine interface operably connected to the set of thin-film EEG sensors, an EEG signal acquired from the set of thin-film EEG sensors; continuously classifying, by the processor, a brain signal as a control signal from the acquired EEG signal through a trained neural network; and outputting, by the processor, the control signal to a virtual reality environment controller to actuate a command in a VR scene generated by the virtual reality environment controller visually recognized by a subject.

13. The method according to claim 12, wherein the set of thin-film EEG sensors is directly disposed on the scalp without a conductive gel or paste.

14. The method according to claim 12 or 13, wherein the set of thin-film EEG sensors includes i) a reference array of flexible epidural penetrating micro-needle electrodes disposed at vertex positions on the scalp, and ii) six arrays of flexible epidural penetrating micro-needle electrodes removably attached to low elastic modulus elastomer bands at a first front position, a second back position, and four side positions for kinematic imaging measurements.

15. The set of thin EEG sensors includes: i) a reference array of flexible epidermis-penetrating micro-needle electrodes installed at the back position on the scalp, and ii) four arrays of flexible epidermis-penetrating micro-needle electrodes removably attached to a low elastic modulus elastomer band in the back region of the scalp for steady-state visual evoked potential (SSVEP) measurement. The method according to claim 12 or 13.

16. Transmitting, by the processor, the acquired EEG signal to a remote or cloud computing device that performs a retraining operation of the trained neural network. Receiving, by the processor, an updated trained neural network from the remote or cloud computing device during the runtime operation of the virtual reality environment controller. The method according to claim 12, further comprising.

17. A non-transitory computer-readable medium storing instructions, wherein the execution of the instructions by a processor of a brain-machine interface controller causes the processor to Receive an EEG signal obtained from a set of thin EEG sensors installed on a user's scalp, wherein each of the sets of thin EEG sensors is manufactured on a flexible circuit board operably connected to an analog-to-digital converter circuit operably connected to a wireless interface circuit and includes an array of flexible epidermis-penetrating micro-needle electrodes, and the set of thin EEG sensors is installed directly on the scalp without a conductive gel or paste. Continuously classify brain signals as control signals from the acquired EEG signals via a trained neural network. Output the control signal to a virtual reality environment controller to activate commands in a VR scene generated by the virtual reality environment controller visually recognized by a subject. A computer-readable medium that causes the above to be performed.

18. The set of thin EEG sensors includes: i) a reference array of flexible epidermis-penetrating micro-needle electrodes installed at the vertex position on the scalp; and ii) six arrays of flexible epidermis-penetrating micro-needle electrodes removably attached to a low elastic modulus elastomer band at a first front position, a second back position, and four side positions for motion image measurement. The computer-readable medium according to claim 17.

19. The set of thin EEG sensors includes: i) a reference array of flexible epidermis-penetrating micro-needle electrodes installed at the back position on the scalp; and ii) four arrays of flexible epidermis-penetrating micro-needle electrodes removably attached to a low elastic modulus elastomer band in the back region of the scalp for steady-state visual evoked potential (SSVEP) measurement. The computer-readable medium according to claim 17.

20. The execution of the instructions causes the processor to transmit the acquired EEG signal to a remote or cloud computing device that performs a retraining operation of the trained neural network; receive an updated trained neural network from the remote or cloud computing device during the runtime operation of the virtual reality environment controller; The computer-readable medium according to claim 17, which further causes the above to be performed.

Images