Asynchronous Brain-Computer Interface in AR Using Steady-State Visually Evoked Potentials

Abstract

A method and system using steady state motor visual evoked potential stimulation in an augmented reality environment is disclosed. Requested stimulus data is received from a user application on a smart device. Sensor data and other context data are also received, where the other context data includes non-sensed data. Based on the sensor data and other context data, the requested stimulus data is converted into a modified stimulus. The modified stimulus and the environmental stimulus are presented to a user using a rendering device configured to blend the modified stimulus and the environmental stimulus, thereby resulting in a rendered stimulus. A biosignal generated in response to the rendered stimulus is received from a user on a wearable biosignal sensing device. Based on the modified stimulus, the received biosignal is classified, resulting in a classified selection, which is returned to the user application.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 170,987, filed on April 5, 2021, the content of which is incorporated herein by reference in its entirety.

Background Art

[0002] Brain - wave - based brain - computer interfaces (BCIs) enable a human to establish a direct communication pathway between the brain and the external environment, bypassing the peripheral nerves and muscles. Steady - state visually evoked potential (SSVEP) - based BCIs are dependent or reactive BCIs that detect electroencephalogram (EEG) responses to repetitive visual stimuli with distinguishable characteristics (e.g., different frequencies). A BCI can determine which stimulus occupies the user's visual attention by detecting the SSVEP response at the targeted stimulus frequency from the EEG recorded at the occipital cortex and the parieto - occipital cortex. This can appear as a significant peak at the targeted stimulus frequency and potentially at its higher or lower harmonics.

[0003] Traditionally, most of these stimuli are presented on a computer screen. However, augmented reality (AR) and virtual reality (VR) devices can allow users to view repetitive visual stimuli and the external environment within the same field of view, providing an improved user experience. Some studies have investigated AR approaches based on the use of video see-through (VST)-based head-mounted displays (HMDs) combined with SSVEP. These studies have applied AR-BCI for applications such as gaming, navigation in 3D space, and quadcopter control. The real-world scene is acquired using a camera placed on top of the HMD and displayed within the VR environment.

[0004] SSVEP stimuli are most commonly designed as monochromatic objects with an intensity modulated at a fixed frequency. As a result, they appear to the user as objects that are flickering. This flickering stimulus can induce visual fatigue and discomfort. Consequently, this reduces the overall signal-to-noise ratio (SNR), decoding performance, and interactivity of the BCI. Furthermore, the presentation of these stimuli on a computer screen or other opaque media in conventional configurations limits the application scenarios of the system in potential augmented reality AR applications. In the real-world environment, users may need to shift their visual attention back and forth between the stimulus presentation on the monitor and the normal field of view, which may include numerous distracting or confusing visual stimuli, further affecting the SNR in the resulting EEG readings and potentially reducing the accuracy of BCI determination of the viewer's gaze and attention. VST-based HMDs provide this ability but offer a limited field of view that is highly constrained by the camera.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Accordingly, while it is possible to test, train, and implement visual stimuli that perform the function of SSVEP stimuli when inducing BCI responses, there is a need for methods and systems that reduce visual fatigue and discomfort of viewers and improve the signal-to-noise ratio, decoding performance, and interactivity of BCI for use in VR / AR applications. In addition, it is necessary to test, train, and implement these stimuli in an environment and device that more closely resembles those of real-world user AR / VR applications. Furthermore, there is a need for a system that can test, train, and implement other stimuli such as auditory or somatosensory evoked potential stimuli that are also known to generate classifiable EEG signals.

Means for Solving the Problem

[0006] In one aspect, the method includes receiving, from a user application on a smart device, one or more requested stimulus data; receiving at least one of sensor data and other context data, where the other context data includes data that is not sensed; converting at least a portion of the requested stimulus data into a modified stimulus based at least in part on the sensor data and the at least one of the other context data; presenting the modified stimulus and an environmental stimulus to a user using a rendering device configured to mix the modified stimulus and the environmental stimulus to result in a rendered stimulus; receiving, from the user, a biosignal generated in response to the rendered stimulus on a wearable biosignal sensing device; classifying the received biosignal using a classifier based on the modified stimulus to result in a classified selection; and returning the classified selection to the user application.

[0007] In one aspect, the system includes a smart device, a rendering device, a wearable biosignal sensing device on the user, a processor, and a memory storing instructions that, when executed by the processor, configure the system to perform the above method.

[0008] In one aspect, the method includes receiving one or more requested stimulus data from a user application on a smart device. The method also includes receiving at least one of sensor data and other context data, where the other context data includes data that is not sensed. The method then includes converting at least a portion of the requested stimulus data into a modified stimulus based at least in part on at least one of the sensor data and the other context data, the modified stimulus including steady state visually evoked potential stimuli as well as other evoked potentials. The method includes presenting the modified stimulus and the environmental stimulus to the user using a rendering device configured to mix the modified stimulus and the environmental stimulus to thereby provide a rendered stimulus, which includes using at least one of a visual device, a tactile device, and an auditory device perceived by the user, and rendering the modified stimulus and the environmental stimulus on an augmented reality optical see-through (AR-OST) device associated with the smart device. The method then includes receiving a biosignal from the user generated in response to the rendered stimulus on a wearable biosignal sensing device. The method further includes determining whether to send the biosignal to a classifier by using at least one of the presence or absence of an intentional control signal, the determination of the presence of the intentional control signal including detecting a manual intent override signal from the smart device and determining at least in part from the received biosignal that the user intends to fixate on at least one of the rendered stimuli. On the condition that the intentional control signal is present, the method includes sending the received biosignal to the classifier. On the condition that the intentional control signal is absent, the method includes continuing to receive the received biosignal from the user.The method then includes using the classifier to classify the received biosignals based on the modified stimuli, resulting in a classified selection. The method finally includes returning the classified selection to the user application. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For ease of identifying discussion of any particular element or step, the leading digit(s) in the reference numbers refer to the figure number in which that element is first introduced.

[0010]

Figure 1

[0011]

Figure 2

[0012]

Figure 3

[0013]

Figure 4

[0014]

Figure 5

[0015]

Figure 6

[0016]

Figure 7

[0017]

Figure 8A

Figure 8B

Figure 8C

[0018]

Figure 9

[0019]

Figure 10

[0020] Figures 11A - 11H show the average magnitude spectra of SSVEP and SSMVEP stimulus responses under non - active and active background conditions.

[0021]

Figure 11A

[0022]

Figure 11B

[0023]

Figure 11C

[0024]

Figure 11D

[0025]

Figure 11E

[0026]

Figure 11F

[0027]

Figure 11G

[0028]

Figure 11H

[0029]

Figure 12

[0030]

Figure 13

[0031]

Figure 14

[0032]

Figure 15

[0033]

Figure 16

Best Mode for Carrying Out the Invention

[0034] Steady-state motion visual evoked potential (SSMVEP)-based BCIs address the drawbacks of SSVEP BCI use, including BCI fatigue, visual discomfort, and relatively low communication performance. In contrast to the on-off style of SSVEP stimuli, SSMVEP stimuli are designed to evoke visually perceived motion. In one embodiment, this design can include an equal-luminance black-and-white radial checkerboard that can be modulated at a fixed frequency. Specifically, this motion pattern can include radial contraction and expansion of the stimulus. SSMVEP BCIs include the advantages of SSVEP BCIs, such as high SNR, high information transfer rate (ITR), and low participant training time, while minimizing SSVEP-related discomfort for the operator. SSMVEP stimuli can be presented via an AR / VR HMD. A portable EEG system can be used with or incorporated within the AR HMD, making AR / VR-based BCIs a promising approach for implementing BCIs outside of the laboratory environment and enabling practical real-world applications. Optic-see-through (OST)-based HMDs can provide improvements over conventional video-see-through (VST) HMDs that are used to provide an overlay of stimuli over real-world environmental visual stimuli. OST AR devices can include a semitransparent screen or optical element. Virtual content, such as generated stimuli, can be directly displayed on the screen and overlaid on the user's normal field of view and the environmental stimuli present within that field of view. In one embodiment, a novel AR-OST-based BCI system can further address the challenges of current SSVEP systems. An AR-OST-based BCI system can use a 4-target SSMVEP BCI as disclosed herein.

[0035] The user can interact with the BCI in an asynchronous manner at any time when desired. This means that the BCI does not have to rely on precise stimulus timing or a predefined time frame. Compared with BCI paced by cues or synchronous, asynchronous operation can involve continuous decoding and analysis of responses. This operation is more technically demanding but can provide a more natural form of interaction. During asynchronous interaction, the operation may be related to two states, namely, the intentional control (IC) state and the no control (NC) state. The "IC state" in the present disclosure refers to the time when the user is detected, determined, or assumed to be gazing at the generated stimulus. The "NC state" in the present disclosure refers to the resting state or the time when the user is detected, determined, or assumed not to be gazing at the generated stimulus. A convolutional neural network (CNN)-based method may be used in the asynchronous classification of SSMVEP BCI and can function with improved accuracy and efficiency compared to more traditional classification algorithms. The offline decoding performance of the BCI system can be evaluated using Complex Spectrum CNN (C-CNN) for data processed in an asynchronous manner.

[0036] Figure 1 shows a process for using an SSMVEP BCI 100 according to an embodiment. The process for using the SSMVEP BCI 100 begins in block 106 by providing a stimulus 102 generated by the SSMVEP to the AR - OST. "Generated stimulus" in the present disclosure refers to a pre - defined or dynamically determined perceptible entity presented to the BCI user by the operation of a computer, an AR / VR headset, or other similar device. In one application, the generated stimulus may be an icon - type digital graphic projected into the user's field of view. The user's perception of such a stimulus can result in changes in the user's EEG pattern that are detected and analyzed by the BCI. In some applications, the EEG pattern changes induced by the user's perception of the generated stimulus can be interpreted as indicating a selection, decision, or action intended by the user.

[0037] In some embodiments, the user may be isolated from one or more types of environmental stimuli. This may be for experimental purposes when determining the robustness of the BCI against interference from environmental stimuli to provide a baseline reading for comparison. In another embodiment, the environmental stimuli may be blocked for reasons specific to its use case. In another embodiment, in addition to the stimuli 102 generated by SSMVEP, in block 106, environmental stimuli 104 may be provided to, or through, the AR - OST. "Environmental stimuli" in the present disclosure refers to visual, auditory, tactile, olfactory, or other stimuli that are external to the BCI user and perceptible in the environment surrounding the BCI user. The user's perception of such stimuli can result in changes in the user's EEG patterns that are detected and analyzed by the BCI. Since environmental stimuli 104 may affect the user's EEG patterns and such effects may compete or conflict with the effects from the stimuli 102 generated by SSMVEP, environmental stimuli 104 can be considered a confounder that causes interference and may reduce the functionality of the BCI in interpreting the user's interaction with the SSMVEP - generated stimuli 102.

[0038] Thus, in some embodiments, environmental stimuli 104 can be used to modulate the SSMVEP - generated stimuli 102 presented to the AR - OST in block 106. For example, if a flickering light is detected in the user's environment and the light is flickering at a frequency close to the frequency of the display alternation associated with the default generated stimuli, the presented SSMVEP - generated stimuli 102 may be presented with some other alternation frequency so that the BCI is not confused in its frequency - based interpretation of the detected brain activity of the user. Other environmental data such as time, location, etc. may be used to modulate the presented SSMVEP - generated stimuli 102 in order to improve the reliability of the BCI performance in interpreting the user's intent from the interaction between the SSMVEP - generated stimuli 102 and some of the environmental stimuli 104.

[0039] In block 106, the AR - OST can present the SSMVEP generation stimulus 102 and the environmental stimulus 104 to the user. The AR - OST may be a computerized system that can pass the environmental stimulus while programmatically generating and presenting new stimuli simultaneously. In certain embodiments, this may be an AR headset with lenses that allows the user to visually perceive the surrounding environment and thus provides the environmental stimulus 104 to the user. The lenses can, in certain embodiments, include a transparent organic light - emitting device (TOLED) that can emit light to produce the generated stimulus for the user. In another embodiment, the lenses may be reflective on the inner surface to the extent that they do not impede the visibility of the environment to the wearer but can reflect an image projected from a device within the visor that is outside the wearer's field of view but behind the lenses. In such an embodiment, the visor may have a gap or slit for the insertion of a smart device such as a smartphone. The SSMVEP generation stimulus 102 can be generated by the display of a smartphone and reflected by the lenses of the AR - OST and returned to the user.

[0040] The SSMVEP generation stimulus 102 and the environmental stimulus 104 presented on or through the AR - OST in block 106 may be transmitted to the user's physiology in block 108. The "user physiology" in the present disclosure refers to the user's sensory and perceptual organs and typically includes the intervening tissues, peripheral nerves, and central nervous system. For light stimuli, this includes the eye tissues, retina, and occipital lobe. The user physiology can be the pathway through which the SSMVEP generation stimulus 102 and the environmental stimulus 104 reach and affect the user's brain as neuroelectrical signals. The neuroelectrical signals can be detected by sensors such as EEG contacts.

[0041] EEG signals can be detected by these EEG contacts at block 108. One or more EEG signals can be obtained in this way through non-invasive contact with the user's skull. In certain embodiments, these signals may be obtained invasively. In another embodiment, EEG signals may be obtained using a combination of invasive and non-invasive sensors.

[0042] At decision block 112, a BCI that monitors and analyzes the EEG signals can determine whether the detected EEG signals indicate that the user is participating in the IC state or the NC state. If no intentional control is discriminated or detected, the BCI can return to monitoring the EEG signals at block 110. If the BCI discriminates or detects the user's intentional control, the process for using the SSMVEP BCI 100 can proceed to the classifier at block 114. In certain embodiments, the IC state may be determined to occur when a stimulus is presented, and the NC state may be determined to occur when no stimulus is presented. In another embodiment, eye tracking performed by a camera incorporated in the AR / VR headset or an additional sensor or other configuration may be used to detect that the user's focus is on a presented stimulus, and this may be used to indicate that the user is interacting in the IC state. In another embodiment, a transition between explicit and implicit perception of a stimulus can be detected in the EEG data and may be used to determine eye focus and intent. This step may be determined by processing external to the BCI in certain embodiments. No training is required to incorporate this determination or detection into the BCI analysis. In another embodiment, intent detection may be determined by heuristics programmed into the device or retrieved from the cloud. In another embodiment, intent detection may be performed in a classifier that can be part of a neural network specifically trained for intent detection.

[0043] In block 114, a classifier may be called to characterize the detected EEG signal. The classifier can evaluate whether the EEG signal corresponds to one or more of the SSMVEP generation stimuli 102. An array of values may be generated within a range of 0 to 1 representing the probability of the presence of the stimulus. In some embodiments, a softmax algorithm or a selection process may select a single element from the classifier output. If the stimulus characteristics are not detected in decision block 116, the process for using the SSMVEP BCI 100 can return to block 114. In some embodiments, the generated array of values may be provided for further processing without a specific classification decision.

[0044] If the stimulus characteristics are detected in decision block 116, the process for using the SSMVEP BCI 100 may proceed to perform further processing (block 118). The further processing may, in some embodiments, include refining the classifier algorithm and updating the classifier in block 114 using these refinements to improve the performance of the BCI.

[0045] In one embodiment, the spatial and temporal frequencies of the movements for the SSMVEP generation stimuli 102 can be modulated based on the distribution of the spatial and temporal confounding factors perceived from the environmental stimuli 104. In this way, the SSMVEP generator can maximize robustness regardless of the ambient stimuli. In another embodiment, a set of the SSMVEP generation stimuli 102 can be analyzed to extract their dominant visual frequencies. These frequencies can then be used during EEG signal analysis to classify the user's attention (e.g., IC state or NC state). In another embodiment, a set of the SSMVEP generation stimuli 102 can be analyzed to calculate their principal spatial frequencies. During visual presentation, the animation rate can be modified to increase or decrease the stimulus update so that a specific temporal frequency is generated. These frequencies may be selected to be maximally different from the ambient frequencies and / or the frequencies (spatial or temporal) of the environmental stimuli 104 or other simultaneously presented stimuli. In another embodiment, the temporal frequency can be modulated using an orthogonal coding scheme, a pseudo-random sequence, or other deterministic generation modulation.

[0046] A and B of FIG. 2 show exemplary SSVEP patterns and variations 200. Four SSVEP stimuli may be arranged in a basic SSVEP pattern 202 as shown in A of FIG. 2. Emojis may be used as the flickering SSVEP stimuli. In one embodiment, this may be a circular emoji as shown, although emojis of any shape known to those skilled in the art may be used. This type of SSVEP stimulus can provide a more realistic and appealing stimulus compared to conventional single-color, colored, circular stimuli.

[0047] Each of the SSVEP stimuli within the pattern may correspond to a different frequency of change. For example, one stimulus may transition between an on state and an off state, as shown by the SSVEP image on 204 and the SSVEP image off 206 in FIG. 2B. Flicker (on / off) frequencies of 8 Hz, 10 Hz, 12 Hz, and 15 Hz may each be applied to one of the SSVEP stimuli within the basic SSVEP pattern 202.

[0048] FIGS. 3A - D show an exemplary SSMVEP pattern and variations 300 according to an embodiment. The basic SSMVEP pattern 302 as shown in FIG. 3A may include the same 4 - stimulus array as that which may be used for the basic SSVEP pattern 202 of FIG. 2A. However, a radial checkerboard image may be used as the SSMVEP stimulus. The SSMVEP stimulus is not intended to turn on, off, blink, or flicker like the SSVEP stimuli discussed above and can incorporate repetitive motion. In one embodiment, these repetitive motions may occur at the same frequencies of 8 Hz, 10 Hz, 12 Hz, and 15 Hz as described with respect to FIGS. 2A and 2B. In another embodiment, a plurality of distinct stimuli may be rendered to the user. These different stimuli can include any combination of SSVEP, radial checkerboard SSMVEP, or animated SSMVEP (described below).

[0049] The repetitive motion of the SSMVEP stimulus can take several forms. In one embodiment, the basic SSMVEP image 304 can transition to, and return from, an SSMVEP image having a different pattern density 306 at a desired frequency, as shown in FIG. 3B. In one embodiment, the basic SSMVEP image 304 can transition to, and return from, an SSMVEP image having a different size 308 at a desired frequency, as shown in FIG. 3C. In one embodiment, the basic SSMVEP image 304 can transition to, and return from, a rotated SSMVEP image 310 at a desired frequency, as shown in FIG. 3D. Other modifications to the images representing the motion may be contemplated by those skilled in the art.

[0050] In another embodiment, the SSMVEP stimulus can be based on a repetitive animated graphic sequence, such as a sticker or emoji that includes one or more still images. The SSMVEP stimulus may be generated from the animated sequence by varying the playback speed to produce specific spatial and / or temporal stimulus frequencies that can be sensed by the BCI.

[0051] Figures 4A and 4B show binocular projections for the SSVEP BCI interaction 400. Such projections may be presented to a user wearing a VST HMD or a novel AR-OST incorporated into the solutions disclosed herein. The visual stimuli within the AR headset may be calibrated for an individual user and may align the intended center point of the display with the center of the user's field of view.

[0052] In one embodiment, as shown in FIG. 4A, the aforementioned basic SSVEP pattern 202 may be overlaid and presented on an inactive background 402, such as a plain black background.

[0053] In another embodiment, as shown in FIG. 4B, the basic SSVEP pattern 202 may be overlaid and presented on an active background 404. For example, a stereo video of an urban scene played at 30 frames per second may be used. The video may show a first-person view, and the camera may be mounted in front of a vehicle navigating through a typical busy North American city street.

[0054] The four SSVEP stimuli of the basic SSVEP pattern 202 may be superimposed on the stereo video and presented in the foreground while the video is played continuously in the background. The video may also show a first-person view of navigating right and left and stopping at traffic signals, may include background sounds, and may include viewpoint movement or pauses at different points within the video.

[0055] A and B of FIG. 5 show binocular projections for an SSMVEP BCI interaction 500 according to an embodiment. The four SSMVEP stimuli of the basic SSMVEP pattern 302 may be superimposed on an inactive background 402 as shown in FIG. 5A and an active background 404 as shown in FIG. 5B, similar to the configuration described with respect to FIGS. 4A and 4B.

[0056] FIG. 6 shows an AR-OST BCI configuration 600 according to an embodiment. In this configuration, a user 602 may wear an AR-OST 604 comprising a lightweight optically see-through AR-OST shield 606 that may be partially transparent and, in some embodiments, may be partially reflective. The AR-OST shield 606 may be held within a frame 608, which also includes a smart device slot 610 into which a smart device 612 is inserted. In this way, an image generated on the screen of the smart device 612 may reflect from the inner surface of the AR-OST shield 606 and be visible to the user 602. In another embodiment, the AR-OST shield 606 may incorporate an active rendering mechanism such as a transparent organic light emitting device (TOLED) that can mix OLED-rendered visual stimuli with ambient light to present a mixed image to the user's eyes. In another embodiment, the AR-OST shield 606 may be completely transparent, but the frame 608 may also include the ability to render light directly onto the user's retina.

[0057] The strap 614 can connect the frame 608 to a compartment accessory or module that includes the BCI 616 and the plurality of integrated EEG electrodes 618. In another embodiment, the EEG electrodes 618 may be configured as part of a separate EEG device or other sensor device. The BCI 616 may likewise be configured as a separate computing device. Other sensor devices may include the g.USBamp and Gammabox (g.tec Guger Technologies of Austria) that are used with wet electrodes (g.Scarabeo) to acquire EEG signals. The AR-OST 604 and the smart device 612 can each transmit and receive signals to and from each other, as well as to additional computing devices and sensing devices, via wired and wireless connections.

[0058] Finally, due to the see-through characteristics of the AR-OST shield 606 of the AR-OST 604, in some embodiments, a monitor 620 may be incorporated into the AR-OST BCI configuration 600, allowing different images to be displayed on the inner surface of the AR-OST shield 606 as foreground features and on the monitor 620 as background features. In one embodiment, all images may be displayed inside the AR-OST shield 606, and in another embodiment, the images displayed within the AR-OST shield 606 may overlap a plurality of real-world objects visible in the user's environment. In another embodiment, the selection of the rendered stimuli is presented in the environment external to the AR-OST 604 wearable device. These external stimuli can be signs / logos / boards fixed in space, advertisements, or other notifications.

[0059] Figures 7A and 7B show projections for SSVEP BCI use with the AR-OST 700 according to an embodiment. The AR-OST BCI configuration 600 of FIG. 6 can be used to perform user interaction with the AR-OST and the BCI using the projections for SSVEP BCI use with the AR-OST 700.

[0060] In one embodiment, as shown at A in FIG. 7, the monitor display 702 may have the aforementioned non - active background 402, such as a black screen. The basic SSVEP pattern 202 may be shown as the AR - OST shield display 704. This basic SSVEP pattern 202 may be generated by a smart device at a fixed position within the smart device slot in the frame, as described with respect to FIG. 6. The smart device may be equipped with an app designed to interact with the SSVEP BCI system, or may be paired with an additional computing device configured to generate the basic SSVEP pattern 202 and its variations through a wired or wireless connection.

[0061] In an alternative embodiment shown at B in FIG. 7, the monitor display 702 may have the active background 404 introduced with respect to B in FIG. 4, and the basic SSVEP pattern 202 and its variations may also be shown as the AR - OST shield display 704.

[0062] FIGS. 8A and 8B show projections for the use of SSMVEP BCI using the AR - OST 800 according to an embodiment. The AR - OST BCI configuration 600 of FIG. 6 may be used to perform user interaction with the AR - OST and the BCI using the projection for the use of SSMVEP BCI using the AR - OST 800.

[0063] In one embodiment, as shown in FIG. 8A, the monitor display 802 may have the aforementioned non-active background 402, such as a black screen. The basic SSMVEP pattern 302 may be shown as an AR-OST shield display 804. This basic SSMVEP image 304 may be generated by a smart device located at a fixed position within a smart device slot within a frame as described with respect to FIG. 6. The smart device may have an app designed to interact with the SSVEP BCI system, or may be paired through a wired or wireless connection with an additional computing device configured to generate the basic SSMVEP image 304 and its variations.

[0064] In an alternative embodiment shown in FIG. 8B, the monitor display 802 may have the active background 404 introduced with respect to FIG. 4B, and the basic SSMVEP pattern 302 and its variations may also be shown as an AR-OST shield display 804.

[0065] FIGS. 8A and 8B show a potential controlled laboratory setup for testing, while FIG. 8C shows a real-world use case for projection for SSMVEP BCI use with the AR-OST 800. The user 602 wearing the AR-OST 604 may be interacting in the real world within the user's environment 808, such as walking on a sidewalk along a city street. In this case, the user view through the AR-OST shield 806 may incorporate the basic SSMVEP pattern 302 as a digital graphical element mixed with the visual perception of the user's environment 808 by the action of components integrated or associated with the AR-OST 604 as disclosed herein. Figure 9 shows a C-CNN process 900 according to an embodiment. Input 902 may be passed through a convolution 904, and the result of the convolution 904 may pass through a batch normalization ReLU activation dropout 906 step. The data from the batch normalization ReLU activation dropout 906 can pass through a convolution 908. The result of the convolution 908 may undergo a batch normalization ReLU activation dropout 910 step and finally generate an output 912 as shown. Such a C-CNN process 900 can provide higher accuracy for asynchronously processed data than conventional techniques such as canonical correction analysis (CCA) and CNNs that use the magnitude spectrum as input. The C-CNN method may be trained for both SSVEP and SSMVEP data for stimulus detection.

[0066] The C-CNN process 900 can process BCI data in an asynchronous manner with a fixed window length (W = [1 second, 2 seconds]) and a step size of 0.1 second. Window lengths longer than 2 seconds can significantly affect the speed of the overall BCI system when applied in real time. The C-CNN may be based on the concatenation of the real and imaginary parts of the fast Fourier transform (FFT) signal provided as input to the C-CNN. In one embodiment, the complex FFT of the segmented EEG data may be calculated with a resolution of 0.2930 Hz. Next, the real frequency components and the imaginary frequency components are extracted along each channel and may be concatenated into a single feature vector, such as I = Re(X)||Im(X). As a result, the feature vectors for each channel are stacked vertically to form an input matrix I ch ×N fc with dimensions of C-CNN . Here, N ch = 3 and N fc = 220.

Number

[0067] The C-CNN process 900 may be trained in a user-dependent scenario, where the classifier is trained on data from a single participant and tested on data from the same participant. The preprocessing step can be provided as a data augmentation strategy to increase the number of training examples for training the C-CNN. To evaluate the performance of the classifier, eight-fold stratified cross-validation may be performed so that there are no overlapping samples between the training fold and the validation fold. This is equivalent to leave one-trial out cross-validation. For W = 1 second, each fold can contain 1456 and 912 segments in the training set and the test set, respectively. For W = 2 seconds, there can be 1176 and 168 segments in the training set and the test set, respectively. Furthermore, the C-CNN can be trained individually for each stimulus type, background type, and window length for a single participant. The total number of trainable parameters may be 5482.

[0068] In one embodiment, the C-CNN can be trained on a processor and memory system such as an Intel Core i5-7200 central processing unit (CPU) with 2.50 GHz and 8GB random access memory (RAM). Categorical cross-entropy loss can be used to train the network. The final parameters of the network can be selected based on the values that provided the highest classification accuracy across the participants. In one embodiment, the selected parameters may be α = 0.001, momentum = 0.9, D = 0.25, L = 0.0001, E = 50, and B = 64, where α is the learning rate, D is the dropout rate, L is the L2 regularization constant, E is the number of epochs, B is the batch size, and these parameters are well understood in the art.

[0069] In another embodiment, the classification process can be performed by a heuristic, an expert system, a transformer, a long short-term memory (LSTM), a recurrent neural network (RNN), CCA, or any other classification algorithm suitable for processing multiple time-dependent input signals and placing them into a pre-defined set of classes.

[0070] The offline decoding performance of asynchronous four-class AR-SSVEP BCI under non-active background and a window length of 1 second can be 82% ± 15% using the C-CNN method described herein. The offline decoding performance that the AR-SSMVEP BCI can achieve can be 71.4% ± 22% for non-active background (NB) and 63.5% ± 18% for active background (AB) for W = 1, and 83.3% ± 27% (NB) and 74.1% ± 22% (AB) for W = 2 using the C-CNN method. The asynchronous pseudo-online SSMVEP BCI using the C-CNN technique can provide high decoding performance without the need to be precisely synchronized with the start of the stimulus. Furthermore, this technique can be robust to changes in background conditions. A difference in performance between the steady state and the transition state may be observed and may be attributed to the segmentation and training methods. Since the transition state windows may not be seen by the classifier during the training phase, these regions may be misclassified in the pseudo-online test phase. Windows during the transition state may contain a mixture of steady state data and transition state data, making it difficult to label such windows. This scenario is similar to the errors that may occur in an online system. One simple solution may be to increase the detection window length. This can reduce errors and improve overall performance.

[0071] FIG. 10 shows the IC state and the NC state 1000 according to an embodiment. The time of a user wearing the AR-OST and the BCI and interacting with them can include a break period 1002, a cue period 1004, and a stimulation period 1006. In a training, calibration, or evaluation scenario, these periods may be set as a time of a predetermined fixed length. For example, a 2-second cue period 1004 may precede a 6-second stimulation period 1006. During the cue period 1004, the stimulus to be focused on during the stimulation period 1006 may be highlighted. During the stimulation period 1006, the stimulus may be modulated at a desired frequency. A 4-second rest period during which no stimulus is presented may follow.

[0072] During the asynchronous interaction between AR - OST and BCI, the wearer may have cue periods 1004 and stimulation periods 1006 at different timings, may not have a cue period 1004, may have a break period 1002 of a specific duration, and may not have an established periodicity of stimulus presentation. Instead, the stimulus may be presented when a situation occurs in the user's environment or when the user asynchronously invokes a certain usage mode.

[0073] In one embodiment, the BCI can assume that the break period 1002 and the cue period 1004 constitute the NC state 1010. When no stimulus is presented, there may be no intention of control by the user. When a stimulus is presented during the stimulation period 1006, the BCI can assume that the user has entered the IC state 1008. As described with respect to the decision block 112 in FIG. 1, other aspects of user interaction such as gaze detection can be monitored to determine that the user intends to control some aspect of their AR experience.

[0074] To refine the processing of EEG signals based on the assumption or detection of the IC state 1008 rather than the NC state 1010, the last softmax layer of the C - CNN architecture can be modified to include a fifth NC class. This can result in a total of five neurons: four IC = (C 1 , C 2 , C 3 , C 4 ) states and one NC = C 5 state. The convolutional layers and kernels can remain the same as in the case of the 4 - class architecture. An 8 - fold cross - validation scheme may be used to evaluate the IC - versus - NC detection. The network can be trained using categorical cross - entropy loss. The final parameters of the network may be selected as α = 0.001, momentum = 0.9, D = 0.25, L = 0.0001, E = 80, and B = 40 in one embodiment.

[0075] In one embodiment, the two-class classification result (IC vs. NC) can be estimated from the results of the five-class C-CNN. The four target stimulus predictions may be combined into a single category IC class, and the rest state / NC may be the second class. From the confusion matrix, the true positive (TP) may be defined when, during the IC state, the user is looking at the target and the classifier correctly predicts this segment as the IC state. The false positive (FP) can be defined when the classifier predicts the segment as IC when the true label is the NC state. If the classifier misclassifies the IC state as NC, this can be defined as a false negative (FN). Then, the F1 score and the false activation rate (FAR) can be calculated as follows.

Number

[0076] In practical applications, classifying the IC / active state into different active states may have a worse impact than classifying it as the non-active class. Therefore, the FAR may be defined as the misclassification rate within different IC states, i.e., the misclassification rate between one IC state and another IC state.

Number

Number

[0077] Finally, the average FAR across all stimulus frequencies can be calculated according to the following formula.

Number

[0078] The trained 5-class C-CNN from one of the cross-validation folds can be applied in a pseudo-online manner for the entire training session. Specifically, this can be applied in a continuous decoding scenario that includes segments of data that include the transition segments between IC and NC. This step can emulate an online asynchronous BCI.

[0079] Figures 11A through 11H show the average amplitude spectra of SSVEP and SSMVEP stimulus responses under non-active background (NB) and active background (AB) conditions, tested using an exemplary configuration. Figure 11A shows result 1100a for SSVEP at 8 Hz. Figure 11B shows result 1100b for SSMVEP at 8 Hz. Figure 11C shows result 1100c for SSVEP at 10 Hz. Figure 11D shows result 1100d for SSMVEP at 10 Hz. Figure 11E shows result 1100e for SSVEP at 12 Hz. Figure 11F shows result 1100f for SSMVEP at 12 Hz. Figure 11G shows result 1100g for SSVEP at 15 Hz. Figure 11H shows result 1100h for SSMVEP at 15 Hz.

[0080] The inset of each graph shows an enlarged version of the fundamental stimulus frequency. For the SSVEP 8Hz result 1100a, the NB peak response 1102 and the AB peak response 1104 are shown. For the SSMVEP 8Hz result 1100b, the NB peak response 1106 and the AB peak response 1108 are shown. For the SSVEP 10Hz result 1100c, the NB peak response 1110 and the AB peak response 1112 are shown. For the SSMVEP 10Hz result 1100d, the NB peak response 1114 and the AB peak response 1116 are shown. For the SSVEP 12Hz result 1100e, the NB peak response 1118 and the AB peak response 1120 are shown. For the SSMVEP 12Hz result 1100f, the NB peak response 1122 and the AB peak response 1124 are shown. For the SSVEP 15Hz result 1100g, the NB peak response 1126 and the AB peak response 1128 are shown. For the SSMVEP 15Hz result 1100h, the NB peak response 1130 and the AB peak response 1132 are shown.

[0081] The average amplitude spectra of SSVEP and SSMVEP responses for four stimulus frequencies (8Hz, 10Hz, 12Hz, and 15Hz) under two background conditions (NB and AB) were averaged across all participants, trials, and electrode channels (O1, Oz, O2) to achieve exemplary results such as those shown herein. The average amplitude spectrum for each SSVEP stimulus clearly shows peaks at the targeted fundamental stimulus frequency and its corresponding harmonics. Next, for each SSMVEP stimulus, prominent peaks at the targeted fundamental frequency may be observed for all frequencies, and no other prominent responses were observed in the corresponding harmonics. These results confirm that the visual stimuli designed for the proposed optical see-through AR system can elicit the desired SSVEP and SSMVEP responses.

[0082] Also, it can be observed that the presence of an active background can reduce the amplitude of the responses at the fundamental frequency and harmonics for SSVEP stimuli. The amplitude differences calculated between NB and AB for each stimulus frequency can be, as shown, 0.3 μV (8 Hz), 0.86 μV (10 Hz), 0.44 μV (12 Hz), and 0.43 μV (15 Hz), respectively. On the other hand, for SSMVEP stimuli, the amplitude differences at the fundamental frequency between NB and AB can be 0.05 μV (8 Hz), 0.19 μV (10 Hz), 0.13 μV (12 Hz), and 0.09 μV (15 Hz), respectively. The average reduction in amplitude from NB to AB for all stimulus frequencies can be 28.2% and 8.3% for SSVEP and SSMVEP responses, respectively. The average SNR across all participants for NB vs AB SSVEP stimuli can be 6.75 vs 5.43 at 8 Hz, 8.15 vs 5.9 at 10 Hz, 6.9 vs 5.32 at 12 Hz, and 8.82 vs 6.7 at 15 Hz (dB). In contrast, the SNR values for SSMVEP stimuli can be 5.65 vs 5.32 at 8 Hz, 6.59 vs 5.77 at 10 Hz, 6.11 vs 6.09 at 12 Hz, and 6.02 vs 6.17 at 15 Hz (dB). The average reduction in SNR between NB and AB across all frequencies for SSVEP and SSMVEP was 1.75 dB and 0.25 dB, respectively.

[0083] For SSVEP stimuli, the active background can result in consistently lower CCA coefficients across all stimulus frequencies compared to the non - active background. In contrast, for SSMVEP stimuli, the magnitude of the CCA coefficients can be similar between the two backgrounds across all stimulus frequencies. This can indicate that the measured perception of SSMVEP stimuli is less affected by the presence of an active background than the measured perception of similar SSVEP stimuli. For both stimulus types, the response to a 15 - Hz stimulus is likely to be most affected by the presence of an active background.

[0084] One reason for the decrease in amplitude due to the active background can be attributed to the presence of competing stimuli in the background. Previous studies have shown that when multiple flickering visual stimuli are placed in the same visual field, they compete for neural representation. This is called the effect of competing stimuli. Therefore, various visual elements within the background video may interfere with the neural representation or compete for it, leading to a decrease in the overall robustness of the SSVEP stimulus. On the other hand, when an active background is introduced, the magnitude of the response for the SSMVEP stimulus may not decrease. This may indicate that the SSMVEP stimulus is more robust even when there are competing stimuli in the background.

[0085] The decrease in the amplitude of the response for both stimulus types can be attributed to an increase in visual and mental load in the presence of an active background. The mental load induced by the flickering SSVEP stimulus may be higher than that of the SSMVEP stimulus. Therefore, the reduction in attention requirements by the SSMVEP stimulus may generally result in higher performance by the SSMVEP stimulus compared to the SSVEP stimulus.

[0086] Figure 12 shows a system 1200 according to an embodiment. The system 1200 may include a sensor 1206 and a wearable biosignal sensing device 1208 worn by a user 1268, and the wearable biosignal sensing device may integrate, be closely associated with, or be paired with a smart device 1210 and a BCI 1226. The smart device 1210 may include a context module 1212, a user application 1214, and a rendering device 1220. The BCI 1226 can include a biosensor 1224, signal conditioning 1228, detection of intentional control signals 1230, and a classifier 1232. In some embodiments, the system 1200 may include a context manager 1252 and a model modification process 1254 stored on a cloud server 1250 and accessed through a connection thereto.

[0087] Sensor 1206 may be integrated with either, or both, of the wearable biosignal sensing device 1208 and the smart device 1210, or may not be integrated with either. Sensor 1206 may also be mounted or attached to a device worn or carried by the user 1268. Sensors integrated with the wearable biosignal sensing device 1208 and the smart device 1210 may sometimes be referred to herein as internal sensors 1260. Sensors not so integrated may sometimes be referred to herein as external sensors 1262.

[0088] In some embodiments, sensor 1206 can receive environmental stimuli 1204 from the surrounding environment 1202. These sensors 1206 can be internal sensors 1260 or external sensors 1262 that detect, as is well understood in the art, visible light, light beyond the visible spectrum, sound, pressure, temperature, proximity of objects in the environment, acceleration and direction of movement of objects or the wearable biosignal sensing device 1208 and the user 1268 in the environment, or other aspects of the environment 1202 and actions within the environment 1202.

[0089] In some embodiments, sensor 1206 can also detect information regarding the physical state and actions 1266 of the user. For example, sensor 1206 can be an external sensor 1262 that is a body-worn sensor such as a camera, a heart rate monitor, a portable medical device, etc. These sensors 1206 can detect aspects of the user's behavior, movement, intent, and state, as is well understood in the art, for example, through gaze detection, voice detection and recognition, sweat detection and composition, etc. Sensor 1206 can provide an output as sensor data 1264. Sensor data 1264 can carry information associated with environmental stimuli 1204 and the physical state and actions 1266 of the user, as detected by the internal sensors 1260 and external sensors 1262 that make up sensor 1206.

[0090] In some embodiments, it may be useful to distinguish environmental stimulus data 1240 from other sensor data 1264, and the environmental stimulus data 1240 may be transmitted as part of a separate data signal stream and, if desired, may undergo additional or alternative logical parsing, processing, and use. In other embodiments, all components of the sensor data 1264 that include environmental data generated in response to the environmental stimulus 1204 may not be distinguished in this regard and may be considered part of a single data signal stream.

[0091] The wearable biosignal sensing device 1208 can be a physical assembly that can include a sensor 1206, a smart device 1210, and a BCI 1226. In one embodiment, the wearable biosignal sensing device 1208 can be an AR-OST 604 as described with respect to FIG. 6. The smart device 1210 can incorporate sensing, display, and networking capabilities. The smart device 1210 can be, in one embodiment, a smartphone or a small tablet computer that can be held within the smart device slot 610 of the AR-OST 604. In one embodiment, the smart device 1210 can be an embedded computer such as a Raspberry Pi single-board computer. In one embodiment, the smart device 1210 can be a system-on-chip (SoC) such as a Qualcomm SnapDragon SoC. In some embodiments, the smart device 1210 can have an operating system such as Android or iOS configured to manage the hardware resources and the life cycle of user applications.

[0092] The smart device 1210 may be further configured using at least one user application 1214 that communicates with a context module 1212 for the purpose of enhancing user interaction with a user application 1214, including AR hands-free control, through the interaction of a sensor 1206, a wearable biosignal sensing device 1208, a user physiology unit 1222, and a BCI 1226. The user application 1214 may be an application that runs on the smart device and depends on user interaction. Such user applications 1214 may include virtual input devices such as virtual keyboards, head-up interactive map interfaces such as Google Maps and Waze, virtual assistants such as Amazon's Alexa or Apple's Siri, and the like.

[0093] In certain embodiments, other context data 1218 may be available for use in the solutions disclosed herein. Other context data 1218 may include data that is not sensed, i.e., data obtained not from the sensor 1206 but through interaction of the smart device 1210 with the Internet and user applications 1214 operating on the smart device 1210. For example, other context data 1218 may include dates and times detected from the built-in clock function of the smart device or obtained from the Internet, schedules from a calendar application including specific locations and times of appointments, application notifications and messages, and the like.

[0094] The Context Module 1212 may be a stand-alone application or may be compiled within other commercially available applications and configured to support the solutions disclosed herein. The Context Module 1212 can receive any combination of environmental stimulus data 1240, sensor data 1264, and other context data 1218 from the sensor 1206 (either or both of the internal sensor 1260 and the external sensor 1262) and the smart device 1210. The data provided to the Context Module 1212 can include at least one of environmental stimulus data 1240, sensor data 1264, and other context data 1218. In certain embodiments, the data provided to the Context Module 1212 includes environmental stimulus data 1240. In certain embodiments, the data provided to the Context Module 1212 includes sensor data 1264. In certain embodiments, the data provided to the Context Module 1212 includes other context data 1218. In certain embodiments, the data provided to the Context Module 1212 includes environmental stimulus data 1240 and sensor data 1264. In certain embodiments, the data provided to the Context Module 1212 includes environmental stimulus data 1240 and other context data 1218. In certain embodiments, the data provided to the Context Module 1212 includes sensor data 1264 and other context data 1218. In certain embodiments, the data provided to the Context Module 1212 includes environmental stimulus data 1240, sensor data 1264, and other context data 1218. The environmental stimulus data 1240, sensor data 1264, and other context data 1218 can include at least one of environmental data, body-worn sensor data, connected portable device data, location-specific connected device data, and network-connected device data.

[0095] The context module 1212 may receive one or more requested stimulus data 1216 from the user application 1214 on the smart device 1210. These requested stimulus data 1216 may indicate that the user application 1214 requires the user to select from several options. The context module 1212 may include a process for determining the device context state from at least one of the environmental stimulus data 1240, sensor data 1264, and other context data 1218 received from the non-sensing data included in the wearable biosignal sensing device 1208 or the internal sensor 1260 implemented on the smart device 1210, the external sensor 1262 communicating with the smart device 1210 or the wearable biosignal sensing device 1208, or other context data 1218.

[0096] The context module 1212 can further incorporate the ability to convert at least a portion of the requested stimulus data 1216 from the user application 1214 into modified stimuli 1238, based at least in part on environmental stimulus data 1240, sensor data 1264, and other context data 1218 that notify the device context state. In this way, the context module 1212 can form modified stimuli 1238 that can be provided to the user 1268 of the wearable biosignal sensing device 1208 to enable the user 1268 to select from among the options indicated by the requested stimulus data 1216 using the BCI-enabled AR interface. The modified stimuli 1238 can incorporate visual icons such as the SSMVEP stimuli introduced with respect to FIG. 3A. These SSMVEP stimuli can be modified by the context module 1212 based on the device context state, for example, to improve the distinction between the options presented to the user and environmental stimuli 1204 that may compete for the user's attention. Thus, the modified stimuli 1238 that the context module 1212 can generate using the context provided by the environmental stimulus data 1240, sensor data 1264, and other context data 1218 can induce a more easily distinguishable response from the user 1268 through the wearable biosignal sensing device 1208 and the BCI 1226 than the default SSMVEP stimuli, other default stimuli, and what the requested stimulus data 1216 could achieve without modification.

[0097] For example, if environmental stimulus 1204 is detected as environmental stimulus data 1240 that exhibits periodic behavior at a frequency of 10 Hz, rather than using a default generated stimulus that exhibits behavior at 10 Hz, context module 1212 can convert that 10 Hz default generated stimulus to exhibit that behavior at a frequency of 12 Hz. Thereby, user attention to the environmental stimulus exhibiting 10 Hz behavior is made less likely to be mistaken for the user 1268 gazing at a menu option that behaves at a similar frequency. Modifications based on environmental stimulus 1204 can also include changing where in the user's field of view the stimulus presented for user selection is located, converting the evoked potential to an auditory or tactile stimulus response, or other modifications that are facilitated by the environmental conditions detected through environmental stimulus data 1240 or specified by user preferences available through the smart device 1210 configuration.

[0098] The smart device 1210 can incorporate passive or active rendering device 1220 functionality. This can allow for modified stimuli 1238 as well as environmental stimuli 1204 to be presented to the user 1268. This rendering device 1220 can mix the environmental stimuli 1204 with the modified stimuli 1238, resulting in a rendered stimulus 1256 for presentation to the user's sensory system, allowing the user 1268 to perceive both the state of the environment 1202 and the choices integral to operating the user interface of the user application 1214. The modified stimuli and environmental stimuli 1204 can be rendered using at least one of a visual device, an auditory device, and a tactile device perceived by the user 1268. In one embodiment, the capabilities of the rendering device 1220 can be provided by a transparent partial-reflection AR-OST shield 606 as described with respect to FIG. 6. The environmental stimuli 1204 may be directly perceived by the user, and visual stimuli are transmitted through the light transmitted through the AR-OST shield 606. The visual aspect of the modified stimuli 1238 is rendered for display and is displayed on the smart device 1210 present within the smart device slot 610 and can reach the user's eyes being reflected due to the partial-reflection characteristics of the AR-OST shield 606 material. In another embodiment, the wearable biometric signal sensing device 1208 may incorporate an opaque headset and may rely on sensing and presentation hardware such as cameras and video projection to provide the user with environmental stimuli 1204 mixed with modified stimuli 1238 as the rendered stimulus 1256. In another embodiment, the wearable biometric signal sensing device 1208 can incorporate a transparent OLED display to enable the optical passage and rendering of the modified stimuli 1256.

[0099] The rendered stimulus 1256 presented to user 1268 can generate a response through user physiology 1222. User physiology 1222 may refer to the user's body as well as the associated peripheral and central nervous systems. Human responses represented by the body's, and particularly the nervous system's, reactions to visual, auditory, tactile, or other stimuli are well understood in the art and may be detected using biosensor 1224. Biosensor 1224 may be a plurality of sensors mounted on the body of user 1268 and / or incorporated within BCI 1226 that detect nervous system activity. These biosensors 1224 may be, as is well understood in the art, EEG electrodes 618, electromyogram (EMG) electrodes, electrocardiogram (EKG) electrodes, other cardiovascular and respiratory monitors, blood oxygen level and glucose level monitors, and other biosensors 1224.

[0100] Biosensor 1224 can provide a biosignal 1236 as an output. Biosignal 1236 is a raw signal recorded by biosensor 1224. Biosignal 1236 may be received from biosensor 1224 and may be generated, at least in part, in response to the rendered stimulus 1256. Biosignal 1236 may be received on wearable biosignal sensing device 1208. In some embodiments, biosignal 1236 may undergo signal conditioning 1228. Signal conditioning 1228 can incorporate methods for filtering and cleaning the raw data in the form of biosignal 1236. Such data may be filtered to remove noise, may undergo a fast Fourier transform to detect energy at discrete frequency levels, may have statistical analysis such as trend removal applied, or may be processed by other digital signal processing algorithms well known in the art. In some embodiments, classifier 1232 of BCI 1226 may be able to accept the raw biosignal 1236 without the need for signal conditioning 1228.

[0101] In some embodiments, BCI 1226 can incorporate intentional control signal detection 1230. This can be similar to the process described with respect to decision block 112 of FIG. 1. Intentional control signal detection 1230 can be a method for determining whether a user intends to fixate on one or more modified stimuli 1238. In certain embodiments, intentional control signal detection 1230 can determine the presence of an intentional control signal, at least in part, by determining from the received biosignals 1236 that the user 1268 intends to fixate on at least one of the rendered stimuli 1256. In another embodiment, the context module 1212 of the smart device 1210 can send a manual intent override 1242 signal to the intentional control signal detection 1230, indicating that the intentional control signal detection 1230 may assume the presence of user intent control regardless of the received biosignals 1236.

[0102] In embodiments that use intentional control signal detection 1230, when no intentional control signal is present, the raw or conditioned biosignals 1236 and the input from the context module 1212 can be used to continue monitoring for intentional control without sending the raw or conditioned biosignals 1236 to the classifier 1232. When an intentional control signal is detected, the intentional control signal detection 1230 can send the raw or conditioned biosignals 1236 to the classifier 1232. In some embodiments, intentional control signal detection 1230 may not be used, and the raw or conditioned biosignals 1236 may be sent directly from the biosensor 1224 or signal conditioning 1228 to the classifier 1232, respectively.

[0103] Classifier 1232 can receive raw or conditioned biosignals 1236. Classifier 1232 may also receive a modified stimulus 1238 from context module 1212 in order to refine the classification through an understanding of the expected user 1268 response. Classifier 1232 may be configured to classify the received biosignal 1236 based on the modified stimulus 1238, resulting in a classified selection 1248. The classified selection 1248 can indicate which of the rendered stimuli 1256 the user is fixating on, based on the modified stimulus 1238 and the biosignal 1236.

[0104] A classifier is an algorithm that maps input data to specific categories, such as a machine learning algorithm used to assign class labels to data inputs, as understood in the art. One example is an image recognition classifier trained to label an image based on objects appearing in the image, such as "person", "tree", "car", etc. Types of classifiers include, for example, perceptrons, naive Bayes, decision trees, logistic regression, k-nearest neighbors, artificial neural networks, deep learning, and support vector machines, as well as ensemble methods such as random forests, bagging, and boosting.

[0105] Traditional classification techniques use machine learning algorithms to classify single trial spatio-temporal activation matrices based on their statistical properties. These methods are based on two main components, namely, a feature extraction mechanism for effective dimensionality reduction and a classification algorithm. A typical classifier uses sample data to learn a mapping rule by which other test data can be classified into one of two or more categories. Classifiers can be broadly classified into linear and non-linear methods. Non-linear classifiers such as neural networks, hidden Markov models, and k-nearest neighbors can approximate a wide range of functions and allow the identification of complex data structures. Non-linear classifiers have the potential to capture complex discriminant functions, but their complexity can also cause overfitting, have heavy computational requirements, and thus may not be very suitable for real-time applications.

[0106] On the other hand, linear classifiers are less complex and thus more robust against data overfitting. Linear classifiers work particularly well for data that can be linearly separated. Fisher Linear Discriminant (FLD), Linear Support Vector Machine (SVM), and Logistic Regression (LR) are examples of linear classifiers. FLD finds a linear combination of features that maps the data of two classes onto a separable projection axis. The criterion for separation is defined as the ratio of the distance between class means to the within-class scatter. SVM finds a separating hyperplane that maximizes the margin between two classes. LR projects the data onto a logistic function, as its name suggests.

[0107] Machine learning software may be custom computer code, may be commercially available for use in classification, or may be a customized version of commercially available machine learning. Examples of machine learning software include IBM Machine Learning, Google Cloud AI Platform, Azure Machine Learning, and Amazon Machine Learning.

[0108] In some embodiments, classifier 1232 can be implemented as a C-CNN as introduced with respect to FIG. 9. A C-CNN, other neural network, or other machine learning algorithm can be trained to recognize patterns in raw or conditioned biosignals 1236 corresponding to a user's physiological response to a rendered stimulus 1256 corresponding to a modified stimulus 1238 related to the required stimulus data 1216. Classifier 1232 may classify the rendered stimulus 1256 as having user focus and send an indication of this focus as a classified selection 1248 for further processing 1234 by smart device 1210, and in some embodiments particularly by user application 1214 on smart device 1210. Further processing 1234 can be processing performed in smart device 1210 and / or user application 1214 that analyzes and / or utilizes the classified selection 1248 from classifier 1232.

[0109] In some embodiments, smart device 1210 and BCI 1226 of wearable biosignal sensing device 1208 may communicate with a cloud server 1250, i.e., a network-connected computing resource. Cloud server 1250 can provide connections to a context manager 1252 and a model modification process 1254. Context manager 1252 can be a cloud-based system that provides additional context information via a network connection. Context module 1212 can send current device context state data and requests for other device context state data 1244 to context manager 1252. Context module 1212 can then receive a response regarding the recommended device context state as well as notifications and data regarding new stimuli 1246 from context manager 1252.

[0110] The model modification process 1254 may also be available through the cloud server 1250. The model modification process 1254 can function offline, i.e., asynchronously, separately from the activities of the components of the wearable biosignal sensing device 1208, the smart device 1210, and the BCI 1226. The model modification process 1254 can be, for example, a service that provides non-real-time updates to the classifier 1232 when the wearable biosignal sensing device 1208 is not in use. One embodiment of the use of the model modification process 1254 will be described in more detail with respect to FIG. 13.

[0111] FIG. 13 shows classifier model modification 1300 according to an embodiment. In the disclosed system, "model modification" is defined as any parameter tuning, hyperparameter optimization, reinforcement learning, model training, cross-validation, or feature engineering method that can be used to facilitate the classification of biosignal data. The classifier model modification 1300 can involve elements of the system 1200 as shown, in addition to the machine learning model transmission controller 1302 implemented within the smart device 1210 along with the context module 1212, the model modification process 1304 capabilities incorporated within the BCI 1226, and the model modification process 1254 within the cloud server 1250, the local data record 1306 stored on the BCI 1226, and the data record 1308 stored within the cloud server 1250. Elements of the system 1200 not shown in FIG. 13 are omitted for simplicity of explanation but can be included in embodiments of the system 1200 configured to implement the classifier model modification 1300.

[0112] As described with respect to FIG. 12, the context module 1212 can send requests for current device context state data and other device context state data 1244 to the context manager 1252 within the cloud server 1250. The context module 1212 can then receive responses regarding the recommended device context state as well as notifications and data regarding new stimuli 1246 from the context manager 1252. The context manager 1252 can send new state data and updated state data 1320 to the model modification process 1254 on the cloud server 1250 for use in classifier model modification 1300. The cloud server 1250 can further receive classified selections 1248 to be used in classifier model modification 1300 from the classifier 1232, either directly from elements on the BCI 1226 or through additional processing 1234 executed on the smart device 1210. The machine learning model 1314 can be updated using at least one model modification process 1254 and at least one of the classified selections 1248 and the new state data and updated state data 1320.

[0113] The cloud server 1250 can send an updated or new machine learning model (as indicated by the new machine learning model and updated machine learning model 1322) to the smart device. The updated machine learning model can be sent to the classifier using the machine learning model transmission controller 1302 on the smart device. In one embodiment, the context module 1212 on the smart device 1210 can request a new machine learning model from the cloud server 1250 using the machine learning model transmission controller 1302 (see new model request 1310). The smart device 1210 may receive a new machine learning model from the cloud server 1250 (see new machine learning model and updated machine learning model 1322) and may send the new machine learning model to the classifier 1232.

[0114] In one embodiment, the context module 1212 of the smart device 1210 can send a request for a new model 1310 to the machine learning model transmission controller 1302. The machine learning model transmission controller 1302 can request and receive a model specification and initial parameters 1312 from the model modification process 1254 within the cloud server 1250. The machine learning model transmission controller 1302 can then send the machine learning model 1314 to the classifier 1232 for use in classifying the biosignal 1236 received by the classifier 1232 as described above. The classifier 1232 can send the selected predicted stimulus and associated metrics 1316 for further processing 1234.

[0115] In one embodiment, data from the further processing 1234 may be sent to the model modification process 1304 module on the BCI 1226, allowing the BCI 1226 to improve the classification performed by the classifier 1232. In one embodiment, the classifier 1232 may send back to the machine learning model transmission controller 1302 a more refined or optimized model formed through the operation of the model modification process 1304, and the machine learning model transmission controller 1302 may then provide the updated model to the model modification process 1254 within the cloud server 1250.

[0116] In one embodiment, the biosignal data and model parameters 1318 from the further processing 1234 may be sent to the local data record 1306 on the BCI 1226 for use in the model modification process 1304 located within the BCI 1226. The local data record 1306 may also be sent to the data record 1308 within the cloud server 1250 for off-device storage. The data record 1308 may be available for the model modification process 1254 for offline classifier model modification 1300 that is executed asynchronously and independently of the BCI 1226 of the smart device 1210 and / or the wearable biosignal sensing device 1208.

[0117] As shown in FIG. 14, the smart device 1400 is shown in the form of a general-purpose computing device. The components of the smart device 1400 may include, but are not limited to, one or more processors or processing units 1404, a system memory 1402, and a bus 1424 that couples various system components including the system memory 1402 to the processor processing unit 1404. The smart device 1400 may include sensors 1426 such as a camera, an accelerometer, a microphone, etc., and actuators 1428 such as a speaker, a vibration or tactile actuator. The smart device 1400 may be a smartphone, a tablet, or other computing device suitable for implementing the disclosed solutions described herein.

[0118] The bus 1424 represents one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include I2C (Inter-Integrated Circuit), SPI (Serial Peripheral Interface), CAN (Controller Area Network), ISA (Industry Standard Architecture) bus, MCA (Micro Channel Architecture) bus, EISA (Enhanced ISA) bus, VESA (Video Electronics Standards Association) local bus, and PCI (Peripheral Component Interconnect) bus.

[0119] The smart device 1400 typically includes a variety of computer system readable media. Such media can be any available media accessible by the smart device 1400, including both volatile and non-volatile media, and both removable and non-removable media.

[0120] System memory 1402 can include computer system readable media in the form of volatile memory such as random access memory (RAM) 1406 and / or cache memory 1410. The smart device 1400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example, storage system 1418 can be provided to read from and write to a non-removable non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy (registered trademark) disk"), and an optical disk drive for reading from and writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media may be provided. In such instances, each can be connected to bus 1424 by one or more data media interfaces. As further shown and described below, system memory 1402 can include at least one program product having a set of (e.g., at least one) program modules configured to execute the functions of the disclosed solutions.

[0121] A program / utilility 1420 having a set of (e.g., at least one) program modules 1422 can be stored in the system memory 1402, such as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, can include an implementation of a networking environment. The program modules 1422 generally execute the functions and / or methods of the disclosed solutions as described herein.

[0122] Smart device 1400 may also communicate with one or more external devices 1412 such as a keyboard, a pointing device, a display 1414; one or more devices that enable a user to interact with smart device 1400; and / or any device (e.g., a network card, a modem, etc.) that enables smart device 1400 to communicate with one or more other computing devices. Such communication may occur via I / O interface 1408. I / O interface 1408 may also manage input from sensors 1426 of smart device 1400, as well as output to actuators 1428. Further, smart device 1400 may communicate via network adapter 1416 with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet). As shown, network adapter 1416 communicates with other components of smart device 1400 via bus 1424. Although not shown, it will be understood by those skilled in the art that other hardware and / or software components may be used with smart device 1400. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, redundant arrays of independent disks (RAID) systems, tape drives, data archival storage systems, etc.

[0123] Referring now to FIG. 15, an exemplary cloud computing system 1500 is shown. "Cloud computing" refers to a model that enables convenient on-demand network access to a shared pool of configurable computing resources (such as networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model promotes availability and is composed of at least five characteristics, at least three service models, and at least four deployment models. Examples of commercially hosted cloud computing systems 1500 include Amazon Web Services (AWS), Google Cloud, Microsoft Azure, and the like.

[0124] As shown, cloud computing system 1500 may comprise one or more cloud servers including, for example, context manager 1252, model modification process 1254, and data record 1308, with which computing devices such as a personal digital assistant (PDA) or smart device 1400, desktop computer 1504, laptop 1502, and / or wearable biosignal sensing device 1208 BCI 1226 may communicate. This allows infrastructure, platform, and / or software to be provided as services from cloud server 1250 (as described above in FIG. 14) such that each client does not need to maintain such resources separately. The types of computing devices shown in FIG. 15 are merely intended to be exemplary, and it should be understood that cloud server 1250 may communicate with any type of computerized device (such as using a web browser) through any type of network and / or network / addressable connection.

[0125] This disclosure includes a detailed description regarding cloud computing, but it should be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0126] This disclosure includes a detailed description regarding cloud computing, but it should be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0127] Cloud computing refers to a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0128] The characteristics are as follows.

[0129] On-demand self-service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with a service provider.

[0130] Broad network access: The capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (such as mobile phones, laptops, and PDAs).

[0131] Resource Pool: The computing resources of the provider are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided and may be able to specify their location at a higher level of abstraction (e.g., country, state, or data center), giving a sense of location independence.

[0132] Rapid Elasticity: Functions can be provisioned quickly and elastically, sometimes automatically, to scale out rapidly and released quickly to scale in. To the consumer, the functions available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

[0133] Measured Service: Cloud systems automatically control and optimize resource use by leveraging a metering function at some appropriate level of abstraction for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource use can be monitored, controlled, reported, and made transparent to both the provider and the consumer of the service being utilized.

[0134] The service model is as follows.

[0135] Software as a Service (SaaS): The functionality provided to consumers is to use the provider's applications that run on cloud infrastructure. The applications are accessible from various client devices through a client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, with limited exceptions for user-specific application configurations.

[0136] Platform as a Service (PaaS): The functionality provided to consumers is to deploy consumer-created or acquired applications created using programming languages and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but have control over the deployed applications and potentially the application hosting environment configuration.

[0137] Infrastructure as a Service (IaaS): The functionality provided to consumers is to provision processing, storage, networks, and other basic computing resources, where consumers can then deploy and run any software that may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have limited control over the operating systems, storage, deployed applications, and potentially selected networking components (e.g., host firewalls).

[0138] The deployment model is as follows.

[0139] Private cloud: The cloud infrastructure is operated only for a certain organization. It may be managed by that organization or a third party, and may exist on-premises or off-premises.

[0140] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community that shares concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by those organizations or a third party, and may exist on-premises or off-premises.

[0141] Public cloud: The cloud infrastructure is made available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0142] Hybrid cloud: The cloud infrastructure remains a separate entity but is a composite of two or more clouds (private, community, or public) connected to each other by standardized technologies or proprietary technologies (e.g., cloud bursting for load distribution between clouds) that enable data and application portability.

[0143] The cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the center of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0144] Referring now to FIG. 16, a set of cloud computing function abstraction layers 1600 provided by a cloud computing system 1500 as shown in FIG. 15 is shown. The components, layers, and functions shown in FIG. 16 are intended to be merely exemplary, and it will be understood by those skilled in the art that the present disclosure is not limited thereto. As shown, the following layers and corresponding functions are provided.

[0145] The hardware and software layer 1602 includes hardware components and software components. Examples of hardware components include mainframes, reduced instruction set computer (RISC) architecture-based servers, servers, blade servers, storage devices, and networks and networking components. Examples of software components include network application server software and database software.

[0146] The virtualization layer 1604 provides an abstraction layer from which the following exemplary virtual entities can be provided: virtual servers, virtual storage, virtual networks including virtual private networks, virtual applications, and virtual clients.

[0147] The management layer 1606 provides the exemplary functions described below. Resource provisioning provides for the dynamic procurement of computing resources and other resources utilized to execute tasks within a cloud computing environment. Metering and pricing provides for cost tracking when resources are utilized within the cloud computing environment, and for charging or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides for authentication of the identities of users and tasks, as well as protection for data and other resources. The user portal provides access to the cloud computing environment for both users and system administrators. Service level management provides for cloud computing resource allocation and management such that required service levels are met. Service level agreement (SLA) formulation and fulfillment provides for the advance arrangement and procurement of cloud computing resources whose future requirements are anticipated in accordance with the SLA.

[0148] The workload layer 1608 provides functions for which the cloud computing environment is utilized. Examples of workloads and functions that can be provided from this layer include mapping and navigation, software development and lifecycle management, virtual classroom education delivery, data analytics processing, transaction processing, and resource credit management. As described above, all of the foregoing examples described with respect to FIG. 16 are merely illustrative, and the present disclosure is not limited to these examples. List of elements of the drawings 100 Process for using SSMVEP BCI 102 Stimuli generated for SSMVEP 104 Environmental stimuli 106 Block 108 Block 110 Block 112 Decision block 114 Block 116 Decision block 118 Blocks 200 Exemplary SSVEP Patterns and Variations 202 Basic SSVEP Pattern 204 SSVEP Image On 206 SSVEP Image Off 300 Exemplary SSMVEP Patterns and Variations 302 Basic SSMVEP Pattern 304 Basic SSMVEP Image 306 SSMVEP Images with Different Pattern Densities 308 SSMVEP Images of Different Sizes 310 Rotated SSMVEP Images 400 Binocular Projection for SSVEP BCI Interaction 402 Inactive Background 404 Active Background 500 Binocular Projection for SSMVEP BCI Interaction 600 AR-OST BCI Configuration 602 User 604 AR-OST 606 AR-OST Shield 608 Frame 610 Smart Device Slot 612 Smart Device 614 Strap 616 BCI 618 EEG Electrodes 620 Monitor 700 Projection for SSVEP BCI Use with AR-OST 702 Monitor Display 704 AR-OST Shield Display 800 Projection for SSMVEP BCI Use with AR-OST 802 Monitor Display 804 AR-OST Shield Display 806 User View through AR-OST Shield 808 User's Environment 900 C-CNN Process 902 Input 904 Convolution 906 Batch Normalization ReLU Activation Dropout 908 Convolution 910 Batch Normalization ReLU Activation Dropout 912 Output 1000 IC State and NC State 1002 Rest Period 1004 Queue Period 1006 Stimulation Period 1008 IC State 1010 NC State 1100a Results of SSVEP 8Hz 1100b Results of SSMVEP 8Hz 1100c Results of SSVEP 10Hz 1100d Results of SSMVEP 10Hz 1100e Results of SSVEP 12Hz 1100f Results of SSMVEP 12Hz 1100g Results of SSVEP 15Hz 1100h Results of SSMVEP 15Hz 1102 NB Peak Response 1104 AB Peak Response 1106 NB Peak Response 1108 AB Peak Response 1110 NB Peak Response 1112 AB Peak Response 1114 NB Peak Response 1116 AB Peak Response 1118 NB Peak Response 1120 AB Peak Response 1122 NB Peak Response 1124 AB Peak Response 1126 NB Peak Response 1128 AB Peak Response 1130 NB Peak Response 1132 AB Peak Response 1200 System 1202 Environment 1204 Environmental Stimulus 1206 Sensor 1208 Wearable Biosignal Detection Device 1210 Smart Device 1212 Context Module 1214 User Application 1216 Required Stimulus Data 1218 Other Context Data 1220 Rendering Device 1222 User Physiology Department 1224 Biosensor 1226 BCI 1228 Signal Conditioning 1230 Intentional Control Signal Detection 1232 Classifier 1234 Further Processing 1236 Biosignal 1238 Modified Stimulus 1240 Environmental Stimulus Data 1242 Manual Intent Override 1244 Request for Current Device Context State Data and Other Device Context State Data 1246 Response Regarding Recommended Device Context State and Notification and Data Regarding New Stimuli 1248 Classified Selection 1250 Cloud Server 1252 Context Manager 1254 Model Modification Process 1256 Rendered Stimulus 1258 User Physical Response 1260 Internal Sensor 1262 External Sensor 1264 Sensor Data 1266 User's Physical State and Actions 1268 User 1300 Classifier Model Modification 1302 Machine Learning Model Transmission Controller 1304 Model Modification Process 1306 Local Data Record 1308 Data Record 1310 Request for New Model 1312 Request and Receive Model Specifications and Initial Parameters 1314 Machine Learning Model 1316 Selected Predicted Stimuli and Associated Metrics 1318 Biosignal Data and Model Parameters 1320 New State Data and Updated State Data 1322 New Machine Learning Model and Updated Machine Learning Model 1400 Smart Device 1402 System Memory 1404 Processing Unit 1406 Random Access Memory (RAM) 1408 I / O Interface 1410 Cache Memory 1412 External Device 1414 Display 1416 Network Adapter 1418 Storage System 1420 Program / Utility 1422 Program Module 1424 Bus 1426 Sensor 1428 Actuator 1500 Cloud Computing System 1502 Laptop 1504 Desktop Computer 1600 Cloud Computing Functional Abstraction Layer 1602 Hardware and Software Layers 1604 Virtualization Layer 1606 Management Layer 1608 Workload Layer

[0149] Various functional operations described in this specification can be implemented in logic that refers to nouns or noun phrases that reflect the operation or function. For example, the association operation can be performed by an "associator" or "correlator". Similarly, switching may be performed by a "switch", selection may be performed by a "selector", and so on.

[0150] Within this disclosure, different entities (which may be variously referred to as "units", "circuits", other components, etc.) can be described or claimed as being "configured" to perform one or more tasks or operations. This expression - [an entity] configured [to perform one or more tasks] - is used herein to refer to a structure (i.e., something physical such as an electronic circuit). More specifically, this formulation is used to indicate that the structure is arranged to perform the one or more tasks during operation. A structure can be said to be "configured" to perform some task even if the structure is not currently operating. A "credit distribution circuit configured to distribute credits to a plurality of processor cores", for example, is intended to cover an integrated circuit having a circuit that performs this function during operation even if the integrated circuit in question is not currently in use (e.g., power is not connected to it). Thus, an entity described or stated as being "configured" to perform some task refers to something physical such as a device, circuit, memory, etc. that stores executable program instructions for implementing the task. This phrase is not used herein to refer to something intangible.

[0151] The term "configured to" is not intended to mean "capable of being configured to". For example, a field programmable gate array (FPGA) that is not programmed is not considered to be "configured to" perform any particular function, although it may be "configurable to" perform that function after programming.

[0152] In the appended claims, it is expressly intended that reciting that a structure is "configured to" perform one or more tasks does not invoke 35 U.S.C. § 112(f) with respect to that claim element. Thus, claims of the present application that do not otherwise recite "means for" performing [the function] should not be construed under 35 U.S.C. § 112(f).

[0153] As used herein, the term "based on" is used to describe one or more factors that affect a determination. This term does not exclude the possibility that additional factors may affect the determination. That is, the determination may be based solely on the specified factors, or may be based on the specified factors as well as other unspecified factors. Consider the phrase "determine A based on B". This phrase specifies that B is a factor that is used to, or affects, the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on some other factor, such as C. This phrase is also intended to encompass embodiments in which A is determined based solely on B. As used herein, the phrase "based on" is synonymous with the phrase "at least in part based on".

[0154] As used herein, the phrase "in response to" recites one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may affect the effect or, alternatively, trigger the effect. That is, the effect may occur simply in response to those factors, or in response to the specified factors as well as other unspecified factors. Consider the phrase "perform A in response to B". This phrase specifies that B is a factor that triggers the performance of A. This phrase does not exclude the possibility that the performance of A may also occur in response to some other factor, such as C. This phrase is also intended to encompass embodiments in which A is performed in response to B only.

[0155] As used herein, terms such as "first", "second", etc. are used as labels for the nouns that follow and, unless otherwise specified, do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, in a register file having eight registers, the terms "first register" and "second register" can be used to refer to any two of the eight registers, and not just, for example, logical registers 0 and 1.

[0156] As used in the claims, the term "or" is used as an inclusive disjunction rather than an exclusive disjunction. For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, as well as any combination thereof.

[0157] Although the exemplary embodiments have been described in detail, it will be apparent that modifications and variations are possible without departing from the scope of the claimed disclosed solution. The scope of the disclosed subject matter is not limited to the illustrated embodiments, but rather is defined by the following claims.

[0158] The terms used in this specification should be given their ordinary meaning in the relevant technical field or the meaning indicated by their use in the context, provided that where a clear definition is provided, that meaning shall govern.

[0159] In this specification, references to "one embodiment" or "an embodiment" do not necessarily refer to the same embodiment, but may do so. Unless the context clearly requires otherwise, throughout this specification and the claims, words such as "comprise", "comprising", etc. shall be construed in an inclusive sense, i.e., in the sense of "including but not limited to", rather than in an exclusive or exhaustive sense. Words using the singular or plural shall include the plural or singular respectively, unless clearly limited to the singular or plural. Additionally, the words "herein", "above", "below", and words of similar meaning, when used in this application, refer to the whole of this application rather than any particular part of this application. When a claim uses the word "or" with respect to a list of two or more items, that word shall cover the following interpretations of the word, i.e., any of the items in the list, all of the items in the list, and all combinations of any of the items in the list, unless explicitly limited to one or the other. Any term not explicitly defined in this specification shall have its ordinary meaning generally understood by those skilled in the art.

[0160] It should be understood that the disclosed subject matter is not limited, in its application, to the details of construction and the arrangement of components described in the following description or shown in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Also, it should be understood that the expressions and terms used in this specification are for the purpose of description and should not be regarded as limiting.

[0161] Accordingly, those skilled in the art will appreciate that the concepts upon which this disclosure is based can be readily utilized as a basis for designing other structures, systems, methods, and media for carrying out some of the purposes of the disclosed subject matter. Thus, it is important that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the disclosed subject matter.

Claims

1. Receiving one or more requested stimulus data from a user application on a smart device; Receiving at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from the surrounding environment, and the other context data includes data that is not sensed; Converting at least a portion of the requested stimulus data into a modified stimulus, at least in part based on at least one of the sensor data and the other context data; Presenting the modified stimulus and the environmental stimulus to a user using a rendering device configured to mix the modified stimulus and the environmental stimulus, thereby resulting in a rendered stimulus; Receiving a biosignal generated in response to the rendered stimulus from the user on a wearable biosignal sensing device; Classifying the received biosignal using a classifier based on the modified stimulus, resulting in a classified selection; Returning the classified selection to the user application. A method.

2. After receiving the biosignal from the user: Determining whether to transmit the received biosignal to the classifier using at least one of the presence of an intentional control signal and the absence of the intentional control signal, wherein determining the presence of the intentional control signal includes: Detecting a manual intent override signal from the smart device; Discriminating, at least in part from the received biosignal, that the user intends to fixate on at least one of the rendered stimuli including at least one of; Transmitting the received biosignal to the classifier if the intentional control signal is present; Continuing to receive the received biosignal from the user if the intentional control signal is absent. The method according to claim 1, further comprising: The method according to claim 1, further comprising: The method according to claim 1, further comprising: The method according to claim 1, further comprising:

3. The method according to claim 1, wherein the modified stimulus is based at least in part on determining a device context state using at least one of the sensor data and the other context data.

4. The method according to claim 1, wherein presenting the modified stimulus and the environmental stimulus to the user includes rendering the modified stimulus and the environmental stimulus using at least one of a visual device, a tactile device, and an auditory device perceived by the user.

5. The method according to claim 1, wherein the modified stimulus includes a steady-state motion visual evoked potential stimulus, and presenting the modified stimulus and the environmental stimulus to the user includes rendering the modified stimulus and the environmental stimulus on an augmented reality optical see-through (AR-OST) device associated with the smart device.

6. The at least one of the sensor data and the other context data is: environmental data, body-worn sensor data, connected portable device data, location-specific connected device data, and network-connected device data The method according to claim 1, including at least one of the above.

7. Receiving, by a cloud server, the classified selection from the classifier, wherein the cloud server includes: a context manager; a machine learning model used by the smart device to facilitate classification of the received biosignals by the classifier; and at least one model modification process for modifying the machine learning model; and receiving, by the context manager, at least one of a request for current context state data and other state data; receiving, by the at least one model modification process, at least one of new state data and updated state data from the context manager; The method according to claim 1, further comprising updating the machine learning model using at least one of the at least one model modification process, the classified selection, the new state data, and the updated state data. The method according to claim 1.

8. The step of transmitting the updated machine learning model from the cloud server to the smart device; The step of further including using a machine learning model transmission controller on the smart device to transmit the updated machine learning model to the classifier; The method according to claim 7.

9. The step of using the machine learning model transmission controller by a context module on the smart device to request a new machine learning model from the cloud server; The step of receiving, by the smart device, the new machine learning model from the cloud server; The step of further including transmitting the new machine learning model to the classifier; The method according to claim 8.

10. The method according to claim 1, further comprising a context manager on the cloud server, wherein the context manager provides additional context information to the smart device.

11. A smart device; A rendering device; A wearable biometric signal sensing device on a user; A processor; A memory storing instructions A system comprising: when the instructions are executed by the processor, the system: The step of receiving one or more requested stimulus data from a user application on the smart device; The step of receiving at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from the surrounding environment and the other context data includes data that is not sensed; The step of converting at least a portion of the requested stimulus data into a modified stimulus, at least in part based on at least one of the sensor data and the other context data; The step of presenting the modified stimulus and the environmental stimulus to the user using the rendering device configured to mix the modified stimulus and the environmental stimulus to thereby result in a rendered stimulus; The step of receiving, on the wearable biometric signal sensing device, a biometric signal generated in response to the rendered stimulus from the user; Based on the modified stimulus, using a classifier to classify the received biosignal and resulting in a classified selection; Configured to perform the step of returning the classified selection to the user application, A system.

12. The instructions further cause the system, after receiving the biosignal from the user: The presence of an intentional control signal, and Using at least one of the presence and absence of the intentional control signal, To determine whether to send the received biosignal to the classifier, wherein the determination of the presence of the intentional control signal is: Detecting a manual intent override signal from the smart device; Determining, at least in part from the received biosignal, that the user intends to fixate on at least one of the rendered stimuli Including at least one of the steps; On the condition that the intentional control signal is present, Sending the received biosignal to the classifier; On the condition that the intentional control signal is absent, Continuing to receive the received biosignal from the user, which is configured to perform the steps of, The system according to claim 11.

13. The modified stimulus is, in part, based on determining a device context state using at least one of the sensor data and the other context data, the system according to claim 11.

14. Presenting the modified stimulus and the environmental stimulus to the user includes rendering the modified stimulus and the environmental stimulus using at least one of a visual device, a tactile device, and an auditory device perceived by the user, the system according to claim 11.

15. The modified stimulus includes a steady-state motion visual evoked potential stimulus, and presenting the modified stimulus and the environmental stimulus to the user includes rendering the modified stimulus and the environmental stimulus on an augmented reality optical see-through (AR-OST) device associated with the smart device, the system according to claim 11.

16. The at least one of the sensor data and the other context data is: The system according to claim 11, comprising at least one of environmental data, body-worn sensor data, connected mobile device data, location-specific connected device data, and network-connected device data. The system according to claim 11, comprising at least one of environmental data, body-worn sensor data, connected mobile device data, location-specific connected device data, and network-connected device data. **Claim 17** The instructions further cause the system to: Receive, by a cloud server, the classified selection from the classifier, where the cloud server comprises: A context manager; A machine learning model used by the smart device to facilitate classification of the received biosignals by the classifier; and At least one model modification process for modifying the machine learning model, the steps including; Receive, by the context manager, at least one of a request for current context state data and other state data; Receive, by the at least one model modification process, at least one of new state data and updated state data from the context manager; Update the machine learning model using at least one of the at least one model modification process, the classified selection, the new state data, and the updated state data. The system according to claim 11. **Claim 18** The instructions further cause the system to: Transmit, by the cloud server, an updated machine learning model to the smart device; and Transmit, using a machine learning model transmission controller on the smart device, the updated machine learning model to the classifier. The system according to claim 17. **Claim 19** The instructions further cause the system to: Request, by a context module on the smart device, using the machine learning model transmission controller, a new machine learning model from the cloud server; Receive, by the smart device, the new machine learning model from the cloud server; and Transmit the new machine learning model to the classifier. The system according to claim 18. **Claim 20** Receiving one or more requested stimulus data from a user application on a smart device; Receiving at least one of sensor data and other context data, wherein the sensor data includes environmental stimuli from the surrounding environment, and the other context data includes data that is not sensed; Converting at least a portion of the requested stimulus data into a modified stimulus, at least in part based on at least one of the sensor data and the other context data, wherein the modified stimulus includes steady-state visually evoked potential stimuli; Presenting the modified stimulus and the environmental stimulus to a user using a rendering device configured to mix the modified stimulus and the environmental stimulus to thereby provide a rendered stimulus, wherein presenting the modified stimulus and the environmental stimulus to the user comprises: Rendering the modified stimulus and the environmental stimulus using at least one of a visual device, a tactile device, and an auditory device sensed by the user; and Rendering the modified stimulus and the environmental stimulus on an augmented reality optical see-through (AR-OST) device associated with the smart device, including at least one of; Receiving a biosignal from the user generated in response to the rendered stimulus on a wearable biosignal sensing device; The presence of an intentional control signal; or The absence of the intentional control signal Determining whether to send the biosignal to a classifier by using at least one of, wherein determining the presence of the intentional control signal comprises: Detecting a manual intent override signal from the smart device, and Determining, at least in part, from the received biosignal that the user intends to fixate on at least one of the rendered stimuli, including at least one of; On the condition that the intentional control signal is present: Sending the received biosignal to the classifier; On the condition that the intentional control signal is absent: continuing to receive the received biosignals from the user; using the classifier to classify the received biosignals based on the modified stimulus, resulting in a classified selection; returning the classified selection to the user application; A method.

Images