Neurally augmented virtual interface

Abstract

Techniques for using neural data to perform an action are presented. The techniques can include: receiving first digitized neural data representing first neural sensor data; passing the first digitized neural data to a first trained machine learning system, where the first trained machine learning system outputs a selection indication; selecting, based on the selection indication, a second trained machine learning system and an application; receiving second digitized neural data representing second neural sensor data; passing the second digitized neural data to the second trained machine learning system, where the second trained machine learning system outputs a control indication; and passing the control indication to the application, where the application performs an action based on the control indication.

Description

NEURALLY AUGMENTED VIRTUAL INTERFACEGovernment Support

[0001] This invention was made with government support under grant no. NS114439, awarded by the National Institutes of Health. The government has certain rights in the invention.Cross-Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No 63 / 611 ,976, filed on December 19, 2023, the disclosure of which is incorporated herein by reference.Field

[0003] This disclosure relates generally to communications with patients having locked-in syndrome and similar movement deficiencies.Background

[0004] Locked-in syndrome patients lose the ability to control nearly all motor functions, but remain cognitively intact. In particular, locked-in syndrome patients may be unable to vocalize, or have very limited vocalization ability. Existing solutions for providing a locked-in patient with the ability to communicate rely on the patient’s eye movements, for example, which can degrade over time.Summary

[0005] According to various embodiments, a machine learning method of using neural data to perform an action is presented. The method includes: receiving first digitized neural data representing first neural sensor data; passing the first digitized neural data to a first trained machine learning system, where the first trained machine learning system outputs a selection indication; selecting, based on the selection indication, a second trained machine learning system and an application; receiving second digitized neural data representing second neural sensor data; passing the second digitized neural data to the second trained machine learning system, where the second trained machine learning system outputs a control indication; and passing the control indication to the application, where the application performs an action based on the control indication.

[0006] Various optional features of the above method embodiments include the following. The method may include: obtaining the first neural sensor data from at least one cortical implant; and obtaining the second neural sensor data from the at least one cortical implant. The method may include: obtaining the first neural sensor data from at least one on-scalp sensor; and obtaining the second neural sensor data from the at least one on-scalp sensor. The first neural sensor data may represent a first physical gesture, and the second neural sensor data may represent a second physical gesture. The first neural sensor data may represent a first word, and where the second neural sensor data may represent a second word. The method may include: displaying on a web page a plurality of icons, where the plurality of icons includes an icon for the application, and where the selection indication includes an identification of the application. The method may include authenticating a user. The application may include at least one of: an appliance application, an illumination application, athermostat application, or a keyboard application. The action may include at least one of: turning an appliance on or off, turning an illumination source on or off, adjusting a thermostat, or operating a keyboard. The control indication may include a direct navigation command.

[0007] According to various embodiments, a non-transitory computer readable medium including instructions that, when executed by an electronic processor, configure the electronic processor to use neural data to perform an action by performing operations is presented. The operations include: receiving first digitized neural data representing first neural sensor data; passing the first digitized neural data to a first trained machine learning system, where the first trained machine learning system outputs a selection indication; selecting, based on the selection indication, a second trained machine learning system and an application; receiving second digitized neural data representing second neural sensor data; passing the second digitized neural data to the second trained machine learning system, where the second trained machine learning system outputs a control indication; and passing the control indication to the application, where the application performs an action based on the control indication.

[0008] Various optional features of the above computer readable medium embodiments include the following. The operations may further include: obtaining the first neural sensor data from at least one cortical implant; and obtaining the second neural sensor data from the at least one cortical implant. The operations may further include: obtaining the first neural sensor data from at least one on-scalp sensor; and obtaining the second neural sensor data from the at least one on-scalp sensor. The first neural sensor data may represent a first physical gesture, and where the second neural sensor data may represent a second physical gesture. The first neuralsensor data may represent a first word, and the second neural sensor data may represent a second word. The operations may further include: displaying on a web page a plurality of icons, where the plurality of icons includes an icon for the application, and where the selection indication includes an identification of the application. The operations may further include authenticating a user. The application may include at least one of: an appliance application, an illumination application, a thermostat application, or a keyboard application. The action may include at least one of: turning an appliance on or off, turning an illumination source on or off, adjusting a thermostat, or operating a keyboard. The control indication may include a direct navigation command.

[0009] Combinations, (including multiple dependent combinations) of the above-described elements and those within the specification have been contemplated by the inventors and may be made, except where otherwise indicated or where contradictory.Brief Description of the Drawings

[0010] Various features of the examples can be more fully appreciated, as the same become better understood with reference to the following detailed description of the examples when considered in connection with the accompanying figures, in which:

[0011] Fig. 1 is a schematic diagram of a system for using neural data to perform an action, according to various embodiments;

[0012] Figs. 2A and 2B illustrate a screenshot of, and flow diagrams for, a standby screen of a web interface for a system for using neural data to perform an action, according to various embodiments;

[0013] Figs. 3A and 3B illustrate a screenshot of, and a flow diagram for, a home screen of a web interface for a system for using neural data to perform an action, according to various embodiments;

[0014] Fig. 4A and 4B illustrate a screenshot of, and a flow diagram for, a communications board screen of a web interface for a system for using neural data to perform an action, according to various embodiments;

[0015] Figs. 5A and 5B illustrates a screenshot of, and a flow diagram for, a messaging screen of a web interface for a system for using neural data to perform an action, according to various embodiments;

[0016] Figs. 6A and 6B illustrates a screenshot of, and a flow diagram for, a television remote control screen of a web interface for a system for using neural data to perform an action, according to various embodiments;

[0017] Fig. 7 illustrates a screenshot of a research screen of a web interface for a system for using neural data to perform an action, according to various embodiments; and

[0018] Fig. 8 illustrates a method of using neural data to perform an action, according to various embodiments.Description of the Examples

[0019] Reference will now be made in detail to example implementations, illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific exemplary examples in which the invention may be practiced. These examples are described in sufficientdetail to enable those skilled in the art to practice the invention and it is to be understood that other examples may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0020] Some embodiments provide to patients with locked-in syndrome, or other movement deficiencies, a method of restored communication. Some embodiments provide movement deficiency patients with the ability to utilize neural control mechanisms to interact with a web application, allowing them to interact with their environment without overt motor control. Some embodiments use control signals provided by neural signal decoders. Because cognition is not impacted by locked-in syndrome, it is expected that this control mechanism will function better than eye control at later stages of neurodegenerative diseases, such as locked-in syndrome.

[0021] In general, a brain computer interface may include several components, such as a hardware device implant in the brain (or placed on the skull) to collect data, a technique for converting such signals to a digital format and processing them, and an end-user application for the patient to interact with. Some embodiments provide an end-user application for movement deficiency patients to interact with based on neural control mechanisms, so as to effectuate communication and other actions.

[0022] Some embodiments include an interface between a patient's implanted or external neural electrodes and a web application. Some embodiments include or utilize a brain computer interface, which allows the patient to interact with a web application to communicate with and control aspects of the outside world via neural signaling. Additionally, according to some embodiments, the web application allows researchers to collect data via the presentation of audio and visual stimuli that instruct the patient on how to perform a task. Such collected data may be used for researchpurposes, and / or to train one or more machine learning systems (e.g., neural networks) to effectuate neural signal based communications and actions.

[0023] These and other features and advantages are shown and described herein in reference to the figures.

[0024] Fig. 1 is a schematic diagram of a system 100 for using neural data to perform an action, according to various embodiments. In general, the system 100 may be conceptionalized as including four parts. A first part, which acquires and processes neural data, includes neural data acquisition components 102. A second part, which convert neural data to purpose-driven signals for controlling applications and / or hardware, includes neural decoders 110. A third part, which serves communications and other functionality and interfaces with third-party application, includes a server 120. And a fourth part includes a user interface, which may be implemented as a web page for browser navigation 140 according to various embodiments.

[0025] The neural data acquisition components 102 include brain electrodes 104, analog-to-digital conversion 106, and data formatting and preprocessing 108. The brain electrodes 104 may be internal (e.g., implanted) or external (e.g., positioned on or coupled with the patient’s scalp). According to some embodiments, the brain electrodes 104 include 128 internal electrodes on the surface of the patient’s brain. According to some embodiments, the brain electrodes 104 include between 64 and 128 external electrodes on the patient’s scalp, e.g., on the patient’s hair and coupled to the scalp using saline or gel. The brain electrodes 104 are electrically coupled to an analog-to-digital converter for analog-to-digital conversion 106. Formatting and preprocessing 108 can include noise reduction, for example.

[0026] The neural decoders 110 process the neural data and generate corresponding signals that represent higher-level information. According to someembodiments, the neural decoders 110 include a plurality of trained machine learning systems, e.g., artificial neural networks. Each machine learning system is trained using neural data and corresponding indications (e.g., selection or control indications) applicable to a respective application. For example, a machine learning system may be trained with neural data (e.g., representing gestures or words) and corresponding indications (e.g., representing selection or control of corresponding hardware or software elements). Once trained, each machine learning system may accept as an input neural data and produce a corresponding output that includes an indication. The output indication may be in the form of an electrical signal representing a specific action, such as a selection action or a control action. The output indication may be specific to a corresponding application, such that there is a direct correspondence (e.g., a one-to-one correspondence) between possible outputs of the trained machine learning system and control options of the corresponding application. In general, as described in detail herein, a user may use a first neural decoder of the neural decoders 110 to select both a second neural decoder of the neural decoders 110 and a corresponding application that is responsive to the outputs of the second neural decoder.

[0027] According to various embodiments, the neural decoders 110 may acquire and interpret various types of neural data. According to various embodiments, one or more of the neural decoders 110 may acquire neural data representing various physical gestures. Such gestures may include, for example, finger movements, hand movements, arm movements, head movements, foot movements, leg movements, etc. Because the user may be suffering from a movement deficiency, such as locked- in syndrome, the patient themselves may not physical move, or may not appreciably physically move, in response to the neural impulses provided by the patient’s brainthat would otherwise produce a corresponding physical movement. According to various embodiments, one or more of the neural decoders 110 may acquire neural data representing vocalizations of various words. Such words may include, for example, command words (e.g., on, off, etc.) and selection words (e.g., lights, temperature, etc.). Because the user may be suffering from a movement deficiency, such as locked-in syndrome, the patient themselves may not actually produce audible words, or may not produce significant vocalizations, in response to the neural impulses provided by the patient’s brain that would otherwise produce a corresponding vocalized word.

[0028] The server 120 provides a user interface in the form of browser navigation 140, coordinates user-selected applications with associated neural decoders 110, and controls selected applications according to user signals provided by the neural data acquisition 102 and interpreted by the respective neural decoder. The server 120 may be implemented in a remote computer or a computer local to a user. Thus, according to some embodiments, the server 120 may be implemented on a personal computer of a user. The server 120 and the neural decoders 110 are communicatively coupled with a bi-directional communication channel for state information. Through the exchanged state information, the neural decoders 110 and server 120 remain in sync with respect to the particular neural decoder to use for the application being served by the server. The server 120 also receives neural data directly from the neural data acquisition components 102. The server 120 is also communicatively coupled to applications 154 via third-party application program interfaces (APIs) 152. Example applications are shown and described herein in reference to Figs. 2-7. The server 120 servers a web page for browser navigation 140 by a user. The web page may be implemented by a computer of the end user, e.g.,the same computer that implements the server 120. The server 120 serves data visualization 132 and can accept real-world data 134. The data visualization 132 can include indications of applications and control data. The real-world data 134 can include data from various sensors and state information, e.g., temperature, light conditions, humidity, or hardware status information (e.g., on / off, active / inactive, etc.).

[0029] The system 100 may be used to implement a method of using neural data to perform an action, as shown and described presently in reference to Figs. 2-8.

[0030] Figs. 2A and 2B illustrate a screenshot 202 of, and flow diagrams 204, 206 for, a standby screen of a web interface for a system for using neural data to perform an action, according to various embodiments. According to various embodiments, the system may run continuously when operational, but may implement a standby mode by user selection or after a period of inactivity. The screenshot 202 illustrates an example display by the system when in standby mode. When the user initiates an alarm command (e.g., by thinking the word “Alarm”), the system initiates the alarm sequence, as shown by the flow diagram 206. The alarm sequence may include any, or a combination, of the following actions: text a designated caregiver, turn on one or more lights, and / or provide an audible indication (e.g., through the computer or a connected audio system) of, for example, “Alert! Alert! Alert!” When the user initiates a wake command, either by providing neural data corresponding to a gesture or a word, (e.g., by thinking a sequence of particular words such as “NAVI NAVI NAVI”) the system switches to active mode, as shown by the flow diagram 204. According to some embodiments, when the user provides any neural data, or provides specific neural data for the wake command, the system may transition to the home screen state as shown and described herein in reference to Fig 2A.

[0031] Figs. 3A and 3B illustrate a screenshot 302 of, and a flow diagram 304 for, a home screen of a web interface for a system for using neural data to perform an action, according to various embodiments. The screenshot 302 depicts various options that a user may activate from the home screen. Such options include, by way of non-liming examples: home, television, music, message, light temperature, confirm, alarm, board, volume, exit, camera, NAVI, and back. A user may activate any of these option by providing neural data representing a word or gesture, as shown by the flow diagram 304. For a gesture-driven selection, the user may provide gestures that move and select using a cursor. In general, from the home screen, the user may select any of a variety of applications, non-limiting examples of which are shown and described herein in reference to Figs. 4-7. Note that the home screen may have one or more associated respective neural decoders, corresponding to the particular navigation and selection technique. Note further that activating the “NAVI” option will return the application to the context of the home screen of Fig. 3A. For example, if the user activates the “Television” option, then all of the neural decode signals will control the television rather than the home screen. Thus, the user may activate the “NAVI” option so that the neural decoder will once again be controlling, for instance, the home screen of Fig. 3A.

[0032] Within a particular application, such as the home screen of Fig. 3A or any of the applications as shown and described herein in reference to any of Figs. 4- 7, the user may use any of a variety of navigation techniques, which may be based on neural data representing a gesture and / or word. For example, the user may utilize direct navigation, e.g., by providing neural data representing a word corresponding to a selection or command. As another example, the user may utilize discrete navigation, e.g., where a cursor automatically moves between various options, pausing at eachoption, and the user may select the highlighted option by providing neural data corresponding to a particular gesture or word corresponding to a selection. As yet another example, the user may utilize six-way navigation, which can, for example, move a cursor to highlight and select from among various options, where the navigation options include up, down, left, right, enter, and back.

[0033] Each of these navigation techniques may be associated with one or more particular neural decoders, e.g., one or more trained artificial neural networks or other trained machine learning systems, each of which maps the neural data, whether representing to a word or a gesture, to the corresponding selection or control indication.

[0034] According to various embodiments, the user may customize the options displayed on the home screen, e.g., by adding or deleting icons representing options. The user may do so by dragging icons, for example. When an option is added, the system will automatically add and associate a corresponding neural decoder, which will interpret neural data for controlling actions within the option. Using the neural decoder, the user may associate particular neural data, representing a gesture or word, with each of the available actions within the option.

[0035] Figs. 4A and 4B illustrate a screenshot 402 of, and a flow diagram 404 for, a communications board screen of a web interface for a system for using neural data to perform an action, according to various embodiments. The screen shot 402 depicts icons corresponding to the following applications and communications options, by way of non-limiting examples: afraid, pain, yes, no, sad, tired, nurse, doctor, feel sick, frustrated, short of breath, choking, angry, dizzy, hot, cold, how am I doing?, what is happening? what time?, come back later, bed up, bed down, home, tv / video, light on / off, alarm, glasses, water, suction, lips moisturized, sleep, and sound off. The usermay utilize direct navigation, discrete navigation, or six-way navigation to navigate to and select any such icon, as shown by the flow diagram 404.

[0036] Figs. 5A and 5B illustrate a screenshot 502 of, and a flow diagram 504 for, a messaging screen of a web interface for a system for using neural data to perform an action, according to various embodiments. The messaging screen allows a user to generate messages, which may be delivered audibly and / or visibly via a computer, and / or sent via text message, to a message recipient. The messaging screen may include spellcheck and / or word completion capabilities. The user may utilize direct navigation, discrete navigation, or six-way navigation to navigate to and select any letter or suggested word, as shown by the flow diagram 504.

[0037] Figs. 6A and 6B illustrate a screenshot 602 of, and a flow diagram 604 for, a television remote control screen of a web interface for a system for using neural data to perform an action, according to various embodiments. The screenshot 602 may further depict an image of television remote control options, e.g., channel up / down, volume, etc. The screenshot shows that the user may activate any of the following navigation possibilities, which may be applied to the remote control options: up, down, left, right, enter, and back. The user may utilize direct navigation, discrete navigation, or six-way navigation to navigate to and select any of the navigation possibilities for the remote control, as shown by the flow diagram 604.

[0038] Fig. 7 illustrates a screenshot 702 of a research screen of a web interface for a system for using neural data to perform an action, according to various embodiments. According to various embodiments, the system may include a research mode, through which data may be collected across a wide variety of applications. The data collection may be trial-based or event-driven, by way of non-limiting example. Shown in the screenshot 702 is a finger movement task, where the screenshot 702illustrates a finger that the user is directed to attempt to move. The research mode then collects the corresponding data, which may be used for a variety of purposes, such as training one or more of the neural decoders. Other research mode possibilities include, by way of non-limiting example: center-out motor tasks, syllable repetition tasks, sentence reading tasks, and keyword training.

[0039] Fig. 8 illustrates a method 800 of using neural data to perform an action, according to various embodiments. The method 800 may be implemented using a system such as the system 100 as shown and described herein in reference to Fig. 1 . The method may be used to navigate to, and activate various options of, any of a variety of applications, such as are shown and described herein in reference to Figs. 2-7, by way of non-limiting examples.

[0040] At 802, the method 800 includes receiving first digitized neural data representing first neural sensor data. By way of non-limiting examples, the actions of 802 may be performed at a standby screen, such as is shown and described herein in reference to Fig. 2A, a home screen, such as is shown and described herein in reference to Fig. 3A, or a communications board screen, such as is shown and described herein in reference to Fig. 4A. The first digitized neural data may represent a gesture or a word. The first digitized neural data may be received from neural data acquisition components, such as are shown and described herein in reference to Fig. 1 , for example.

[0041] At 804, the method 800 includes passing the first digitized neural data to a first trained machine learning system, where the first trained machine learning system outputs a selection indication. The first trained machine learning system may be a neural decoder, such as is shown and described herein in reference to Fig. 1 . The first trained machine learning system may correspond to, by way of non-limitingexample, a standby screen, such as is shown and described herein in reference to Fig. 2A, a home screen, such as is shown and described herein in reference to Fig. 3A, or a communications board screen, such as is shown and described herein in reference to Fig. 4A. The first trained machine learning system may output, as a result of inputting the first digitized neural data, a selection indication, which may correspond to an option, such as an application, provided on a screen displayed by the system, e.g., as shown and described herein in reference to Figs. 2A, 3A and 4A.

[0042] At 806, the method 800 includes selecting, based on the selection indication, a second trained machine learning system and an application. The actions of 806 may include determining and activating a neural decoder that corresponds to the selection indication of 804. The neural decoder may be associated with a selected application or communication option, as shown and described herein in reference to Fig. 1 , for example.

[0043] At 808, the method 800 includes receiving second digitized neural data representing second neural sensor data. By way of non-limiting examples, the actions of 808 may be performed at a communications board screen, such as is shown and described herein in reference to Fig. 4A, a messaging screen, such as is shown and described herein in reference to Fig. 5A, or a remote control screen, such as is shown and described herein in reference to Fig. 6A. The second digitized neural data may represent a gesture or a word. The second digitized neural data may be received from neural data acquisition components, such as are shown and described herein in reference to Fig. 1 , for example.

[0044] At 810, the method 800 includes passing the second digitized neural data to the second trained machine learning system, where the second trained machine learning system outputs a control indication. The second trained machinelearning system may be a neural decoder, such as is shown and described herein in reference to Fig. 1 . The second trained machine learning system may correspond to, by way of non-limiting example, a communications board screen, such as is shown and described herein in reference to Fig. 4A, a messaging screen, such as is shown and described herein in reference to Fig. 5A, or a remote control screen, such as is shown and described herein in reference to Fig. 6A. The second trained machine learning system may output, as a result of inputting the second digitized neural data, a control indication, which may correspond to an option provided on a screen displayed by the system, such as, by way of non-limiting examples, is shown and described herein in reference to Figs. 4, 5 and 6.

[0045] At 812, the method 800 includes passing the control indication to the application of 810, where the application performs an action based on the control indication. The action may correspond to an option provided on a screen displayed by the system, such as, by way of non-limiting examples, is shown and described herein in reference to Figs. 4, 5 and 6.

[0046] Although embodiments are disclosed with respect to artificial neural networks as examples of trained machine learning systems, embodiments are not so limited. Other suitable machine learning systems include, by way of non-limiting examples: decision trees, random forests, Bayesian networks, support vector machines, etc.

[0047] Certain examples can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program (s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files.Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory, and magnetic or optical disks or tapes.

[0048] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented using computer readable program instructions that are executed by an electronic processor.

[0049] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the electronic processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0050] In embodiments, the computer readable program instructions may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the C programming language or similar programming languages. The computer readable program instructions may execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

[0051] As used herein, the terms “A or B” and “A and / or B” are intended to encompass A, B, or {A and B}. Further, the terms “A, B, or C” and “A, B, and / or C” are intended to encompass single items, pairs of items, or all items, that is, all of: A, B, C, {A and B}, {A and C}, {B and C}, and {A and B and C}. The term “or” as used herein means “and / or.”

[0052] As used herein, language such as “at least one of X, Y, and Z,” “at least one of X, Y, or Z,” “at least one or more of X, Y, and Z,” “at least one or more of X, Y, or Z,” “at least one or more of X, Y, and / or Z,” or “at least one of X, Y, and / or Z,” is intended to be inclusive of both a single item (e.g., just X, or just Y, or just Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase “at least one of” and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.

[0053] The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature thatdemonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [performing [a function]...” or “step for [performing [a function]...”, it is intended that such elements are to be interpreted under 35 U.S.C. § 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. § 112(f).

[0054] While the invention has been described with reference to the exemplary examples thereof, those skilled in the art will be able to make various modifications to the described examples without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.

Claims

What is claimed is:1 . A machine learning method of using neural data to perform an action, the method comprising: receiving first digitized neural data representing first neural sensor data; passing the first digitized neural data to a first trained machine learning system, wherein the first trained machine learning system outputs a selection indication; selecting, based on the selection indication, a second trained machine learning system and an application; receiving second digitized neural data representing second neural sensor data; passing the second digitized neural data to the second trained machine learning system, wherein the second trained machine learning system outputs a control indication; and passing the control indication to the application, wherein the application performs an action based on the control indication.

2. The method of claim 1 , further comprising: obtaining the first neural sensor data from at least one cortical implant; and obtaining the second neural sensor data from the at least one cortical implant.

3. The method of claim 1 , further comprising: obtaining the first neural sensor data from at least one on-scalp sensor; andobtaining the second neural sensor data from the at least one on-scalp sensor.

4. The method of claim 1 , wherein the first neural sensor data represents a first physical gesture, and wherein the second neural sensor data represents a second physical gesture.

5. The method of claim 1 , wherein the first neural sensor data represents a first word, and wherein the second neural sensor data represents a second word.

6. The method of claim 1 , further comprising: displaying on a web page a plurality of icons, wherein the plurality of icons comprises an icon for the application, and wherein the selection indication comprises an identification of the application.

7. The method of claim 1 , further comprising authenticating a user.

8. The method of claim 1 , wherein the application comprises at least one of: an appliance application, an illumination application, a thermostat application, or a keyboard application.

9. The method of claim 1 , wherein the action comprises at least one of: turning an appliance on or off, turning an illumination source on or off, adjusting a thermostat, or operating a keyboard.

10. The method of claim 1 , wherein the control indication comprises a direct navigation command.

11. A non-transitory computer readable medium comprising instructions that, when executed by an electronic processor, configure the electronic processor to use neural data to perform an action by performing operations comprising: receiving first digitized neural data representing first neural sensor data; passing the first digitized neural data to a first trained machine learning system, wherein the first trained machine learning system outputs a selection indication; selecting, based on the selection indication, a second trained machine learning system and an application; receiving second digitized neural data representing second neural sensor data; passing the second digitized neural data to the second trained machine learning system, wherein the second trained machine learning system outputs a control indication; and passing the control indication to the application, wherein the application performs an action based on the control indication.

12. The non-transitory computer readable medium of claim 11 , wherein the operations further comprise: obtaining the first neural sensor data from at least one cortical implant; and obtaining the second neural sensor data from the at least one cortical implant.

13. The non-transitory computer readable medium of claim 11 , wherein the operations further comprise: obtaining the first neural sensor data from at least one on-scalp sensor; and obtaining the second neural sensor data from the at least one on-scalp sensor.

14. The non-transitory computer readable medium of claim 11 , wherein the first neural sensor data represents a first physical gesture, and wherein the second neural sensor data represents a second physical gesture.

15. The non-transitory computer readable medium of claim 11 , wherein the first neural sensor data represents a first word, and wherein the second neural sensor data represents a second word.

16. The non-transitory computer readable medium of claim 11 , wherein the operations further comprise: displaying on a web page a plurality of icons, wherein the plurality of icons comprises an icon for the application, and wherein the selection indication comprises an identification of the application.

17. The non-transitory computer readable medium of claim 11 , wherein the operations further comprise authenticating a user.

18. The non-transitory computer readable medium of claim 11 , wherein the application comprises at least one of: an appliance application, an illumination application, a thermostat application, or a keyboard application.

19. The non-transitory computer readable medium of claim 11 , wherein the action comprises at least one of: turning an appliance on or off, turning an illumination source on or off, adjusting a thermostat, or operating a keyboard.

20. The non-transitory computer readable medium of claim 11 , wherein the control indication comprises a direct navigation command.

Images