Systems and methods for neurohaptic signals

Abstract

A system for providing a neuro-haptic interface is provided. The system can include a memory unit configured to store computer instructions and at least one processor configured to execute the instructions. The instructions can cause a device to receive a notification from a peripheral device, determine a neuro-haptic effect to be performed according to the notification, generate a neuro-haptic signal corresponding to the neuro-haptic effect, and provide the neuro-haptic signal to a neuro-haptic device to render the neuro-haptic effect by stimulating a user's peripheral nervous system.

Description

[Technical Field]

[0001] The present disclosure relates to neuro-haptic signals and devices, systems, and methods for generating and providing neuro-haptic signals. [Background technology]

[0002] Haptic feedback is mediated touch between humans, between humans and robots, or even between robots. Touch is mediated using a computational device that allows haptic feedback to be programmable, as opposed to touch feedback provided by everyday objects. A haptic device is therefore a device that provides haptic feedback. Haptic feedback can be divided into two types of feedback: cutaneous feedback, which targets the skin to provide vibration, temperature, pain, etc., and kinesthetic feedback, which targets movement of the body's extremities (e.g., fingers, arms, legs, or the whole body).

[0003] Haptic devices are common in most parts of the world today and can be seen in mobile devices that provide tactile feedback in the form of vibrations when a user touches the touchscreen of a smart mobile phone device; such devices command an internal vibration actuator (e.g., a linear resonant actuator or LRA) to produce vibrations that stimulate the sensation of touch on the user's fingers.

[0004] Some gaming devices provide haptic feedback. For example, game controller devices used in gaming consoles use eccentric rotating mass (ERM) actuators or voice coil motor (VCM) actuators to provide vibration sensations.

[0005] The latest generation of game controllers now include haptic triggers with semi-kinesthetic feedback, meaning that the triggers can be pushed or pulled with a small amount of force. Another device that provides haptic feedback is the haptic glove, which is a glove equipped with actuators that vibrate or move the hand or fingers.

[0006] All commonly used haptic devices on the market rely on electromechanical components to produce both tactile and kinesthetic sensations.

[0007] Haptic devices are used in fields such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), all known as extended reality (XR), as well as in the metaverse, where interaction with virtual objects or any kind of digital information requires haptic feedback to create a truly immersive experience.

[0008] All of these immersive environments require the use of headsets or glasses that present an entirely virtual world or that overlay virtual objects and information within the real world when worn by the user. Summary of the Invention [Problem to be solved by the invention]

[0009] Currently, interaction within these immersive environments is performed using hand controllers equipped with vibration motors to provide users with vibrotactile feedback. The controllers are used to track the movement and position of the hands. However, because the user's hands are holding the controller, preventing them from manually exploring the immersive environment, the interaction is not natural and the user must learn how to use the controller, which is detrimental to some applications where the user needs to focus on learning something else, adding to the user's cognitive load.

[0010] Other systems use gloves that allow hands-free interaction within these immersive environments because the gloves contain sensors that track hand movement and position and have actuators that provide cutaneous or kinesthetic feedback. However, these gloves cannot be worn for extended periods of time because they cause the user to sweat and make the gloves uncomfortable. Another problem with gloves is that they are not one size fits all, are difficult to put on, take time to adapt to the hand, and sometimes require the assistance of another person to put them on.

[0011] Other immersive systems use none, either using cameras in the room or in a headset to directly track the hands, or using wearable bracelets to "estimate" the movement and position of the fingers and hands. The problem with avoiding the use of controllers or gloves is that the system can no longer provide haptic feedback at the fingertips or hands. [Means for solving the problem]

[0012] In one embodiment, a system for providing a neuro-haptic interface is provided. The system may include a memory unit configured to store computer instructions and at least one processor, where the processor is configured to execute the instructions to receive a notification from a peripheral device, determine a neuro-haptic effect to be performed according to the notification, generate a neuro-haptic signal corresponding to the neuro-haptic effect, and provide the neuro-haptic signal to a neuro-haptic device to render the neuro-haptic effect by stimulating a user's peripheral nervous system. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates a neuro-haptic system consistent with embodiments herein. [Figure 2]FIG. 1 illustrates a neuro-haptic system as incorporated into a variety of different user-wearable devices. [Figure 3] FIG. 1 illustrates the ulnar and median nerves of the arm. [Figure 4] FIG. 1 illustrates an example of a neuro-haptic system embodied as a bracelet or watch with an electrode pattern consistent with embodiments herein. [Figure 5] FIG. 1 illustrates an example of a neuro-haptic system embodied as a bracelet or watch with an electrode pattern consistent with embodiments herein. [Figure 6] FIG. 1 illustrates an example of a neuro-haptic system embodied as a bracelet or watch with an electrode pattern consistent with embodiments herein. [Figure 7] FIG. 1 illustrates an example of a neuro-haptic system embodied as a bracelet or watch with an electrode pattern consistent with embodiments herein. [Figure 8] FIG. 1 illustrates an example of a neuro-haptic system embodied as a bracelet or watch with an electrode pattern consistent with embodiments herein. [Figure 9] FIG. 1 illustrates the use of a neuro-haptic system according to embodiments herein. [Figure 10] FIG. 1 illustrates the use of a neuro-haptic system according to embodiments herein. [Figure 11] FIG. 1 illustrates the use of a neuro-haptic system according to embodiments herein. [Figure 12] 10 is a flowchart illustrating steps of a method for providing neuro-haptic notification of an alert effect. [Figure 13A] FIG. 1 illustrates a neuro-haptic system 100 used to detect neural signals and provide a neuro-haptic signal as an output. [Figure 13B] FIG. 1 illustrates a neuro-haptic system 100 used to detect neural signals and provide a neuro-haptic signal as an output. [Figure 13C] FIG. 1 illustrates a neuro-haptic system 100 used to detect neural signals and provide a neuro-haptic signal as an output. [Figure 13D] FIG. 1 illustrates a neuro-haptic system 100 used to detect neural signals and provide a neuro-haptic signal as an output. [Figure 14] 10 is a flowchart illustrating steps in various methods for generating and providing neurohaptic effects associated with an interactive user system. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present disclosure provides systems, devices, and methods for providing haptic feedback or sensation via a neural interface.

[0015] The nervous system is a complex system consisting of a network of nerve cells and fibers that coordinates the body's movements and sensory information and transmits signals to and from various parts of the body. A neural interface is a device that interacts with the nervous system through receiving or recording information from the nervous system (e.g., by sensing nerve impulses) and / or stimulating the nervous system (e.g., by providing or otherwise directing nerve impulses). As referred to herein, the "nervous system" may include either the central nervous system (CNS) or the peripheral nervous system (PNS) and all their components, including at least motor (efferent) neurons, sensory (afferent) neurons, spinal nerves, and the brain. Neural interfaces consistent with the present disclosure may be implanted within the body (invasively relative to the body) or located on the surface of the body (externally or non-invasively relative to the body).

[0016] There are a variety of techniques and methods that can be used to interact with the nervous system using a variety of principles: some techniques read from the nervous system, others write to or stimulate the nervous system, and still others that can both read and write to the nervous system.

[0017] Table 1 provides a list of technologies that may be used to read information from the nervous system. These techniques may be electrical, magnetic, light, and / or sound sensitive. [Table 1]

[0018] Table 2 provides a list of techniques that may be used to transmit information to the nervous system, for example, to stimulate nerves. These techniques may use electricity, magnetic fields, light, and / or sound. [Table 2]

[0019] The neural interface technologies listed above may interact with different functions within the human body, including vision, hearing, speech, and touch. The embodiments described herein may include any of the neural interface technologies listed above, or any other suitable neural interface technology.

[0020] Some specific embodiments discussed herein involve the use of neural interface technologies to interact with the human sense of touch. For example, it has been shown that sound or ultrasound can generate the sensation of touch when a beam of ultrasound signals is targeted to the somatosensory cortex of the human brain.

[0021] Other specific embodiments described herein may include optogenetic techniques, which are biological techniques that control neuronal activity with light to target the somatosensory cortex and induce the sensation of touch. In further embodiments of the present disclosure, techniques used to stimulate the nervous system include the use of electricity directed at either neurons in the brain or neurons in the peripheral nervous system. Thus, the systems, devices, and methods described herein may generate the sensation of touch by applying different excitation modes to different parts of the body using light, sound, electricity, magnetism, and / or any combination of these techniques.

[0022] 1 illustrates a neuro-haptic system. Neuro-haptic system 100 may include any combination of at least one processing unit (also referred to as a CPU) 101, input / output system 103, visual output device 104, audio output device 105, haptic output device 106, sensor 107, communication interface 108, data bus 102, and memory device 110. The memory device may be configured to store instructions for processing unit 101 to implement various features of neuro-haptic system 101, including at least signal decoder 112, haptic effect creator 114, and haptic effect rendering module 116. Neuro-haptic system 100 may be configured to communicate with other devices either directly (e.g., via Bluetooth or other direct communication technology) or over network 200.

[0023] 1 as a single device, neuro-haptic system 100 may include multiple separate components. For example, various output devices (visual, audio, haptic) may not be co-located with CPU 101, data bus 102, memory 110, and input / output system 103. Similarly, sensor 107 may not be co-located. In various embodiments, each of the components of neuro-haptic system 100 may be co-located or non-co-located in any suitable combination.

[0024] Network 200 may be connected via wired or wireless links. Wired links may include digital subscriber lines (DSL), coaxial cable lines, or fiber optic lines. Wireless links may include Bluetooth®, Bluetooth Low Energy (BLE), ANT / ANT+, ZigBee, Z-Wave, Thread, Wi-Fi®, Worldwide Interoperable Microwave Access (WiMAX®), Mobile WiMAX®, WiMAX® Advanced, NFC, SigFox, LoRa, Random Phase Multiple Access (RPMA), Weightless-N / P / W, infrared channels, or satellite bands. Wireless links may also include any cellular network standard for communicating between mobile devices, including standards that qualify as 2G, 3G, 4G, or 5G. Wireless standards may use various channel access methods, such as FDMA, TDMA, CDMA, or SDMA. In some embodiments, different types of data may be transmitted over different links and standards. In other embodiments, the same type of data may be transmitted over different links and standards. Network communication may occur over any suitable protocol, including, for example, HTTP, TCP / IP, UDP, Ethernet, ATM, etc.

[0025] Network 200 may be any type and / or form of network. The geographic scope of the network may vary widely, and network 200 may be a body area network (BAN), a personal area network (PAN), a local area network (LAN), such as an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of network 200 may be any form, including, for example, point-to-point, bus, star, ring, mesh, or tree. Network 200 may be any such network topology known to those skilled in the art that can support the operations described herein. Network 200 may utilize layers or stacks of different technologies and protocols, including, for example, Ethernet protocols, the Internet Protocol Suite (TCP / IP), ATM (Asynchronous Transfer Mode) techniques, SONET (Synchronous Optical Network) protocols, or SDH (Synchronous Digital Hierarchy) protocols. The TCP / IP Internet Protocol Suite may include an application layer, a transport layer, an Internet layer (e.g., including IPv4 and IPv6), or a link layer. The network 199 may be a type of broadcast network, a telecommunications network, a data communications network, or a computer network.

[0026] Memory 110 includes any type of non-transitory computer-readable storage medium (or media) and / or non-transitory computer-readable storage device. Such computer-readable storage medium or device may store computer-readable program instructions for causing a processor to perform one or more of the methodologies described herein. Examples of computer-readable storage media or devices may include, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof, such as, but not limited to, computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, etc.

[0027] At least one processor 101 (also referred to herein, for convenience, interchangeably as processor(s) 101, processor(s) 101, or processor 101) may include any suitable computer processing unit (CPU). In embodiments, the functionality of the processor may be implemented by hardware (e.g., through an application specific integrated circuit ("ASIC"), programmable gate array ("PGA"), field programmable gate array ("FPGA"), etc.) or by any combination of hardware and software. Storage 120 includes any type of non-transitory computer-readable storage medium(s) and / or non-transitory computer-readable storage device. Such computer-readable storage medium or device may store computer-readable program instructions for causing the processor to perform one or more methodologies described herein. Examples of computer-readable storage media or devices may include, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof, such as, but not limited to, computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, etc.

[0028] Processor 101 may be programmed by one or more computer program instructions stored in storage device 110. For example, processor 101 may be programmed by one or more of signal decoder 112, haptic effect creator 114, and haptic effect rendering module 116, as well as other suitable program modules. It will be understood that the functionality of the various coding modules discussed herein is exemplary and not limiting. Additionally, storage device 110 may function as a data retention system to provide data storage. As used herein, for convenience, various managers or software modules may be described as performing an operation when, in effect, the manager programs processor 101 (and thus neurohaptic system 100) to perform the operation.

[0029] The various components of neuro-haptic system 100 work in concert to receive input from various modalities (as discussed further below) and provide neuro-haptic feedback to the user (e.g., via stimulation of the nervous system).

[0030] Signal decoder 112 is a software protocol that runs on neurohaptic system 100. Signal decoder 112 is configured to receive neurological signals from one or more user devices configured to monitor, sense, detect, or otherwise capture neurological signals within the body. Signal decoder 112 receives one or more neurological signals and decodes them according to several factors, such as device parameters (e.g., type, sensor used, etc.), device location, user factors (e.g., age, size, weight, etc.), environmental context factors (e.g., usage situation, movement, temperature, humidity, etc.), calibration factors, etc. Decoding the neurological signals allows signal decoder 112 to generate signal information including a signal classification (e.g., as motor or sensory), signal strength, and signal properties. The signal properties may include an indication of the type of sensory input that generated the signal, such as pleasure, pain, temperature, vibration, touch, hearing, olfaction, taste, pressure, surface texture, surface curvature, surface friction, surface shape, direction of tangentially applied force, etc.

[0031] Haptic effect creator 114 is a software protocol that runs on neurohaptic system 100. Haptic effect creator 114 is configured to create a neurohaptic effect for output to a user. The haptic effect may be created according to one or more of signal information, context information, and / or intent information. Signal information is discussed above. Context information may include applications that may be running, environmental information, XR environment information used in the simulation, biometric information picked up by the device, hand tremor information, etc. Intent information may include information regarding the purpose or intent of the signal, such as notification intent, information conveyance intent, status conveyance intent, signal modification intent (masking or modifying human tremors in people with Parkinson's disease, preventing diabetics from stepping into dangerous areas), etc.

[0032] Haptic effect rendering module 116 is a software protocol that runs on neuro-haptic system 100. Haptic effect rendering module 116 is configured to generate a haptic signal to be output to neuro-haptic output device 106 to provide neuro-haptic stimulation to a user. Haptic effect rendering module 116 can generate the neuro-haptic signal according to a neuro-haptic effect created by a haptic effect creator and by haptic output parameters associated with neuro-haptic output device 106. For example, the haptic output parameters can include the signal shape (e.g., square, sinusoidal, etc.), polarity (positive, negative, or both), signal frequency and frequency variation over time (rate coding), duty cycle pulse width modulation and time modulation, signal magnitude, signal duration, use of multiple signals (e.g., when using two or more electrodes), consideration of phase between signals, delay between signals, weighting coefficients between signals, etc.

[0033] Input / output system 103 is a system including both hardware and software configured to receive one or more inputs and provide them to CPU 101 (e.g., via data bus 102) and / or store such inputs in memory 110. Input / output system 103 is further configured to provide output to various output devices associated with neuro-haptic system 100.

[0034] The input / output system 103 may receive information from sensors 107. The sensors 107 may include, for example, neural sensors, microphones, cameras, light sensors, motion sensors, pressure sensors, EEG sensors, heart rate sensors, radiation sensors, proximity sensors, humidity sensors, chemical sensors, force sensors, Hall effect sensors, oxygen sensors, gyroscopes, accelerometers, capacitance sensors, inclinometers, LIDAR, colorimeters, infrared sensors, etc. The neural sensors 107 associated with the neuro-haptic system may employ any suitable neural detection technique, including, but not limited to, those shown in Table 1.

[0035] Input / output system 103 may receive information from communications interface 108. Communications interface 108 may include the hardware and software components necessary for external communication with any external device, including any device connected via network 200 of the various forms described above and any device associated with neuro-haptic system 100. In embodiments, communications interface 108 may be configured to communicate with a user device, such as a user's tablet, smartphone, smartwatch, etc., thereby enabling the user's device to engage neuro-haptic system 100 and provide haptic feedback to the user. In further embodiments, neuro-haptic system 100 may be integrated within such a user's device and may provide the full suite of functionality associated with such a device in addition to the functionality described herein.

[0036] Visual output device 104 may be configured to provide visual output to a user and may include any suitable device for outputting visual information, including, but not limited to, an LED display, an LCD display, a CRT display, a light, an E-ink, a plasma screen, an electroluminescent display, an OLED display, an AMOLED display, and a quantum dot display.

[0037] Audio output device 105 may be configured to provide audio output to a user and may include any suitable device for outputting audio information, including, but not limited to, a speaker, a bone conduction speaker, or a direct nerve signal (e.g., via auditory nerve stimulation).

[0038] Haptic output device 106 may be configured to provide a neuro-haptic output to a user, as described further herein. Haptic output device 106 consistent with embodiments herein is described in more detail below. Haptic output device 106 associated with a neuro-haptic system may employ any suitable neurostimulation technique, including, but not limited to, those shown in Table 2.

[0039] FIG. 2 illustrates neuro-haptic system 100 as incorporated into a variety of different user-wearable devices, including, for example, a ring (100F), a bracelet or watch (100B), an ankle bracelet (100E), a belt (100C), a necklace (100G, not shown), an ear device (100H, not shown), eyeglasses (100A), a vest with spinal electrodes (100D), a leg band (100I, not shown), or pants with multiple electrodes (100J, not shown). Each of these devices may house all or a portion of neuro-haptic system 100. In embodiments, any or all of these devices may be combined to enhance the input / output capabilities of neuro-haptic system 100. In embodiments, any or all of these devices, in any combination, may act as peripherals to provide input / output capabilities to neuro-haptic system 100, where several components of the neuro-haptic system are housed within a central unit.

[0040] 3-9 provide examples of haptic neural interfaces with peripheral nerves in a user's arm. FIG. 3 illustrates the ulnar and median nerves of the arm. The ulnar nerve 302 and median nerve 301 provide motor innervation to various muscles 306 in the arm and hand, and also provide sensory innervation to various parts of the arm and hand. FIG. 3 further illustrates ligaments 305, tendons 304, and bones 303. While specific depictions of the nerves of the arm are shown in FIGS. 3-9, these illustrations are provided by way of example only, and the invention described herein is not limited thereby. As discussed above, a neuro-haptic system 100 consistent with the present disclosure may be configured to stimulate various nerves throughout the body using a variety of techniques.

[0041] 4 illustrates an example of neuro-haptic system 100 (or components thereof) embodied as a bracelet (or watch) 100B. Neuro-haptic system 100B can include one or more electrodes 401 arranged around the arm. Electrode 401 is configured to stimulate one or more of ulnar nerve 302 and median nerve 301 when activated. Electrode 401 is configured for transcutaneous nerve stimulation. Electrode 401 can be considered to be, or belong to, haptic output device 106 of neuro-haptic system 100B.

[0042] FIG. 5 illustrates an example of neurohaptic system 100 (or its components) embodied as a bracelet (or watch) 100B. Neurohaptic system 100B may include one or more electrodes 401 positioned around the arm for transcutaneous stimulation, as well as one or more subcutaneous electrodes 501. Subcutaneous electrodes 501 are configured for subcutaneous stimulation of one or more of ulnar nerve 302 or median nerve 301. Subcutaneous electrodes 501 may not be physically connected to neurohaptic system 100B and may receive signals wirelessly for neural stimulation. While this embodiment is shown with both transcutaneous electrodes 401 and subcutaneous electrodes 501, it may also be implemented using only subcutaneous electrodes 501.

[0043] FIG. 6 illustrates an example of neurohaptic system 100 (or its components) embodied as a bracelet (or watch) 100B. Neurohaptic system 100B may include one or more externally placed electrodes 401 for transcutaneous stimulation, as well as one or more subcutaneous electrodes 501. Subcutaneous electrodes 501 are configured for subcutaneous stimulation of one or more of ulnar nerve 302 or median nerve 301. Subcutaneous electrodes 501 may be physically connected to neurohaptic system 100B, for example, via conduit 502, to percutaneous electrodes 401. Subcutaneous electrodes 501 may receive signals for neural stimulation via conduit 502. While this embodiment is shown with both percutaneous electrodes 401 and subcutaneous electrodes 501, it may also be implemented using only subcutaneous electrodes 501.

[0044] FIG. 7 illustrates an example of neurohaptic system 100 (or its components) embodied as a bracelet (or watch) 100B. Neurohaptic system 100B may include one or more externally positioned electrodes 401 for transcutaneous stimulation, as well as one or more nerve cuff electrodes 601. Nerve cuff electrode 601 is configured for localization on (e.g., partially or completely enveloping) and stimulation of one or more of ulnar nerve 302 or median nerve 301. Nerve cuff electrode 601 may include a single electrode or an array of electrodes. Subcutaneous electrode 601 may be wirelessly connected to neurohaptic system 100B, for example, to transcutaneous electrode 401, or to any other signal-providing component of neurohaptic system 100B. Nerve cuff electrode 601 may wirelessly receive signals for neural stimulation. Although this embodiment is shown with both percutaneous electrodes 401 and nerve cuff electrodes 601, it may also be implemented using nerve cuff electrodes 601 only.

[0045] FIG. 8 illustrates an example of neurohaptic system 100 (or its components) embodied as a bracelet (or watch) 100B. Neurohaptic system 100B may include one or more externally positioned electrodes 401 for transcutaneous stimulation, as well as one or more nerve cuff electrodes 601. Nerve cuff electrode 601 is configured for localization on (e.g., partially or completely enveloping) and stimulation of one or more of ulnar nerve 302 or median nerve 301. Nerve cuff electrode 601 may include a single electrode or an array of electrodes. Subcutaneous electrode 601 may be physically connected to neurohaptic system 100B, for example, to percutaneous electrode 401 or to any other signal-providing components of neurohaptic system 100B via conduit 502. Nerve cuff electrode 601 may receive signals for neural stimulation in a wired manner. Although this embodiment is shown with both percutaneous electrodes 401 and nerve cuff electrodes 601, it may also be implemented using nerve cuff electrodes 601 only.

[0046] FIG. 9 illustrates the use of a neuro-haptic system according to embodiments herein. As shown in FIG. 9, a user utilizing neuro-haptic system 100C (e.g., embodied as a belt) and neuro-haptic system 100B (e.g., embodied as a bracelet) can receive notifications, alerts, instructions, and the like via neuro-haptic signaling. For example, one or more of neuro-haptic systems 100B and 100C can communicate with a user's smartphone or other device. When a notification (e.g., a call, text, email, etc.) arrives at the user device, the user device communicates with neuro-haptic system 100B / 100C to provide the user with a neuro-haptic stimulus associated with the notification. While neuro-haptic system 100B / 100C is illustrated, any suitable form factor of neuro-haptic system 100 may be used.

[0047] 10 illustrates the use of a neuro-haptic system according to embodiments herein. As shown in FIG. 10, a user utilizing neuro-haptic system 100B (e.g., embodied as a bracelet) can receive neuro-haptic stimulation related to or associated with an application or activity being interacted with on a tablet, computer, video game console, or other user interactive system 1000. As shown in FIG. 10, neuro-haptic system 100B can establish a connection with user interactive system 1000 and provide neuro-haptic stimulation associated with interaction with user interactive system 1000. While neuro-haptic system 100B is illustrated, any suitable form factor of neuro-haptic system 100 may be used.

[0048] 11 illustrates the use of a neuro-haptic system according to embodiments herein. As shown in FIG. 11, a user utilizing neuro-haptic system 100B (e.g., embodied as a bracelet) can receive neuro-haptic stimulation related to or associated with an application or activity being interacted with on virtual reality system 1100 (or a mixed reality or augmented reality system). As shown in FIG. 11, neuro-haptic system 100B can establish a connection with user interactive system 1100 and provide neuro-haptic stimulation associated with interaction with virtual reality system 1100. While neuro-haptic system 100B is illustrated, any suitable form factor of neuro-haptic system 100 may be used.

[0049] 12 is a flowchart illustrating steps of a method for providing neuro-haptic notification of an alert effect. Neuro-haptic effect process 1200 may be performed by any of neuro-haptic systems 100 described herein.

[0050] In operation 1202, neuro-haptic system 100 establishes a connection with an external interactive user device, such as a computer system or other device. Such a device or computer system may include, for example, a smartphone, a tablet, a personal computer, a vehicle computer, an AR / VR display or system, or any other interactive user device.

[0051] In act 1204, neuro-haptic system 100 receives a haptic notification from an interactive user device, e.g., a notification that a haptic effect should be output to alert the user of a notification or other alert. The haptic notification may be generated by the interactive user device in response to an event occurring on the interactive user device, such as, for example, an incoming text, phone call, or other notification, or any other event for which such interactive user device can provide a haptic output.

[0052] In operation 1206, neuro-haptic system 100 generates and provides a neuro-haptic signal to trigger a peripheral nervous system stimulation. The neuro-haptic signal can be generated by haptic effect rendering module 116, for example, based on a haptic effect created by haptic effect creator 114. Haptic effect creator 114 can create the haptic effect based on, for example, context information, signal information, and / or intent information. The haptic notification is included within the intent information. The neuro-haptic signal is provided to trigger a peripheral nervous system stimulation that alerts or notifies the user that an event is occurring on the interactive user device.

[0053] FIGS. 13A-13D illustrate the use of neuro-haptic system 100, which is used to detect neural signals and provide a neuro-haptic signal as an output. FIGS. 13A and 13B illustrate the use of neuro-haptic system 100B (e.g., having a bracelet form factor). While FIGS. 13A and 13B illustrate neuro-haptic system 100B consisting of two parts (e.g., a bracelet and a computing system), as discussed above, neuro-haptic system 100B may be provided with various components co-located within one or more physical devices or parts. The illustrations of FIGS. 13A and 13B are merely examples, and there is no requirement that system processing be performed remotely from a wearable device. FIG. 13C illustrates process steps for capturing and processing neural signals and generating stimulation signals. FIG. 13D illustrates process steps for capturing and processing neural signals and generating stimulation signals using machine learning and artificial intelligence techniques.

[0054] In an embodiment, signal processing method 1300 may be used. Neuro-haptic system 100 may detect neural signals in a calibration mode, for example, in operation 1301. As shown in FIG. 13A , sensor 107 in neuro-haptic system 100 may detect neural signals generated in response to a user action, such as touching a surface. The neural signals may be filtered, for example, to reduce or remove noise and / or artifacts, for example, in operation 1302. For example, detected neural signals in the form of neuronal spikes may be decoded (e.g., by signal decoder 112 in operation 1303) and processed by neuro-haptic system 100. The spikes may be processed and characterized by appropriate signal processing methods to determine and / or identify spike pattern characteristics of the user action. The spike pattern may represent one or more neuronal spikes and may be characterized by a list or map of spikes, inter-spike timing, spike amplitude, spike frequency, and / or any other suitable characteristic. The spike pattern may then be associated with the user action. Thus, neuro-haptic system 100 can associate the detected neural signal with a particular user action. Neuro-haptic system 100 can then, for example, create a neuro-haptic signal based on the detected neural signal (e.g., spike pattern) to recreate a sensation associated with the user action, at operation 1304. The generated neuro-haptic signal is created to recreate a sensation similar to the detected spike pattern. Thus, neuro-haptic system 100 can be calibrated by generating / creating one or more neuro-haptic signals associated with various user actions stored in a neuro-haptic library. The generated neuro-haptic signal can then be delivered to a user, for example, at operation 1305, via any suitable device, including those described herein.

[0055] At a later point in time, this neuro-haptic library can be accessed to provide an output of stored neuro-haptic signals to recreate the sensation associated with the original user action. In embodiments, neuro-haptic system 100 can execute a calibration program that requests a user to repeatedly perform a particular action and measures the neural signals generated in response, thereby building a library of neuro-haptic signals for later use. In further embodiments, the library of neuro-haptic signals can be a universal library built based on the actions and responses of several users. Such a universal library can then be used for any system user without additional calibration or with calibration used for fine-tuning.

[0056] In embodiments, a neuro-haptic signal, e.g., a stimulation signal, may be mapped or associated with a neural signal (e.g., a neuronal spike pattern) through a calibration operation similar to that of method 1300. In lieu of the user action discussed above, a neuro-haptic signal may be provided to a user, and the resulting neural signal may be captured and decoded. The captured and decoded signal (e.g., a spike pattern characteristic of the neural signal) may then be associated with the neuro-haptic signal used to create it. These associations may then be used when selecting a neuro-haptic signal to trigger a particular spike pattern associated with a user action. Thus, a first spike pattern may be captured, decoded, and associated with a user action. A neuro-haptic signal configured to trigger a neural signal having a second spike pattern similar to the first spike pattern may then be generated to simulate the user action. Spike patterns may be considered similar if they match within 70%, 80%, 90%, 95%, and / or 99%, for example, by comparing pulse width and pulse timing and frequency.

[0057] In embodiments, neural signals consistent with embodiments herein may be provided to the peripheral nervous system by devices described herein as pulses or a series of pulses (e.g., pulse trains). In examples, pulses consistent with embodiments herein may have pulse widths of 20-250 microseconds, or 24-60 microseconds. Pulse trains may be delivered at frequencies between 1-200 Hz and / or about 20 Hz. The pulse signals may vary in magnitude (e.g., as measured by current delivered to electrodes, e.g., at the skin) between 0-4 mA, 0-2 mA, or 0-1.5 mA. In embodiments, the neural signals may vary over time according to a ramp, e.g., gradually increasing from 0 mA to a set current over a specific period of time, e.g., 1 second, 2 seconds, 4 seconds, etc. In embodiments, pulse trains consistent with embodiments herein may be pulse-width modulated by a lower frequency sinusoidal signal. In one example, the amplitude of a 1 Hz sine wave may be used to modulate the pulse width of a higher frequency pulse train (e.g., up to 10 Hz, up to 50 Hz, up to 100 Hz, up to 200 Hz, etc.). In embodiments, pulse trains consistent with embodiments herein may be amplitude modulated by a lower frequency sinusoidal signal. For example, the amplitude of a 1 Hz sine wave may be used to modulate the pulse amplitude of a higher frequency pulse train (e.g., up to 10 Hz, up to 50 Hz, up to 100 Hz, up to 200 Hz, etc.).

[0058] FIG. 13D illustrates method 1350 of processing neural signals and generating neuro-haptic signals using artificial intelligence or machine learning techniques. Neuro-haptic system 100 can detect neural signals captured as multi-dimensional or multi-channel time-domain data in a calibration mode during a user action, for example, in act 1351. The neural signals can be preprocessed as needed for future actions. In embodiments, neural signals can be detected from tens, hundreds, thousands, and / or millions of users during similar user actions. User actions can include specific movements and / or longer duration actions, such as exploring an object's characteristics (texture, shape, etc.).

[0059] In operation 1352, the neuro-haptic system may operate to extract specific features from the captured neural signals. Feature extraction may include extraction of statistical features, time-based features, shape-based features, autocorrelation features, wavelet transform features, Fourier transform features, entropy measures, higher-order crossings, empirical mode decomposition, etc. Statistical features may include, for example, mean, standard deviation, variance, median, skewness, kurtosis, maximum and minimum values, range, etc. Time-based features may include, for example, zero-crossing rate (ZCR), root-mean-square (RMS), peak-to-peak distance, signal slope change, etc. Shape features may include, for example, crest factor, form factor, signal-to-noise ratio, etc. Entropy measures may include, for example, Shannon entropy and / or spectral entropy, etc.

[0060] In operation 1354, the neuro-haptic system 100 may operate to generate a tactile model according to the extracted features. The tactile model may be a feature of the physical world associated with a user action performed during data capture. In an embodiment, the tactile model may be generated according to one or more of the following methods: a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a transformer model, a feedforward neural network (FNN), a deep belief network (DBN), a capsule network, an attention mechanism, a variational autoencoder (VAE), etc.

[0061] In act 1355, neuro-haptic system 100 can operate to apply the tactile model to generate neuro-haptic signals that are transmitted to electrodes of a neuro-haptic device, such as a haptic output device, to generate sensations consistent with the user actions associated with the tactile model. Such signals can then be transmitted to the neuro-haptic device to generate sensations.

[0062] In another example of a calibration mode, neurohaptic system 100 may be configured to identify or locate a nerve and / or identify or select the best electrode for neurostimulation. For example, referring now to FIG. 4 , neurohaptic system 100B is shown including multiple transcutaneous electrodes 401. During calibration mode, transcutaneous electrodes 401 may be activated singly and / or in various combinations. As each electrode 401 or combination of electrodes 401 is activated, a user may input information into neurohaptic system 100 to identify which activations were effective and which were not. After performing such calibration, neurohaptic system 100 may identify the electrode 401 that provides the most effective neurostimulation according to the user. The identified electrode 401 may indicate or represent the location of ulnar nerve 302 or median nerve 301 within the user. In embodiments, such a calibration mode may further include the use of neurohaptic signals of different intensities, frequencies, patterns, etc. to further enhance the calibration. In further embodiments, the location of the nerves may be detected by ultrasound or other imaging techniques, in which case the imaging results may undergo image processing to detect, determine, and / or locate the location of the ulnar nerve 302 and median nerve 301.

[0063] In yet another example of a calibration mode, neuro-haptic system 100 can perform a calibration method in conjunction with a virtual environment and virtual reality system 1100 (such as in FIG. 11 ). For example, in a motor neuron signal calibration operation, neuro-haptic system 100 can receive and decode motor neural signals and associate such signals with movements occurring in the virtual environment. For example, a user can move their arm to touch an object. Virtual reality system 1100 can track the user's movements according to a camera or other sensor associated with virtual reality system 1100. Simultaneously, neuro-haptic system 100 can receive and decode motor neural signals associated with the same movements. The neuro-haptic system can then associate the received motor neural signals with the captured movements, thus providing the ability to track the user's movements based on the neural signals. Furthermore, when a user touches an object in the virtual reality environment, virtual reality system 1100 can provide a tactile haptic stimulus associated with such contact. Neuro-haptic system 100 may receive and decode sensory neural signals generated by tactile haptic stimulation and then associate the received / decoded sensory neural signals with haptic stimulation, thus providing the ability to provide neuro-haptic stimulation to replace or augment the tactile haptic stimulation provided by virtual reality system 1100.

[0064] 14 illustrates a flowchart showing steps in various methods for generating and providing neuro-haptic effects associated with an interactive user system. Neuro-haptic effect processes 1400, 1420, and 1430 may be performed by any of neuro-haptic systems 100 described herein.

[0065] Neuro-haptic effect process 1400 provides operational steps for generating a neuro-haptic signal associated with a user action.

[0066] At operation 1401, neuro-haptic effect process 1400 includes capturing neural information (e.g., neural signals) from a user's nerves, as described herein. The neural information can be captured while the user is performing a user action, for example, while touching an object.

[0067] In act 1402, the neuro-haptic effect process 1400 includes storing the neural information.

[0068] In act 1404, the neurohaptic effect process 1400 includes processing the neural information.

[0069] In act 1406, neurohaptic effect process 1400 includes extracting features, parameters, sub-signals, and any other relevant information from the neural information that may correspond to user actions.

[0070] In act 1408, neurohaptic effect process 1400 includes storing the processed and extracted information.

[0071] At operation 1410, neurohaptic effect process 1400 includes associating the processed information with user actions. The information may be stored in a user-specific or user-generic manner. For example, a database or library of processed information and extracted features may be generated for an individual user associating neural signals with various user actions. In further embodiments, personal user databases may be combined across users to generate larger, more powerful data sets. Such user-generic libraries may be utilized to create default knowledge bases that may be applicable to any user. Such user-generic libraries may be personalized for a particular user based on neurohaptic effect process 1400.

[0072] Neuro-haptic effect process 1420 provides operational steps for generating a neuro-haptic signal that is output in accordance with a user's interaction with an interactive user device.

[0073] In act 1421, neuro-haptic effect process 1420 includes connecting neuro-haptic system 100 with an interactive user device such as a smartphone, tablet, personal computer, game console, AR / VR device, etc. In embodiments, neuro-haptic system 100 may be incorporated into the interactive user device. For example, a wearable device (such as a smartwatch) may be an interactive user device and may incorporate any or all of the functionality of neuro-haptic system 100 described herein.

[0074] In act 1422, the neurohaptic effect process 1420 includes detecting user interactions, both input and output, with an application, game, or other software running on the interactive user device.

[0075] In operation 1424, neuro-haptic effect process 1420 includes generating and providing information related to the user interaction, e.g., in the form of a haptic event or a desired haptic effect requiring haptic output, to neuro-haptic system 100 by the interactive user device.

[0076] In act 1426, neuro-haptic effect process 1420 includes outputting a neuro-haptic signal to provide a neuro-haptic effect to a user of neuro-haptic system 100. The neuro-haptic effect can be associated with an interactive event occurring within a game, application, or other software running on an interactive user device. Neuro-haptic system 100 can generate / determine the neuro-haptic signal according to methods discussed herein.

[0077] Neuro-haptic effect process 1430 provides operational steps for generating neuro-haptic signals that are output according to hand tracking of a user's interaction with an interactive user device.

[0078] In act 1431, neuro-haptic effect process 1420 includes connecting neuro-haptic system 100 with an interactive user device such as a smartphone, tablet, personal computer, game console, AR / VR device, or the like.

[0079] In act 1432, the neurohaptic effect process 1420 includes detecting user interactions, both input and output, with an application, game, or other software running on the interactive user device.

[0080] In act 1434, the neurohaptic effect process 1420 includes tracking a user's hand gestures to identify user interaction with the interactive user device.

[0081] In operation 1436, neuro-haptic effect process 1420 includes generating and providing, by the interactive user device, information related to the user interaction, for example, in the form of a haptic event or a desired haptic effect requiring haptic output, to neuro-haptic system 100.

[0082] In act 1438, neuro-haptic effect process 1420 includes outputting a neuro-haptic signal to provide a neuro-haptic effect to a user of neuro-haptic system 100. The neuro-haptic effect can be associated with an interactive event occurring within a game, application, or other software running on an interactive user device. Neuro-haptic system 100 can generate / determine the neuro-haptic signal according to methods discussed herein.

[0083] In embodiments, in addition to the above-described methods of providing neurohaptic effects for notification and interaction purposes, neurohaptic system 100 described herein can further be utilized to assist users with disabilities. For example, a user with Parkinson's disease can benefit from receiving neurohaptic notifications to replace mechanical haptic notifications. In another example, a user who has lost sensation in a body part can benefit from neurohaptic signals being provided to replace the missing sensation. In yet another example, a user who has lost a body part can also benefit from neurohaptic signals being provided to replace the missing sensation. In such embodiments, sensors in an artificial body part can capture environmental information that can have neurohaptic signals received by and provided for neurohaptic system 100.

[0084] It will be readily apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments.

[0085] Although specific embodiments have been illustrated and described herein, it is to be understood that the claims are not limited to the specific forms or arrangements of parts described and shown. Although exemplary embodiments are disclosed herein, and specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation. Modifications and variations of the embodiments are possible in light of the above teachings. It is therefore to be understood that the embodiments may be practiced otherwise than as specifically described.

Claims

1. 1. A system for providing a neuro-haptic interface, comprising: a memory unit configured to store computer instructions; at least one processor, wherein the at least one processor: Receive notifications from peripheral devices, determining a neurohaptic effect to be performed in accordance with the notification; generating a neurohaptic signal corresponding to the neurohaptic effect; The system is configured to execute the instructions to provide the neuro-haptic signal to a neuro-haptic device to render the neuro-haptic effect by stimulating a user's peripheral nervous system.

2. The system of claim 1 , wherein the notification is an event notification including at least one of an alarm, an alert, or a status update.

3. 2. The system of claim 1, wherein the notification is an environmental notification, and the at least one processor is further configured to determine the neuro-haptic effect to provide the user with a sensation corresponding to the environmental notification.

4. The system of claim 1 , wherein at least one of the neuro-haptic effect and the neuro-haptic signal is selected from a lookup table or library.

5. The system of claim 1 , wherein the notification is associated with a gaming system and the neuro-haptic effect is determined to provide feedback associated with an application of the gaming system.

6. The system of claim 1 , wherein the notification is associated with a vehicle and the neuro-haptic effect is determined to provide feedback associated with operation of the vehicle.

7. The system of claim 4 , wherein the neuro-haptic effect is rendered to provide a user with a sensation that replaces a non-felt sensation associated with an environmental notification.

8. The system of claim 1 , wherein the neuro-haptic device is a wearable device.

9. The system of claim 1 , wherein the neuro-haptic device is a chair.

10. 1. A method for generating a neuro-haptic signal, comprising: detecting a neural signal by at least one neural sensor of the neuro-haptic system during performance of a user action; processing, by at least one processor of the neuro-haptic system, the neural signal to identify a characteristic of a first spike pattern of the neural signal; associating, by the at least one processor of the neuro-haptic system, the first spike pattern with the user action; generating, by the at least one processor, a neuro-haptic signal configured to provide stimulation to a user to produce a neural response having a second spike pattern similar to the first spike pattern; and delivering the neuro-haptic signal via at least one electrode.

11. The method of claim 10 , further comprising storing, by the at least one processor, the neuro-haptic signal associated with the user action in a neuro-haptic library.

Images