Stimulation control

Abstract

Presented herein are techniques for controlling stimulation provided by an implantable device. Stimulation control may be performed at an external device, and machine learning (e.g., artificial intelligence (AI)) may be used. The techniques provide a cochlear implant stimulation strategy that utilizes computational models of a healthy hearing system and an implanted hearing system to more closely simulate natural acoustic hearing.

Description

BACKGROUND TECHNICAL FIELD

[0001] Aspects of the present disclosure generally relate to controlling stimulation delivered by an electronic device. BACKGROUND

[0002] Medical devices have provided a wide range of therapeutic benefits to recipients in recent decades. Medical devices can include internal or implantable components / devices, external or wearable components / devices, or combinations thereof (e.g., devices with external components that cooperate with implantable components). Medical devices, such as traditional hearing aids, partially or completely implantable hearing prostheses (e.g., bone conduction devices, mechanical stimulators, cochlear implants etc. ), pacemakers, defibrillators, functional electrical stimulation devices, completely implantable visual prostheses, vagus nerve stimulators, spinal cord stimulators, and other medical devices have been successful in performing life-saving and / or lifestyle improvement functions and / or recipient monitoring for many years.

[0003] The types of medical devices, and the range of functions performed by them, have increased over the years. For example, many medical devices, sometimes referred to as “implantable medical devices,” now often include one or more instruments, apparatus, sensors, processors, controllers or other functional mechanical or electrical components implanted in a recipient, either permanently or temporarily. These functional devices are typically used to diagnose, prevent, monitor, treat, or manage a disease / injury or symptom thereof, or study, replace or modify an anatomical structure or a physiological process. Many of these functional devices utilize power and / or data received from external devices that are part of, or operate in conjunction with, the implantable components. SUMMARY

[0004] In one aspect, a method is provided. The method includes receiving, at an implantable medical device system, a signal associated with a physiological function; determining, by a machine learning model, information for a stimulation signal to stimulate the physiological function based on the signal, wherein the machine learning model is trained based on modeling physiological effects from the stimulation; and controlling the stimulation of a recipient of the implantable medical device system based on the determined information.

[0005] In another aspect, one or more non-transitory computer-readable storage media including instructions is provided. The instructions, when executed by one or more processors, cause the one or more processors to receive, at an implantable medical device system, a signal associated with a physiological function; determine, by a machine learning model, information for a stimulation signal to stimulate the physiological function based on the signal, wherein the machine learning model is trained based on modeling physiological effects from the stimulation; and control the stimulation of a recipient of the implantable medical device system based on the determined information.

[0006] In another aspect, another method is provided. The method includes determining, by a machine learning model of at least one processor, information for a stimulation signal to stimulate a physiological function based on a signal associated with the physiological function; modeling, via the at least one processor, a physiological effect from the stimulation signal; and updating, via the at least one processor, the machine learning model based on a difference between the modeled physiological effect and a reference physiological effect representative of normal physiological function.

[0007] In another aspect, another method is provided. The method includes determining, by a machine learning model of at least one processor, information for a stimulation signal to stimulate a physiological function based on a signal associated with the physiological function; modeling, via the at least one processor, a physiological effect from the stimulation signal; and updating, via the at least one processor, the machine learning model based on a difference between the modeled physiological effect and a reference physiological effect representative of normal physiological function. BRIEF DESCRIPTION OF DRAWINGS

[0008] Embodiments of the application are described herein with reference to the accompanying drawings, of which: Figure 1A is a schematic diagram illustrating a cochlear implant system with which aspects of the technology presented herein can be implemented; Figure 1B is a side view of a recipient of a sound processing unit of the cochlear implant system of Figure 1A ; Figure 1C is a schematic diagram of components of the cochlear implant system of Figure 1A ; Figure 1D is a block diagram of the cochlear implant system of Figure 1A ; Figure 1E is a schematic diagram illustrating a computing device with which aspects of the technology presented herein can be implemented; Figure 2 is a functional block diagram illustrating an exemplary audio signal processing path of a cochlear implant system with which aspects of the technology presented herein can be implemented; Figure 3 is a functional block diagram illustrating a method of training a hearing machine learning (ML) model in accordance with certain embodiments; Figure 4A illustrates an exemplary spectrogram for a rising complex tone; Figure 4B illustrates an exemplary neurogram of inner hair cell voltage for a rising complex tone of Figure 4A ; Figure 4C an exemplary neurogram of the auditory nerve fine structure for a rising complex tone; Figure 4A Figure 5 is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model to control stimulation according to certain embodiments; Figure 6 is a schematic diagram of an exemplary neural network with which aspects of the technology presented herein can be implemented; Figure 7 is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model using feature extraction applied to an input audio signal according to certain embodiments; Figure 8 is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model with another feature extraction applied to an input audio signal according to certain embodiments; Figure 9 is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model with a neurogram of an input audio signal according to certain embodiments; Figure 10 is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model to reduce noise according to certain embodiments; Figure 11A is a functional block diagram illustrating a method of training a sound processor machine learning (ML) model with a neurogram of an input audio signal for focused multipolar stimulation according to certain embodiments; Figure 11B is a functional block diagram illustrating another method of training a sound processor machine learning (ML) model with a neurogram of an input audio signal for focused multipolar stimulation according to certain embodiments; Figure 12 is a flowchart illustrating an exemplary process of controlling stimulation according to certain embodiments; and Figure 13 is a flowchart illustrating an exemplary process for training a machine learning model to control stimulation according to certain embodiments. DETAILED DESCRIPTION

[0009] Presented herein are technologies for controlling stimulation provided by an electronic device such as an implantable medical device. The stimulation control can be performed at an external device and can use machine learning (e.g., artificial intelligence (AI)). In certain aspects, the technologies provide a stimulation strategy that utilizes a healthy auditory system and a computational model of an implanted auditory system to more closely emulate natural acoustic hearing.

[0010] ​The exemplary techniques presented in this paper minimize the differences between normal hearing and electro-hearing computational models to deliver sound information more consistent with normal hearing and improve outcomes for hearing device users.

[0011] It should be understood that there are several different types of medical devices in which / utilizing them can implement the techniques presented herein. For ease of description only, the techniques presented herein are described primarily with reference to a specific medical device in the form of a cochlear implant. However, it should be understood that the techniques presented herein can also be implemented, in part or in whole, by any of several different types of medical devices, particularly any other type of device for delivering electrical stimulation signals to a recipient. For example, the techniques can be implemented in devices that deliver electrical stimulation to the auditory nerve, middle ear, vestibular system, retina and / or brain, as well as other areas of the body. As used herein, the term “hearing device” should be interpreted broadly as any device that delivers sound signals to a user in any form, including acoustic stimulation, mechanical stimulation, electrical stimulation, optical stimulation, etc. In this way, hearing devices can be devices for use by people with hearing loss (e.g., hearing aids, middle ear prostheses, bone conduction devices, direct acoustic stimulators, electroacoustic hearing prostheses, auditory brainstem stimulators, bimodal hearing prostheses, bilateral hearing prostheses, dedicated tinnitus treatment devices, tinnitus treatment device systems, combinations or variations thereof) or devices for use by people with normal hearing (e.g., consumer devices that provide audio streaming, consumer headphones, headphones and other listening devices).

[0012] Figures 1A-1E An exemplary cochlear implant system 102 is shown that can be used to implement aspects of the techniques presented herein. The cochlear implant system 102 includes an external component 104 and an implantable component 112. Figures 1A-1E In the examples, the implantable component is sometimes referred to as a "cochlear implant". Figure 1A The image shows a cochlear implant 112 implanted in the user's head 154, while... Figure 1B This is a schematic diagram of an external component 104 worn on the user's head 154. Figure 1C This is another schematic diagram of the cochlear implant system 102, and Figure 1D Further details of the cochlear implant system 102 are shown. For ease of description, they will generally be described together. Figures 1A-1E .

[0013] The cochlear implant system 102 includes an external component 104 configured to be directly or indirectly attached to a user's body, and an implantable component (or implant) 112 configured to be implanted into the user. Figures 1A-1E In the example, external component 104 includes a sound processing unit 106, while cochlear implant 112 includes an implantable coil 114, an implant body 134, and an elongated stimulation component 116 configured for implantation in a user's cochlea.

[0014] In Figures 1A-1E the example, the sound processing unit 106 is an off-the-ear (OTE) sound processing unit, sometimes referred to herein as an OTE piece, configured to send data and power to the implantable piece 112. Generally, the OTE sound processing unit is a piece having a generally cylindrical housing 111 and configured to magnetically couple to the user’s head (e.g., including an integrated external magnet 150 configured to magnetically couple to an implantable magnet 152 in the implantable piece 112). The OTE sound processing unit 106 also includes an integrated (headpiece) coil 108 configured to inductively couple to the implantable coil 114.

[0015] It should be appreciated that the OTE sound processing unit 106 is merely illustrative of an external device that can operate with the implantable piece 112. For example, in alternative examples, the external piece can include a behind-the-ear (BTE) sound processing unit or a micro-BTE sound processing unit and a separate external coil assembly. Generally, a BTE sound processing unit includes a housing shaped to be worn over the user’s outer ear and connected to a separate external coil assembly via a cable, where the external coil assembly is configured to magnetically and inductively couple to the implantable coil 114. It should also be appreciated that alternative external pieces can be located in the user’s ear canal, worn on the body etc. .

[0016] As described above, the cochlear implant system 102 includes a sound processing unit 106 and a cochlear implant 112. However, as described further below, the cochlear implant 112 can operate independently of the sound processing unit 106 for at least a period of time to stimulate the user. For example, the cochlear implant 112 can operate in a first general mode (sometimes referred to as an “external hearing mode”), in which the sound processing unit 106 captures a sound signal, which is then used as a basis for delivering a stimulation signal to the user. The cochlear implant 112 can also operate in a second general mode (sometimes referred to as a “stealth hearing” mode), in which the sound processing unit 106 is unable to provide a sound signal to the cochlear implant 112 (e.g., the sound processing unit 106 is not present, the sound processing unit 106 is powered off, the sound processing unit 106 is malfunctioning etc.). Thus, in the cochlear implant 112 in the invisible hearing mode, the cochlear implant 112 captures the sound signals themselves via the implantable sound sensor and then uses these sound signals as the basis for delivering stimulation signals to the user. In certain examples, in the invisible hearing mode, the external device is still able to deliver power to the implant. In such examples, the external device can implement the techniques presented herein to calculate the optimal power level using information (e.g., stimulation parameters) from the cochlear implant 112 that is retrieved or stored on the external device. Further details regarding the operation of the cochlear implant 112 in the external hearing mode are provided below, followed by details regarding the operation of the cochlear implant 112 in the invisible hearing mode. It should be appreciated that the references to the external hearing mode and the invisible hearing mode are merely illustrative, and the cochlear implant 112 can also operate in an alternating mode.

[0017] In Figure 1A and 1C , the cochlear implant system 102 is shown with an external computing device 110 configured to implement aspects of the presented techniques. The computing device 110 (shown in more detail in Figure 1E ) is, for example, a personal computer, a server computer, a handheld device, a laptop device, a multiprocessor system, a microprocessor-based system, a programmable consumer electronics (e.g., a smart phone), a network PC, a minicomputer, a mainframe computer, a tablet, a remote control unit, a distributed computing environment that includes any of the above systems or devices, or the like. The computing device 110 can be a single virtual or physical device operating in a networked environment through a communication link to one or more remote devices, such as an implantable medical device or an implantable medical device system.

[0018] In its most basic configuration, the computing device 110 includes at least one processing unit 183 and memory 184. The processing unit 183 includes one or more hardware or software processors (e.g., central processing units) that can obtain and execute instructions. The processing unit 183 can be in communication with and control the performance of other components of the computing device 110.

[0019] The memory 184 is one or more software or hardware-based computer readable storage media accessible by the processing unit 183 to store information accessible to (or usable by) the processing unit 183. The memory 184 can include, for example, RAM, ROM, EEPROM, flash memory, optical media, magnetic media, solid-state memory, or other storage technology, including an artificial storage medium or other artificial computer-readable medium. The memory 184 can include one or more removable or non-removable memory devices including, for example, a volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. In an example, the memory 184 includes a modulated data signal (for example, a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal) that includes one or more characteristics of a carrier wave or other transport mechanism and includes a propagated signal. The memory 184 can include a wired medium (for example, a wired network or direct-wired connection) and / or a wireless medium (for example, acoustic, RF, infrared, and other wireless media) or a combination of both wired and wireless media. In some embodiments, the memory 184 includes stimulation control logic 185 (with a stimulation generator model 192) that, when executed, enables the processing unit 183 to perform aspects of the presented technology.

[0020] In the illustrated example, the computing device 110 also includes a network adapter 186, one or more input devices 187, and one or more output devices 188. The computing device 110 can include other components, such as a system bus, component interfaces, graphics systems, power supplies (for example, batteries), and other components.

[0021] The network adapter 186 is a component of the computing device 110 that provides network access (for example, access to at least one network 189). The network adapter 186 can provide wired or wireless network access and can support one or more of a variety of communication technologies and protocols, such as Ethernet, cellular, Bluetooth, near-field communication, and RF (radio frequency), among others. The network adapter 186 can include one or more antennas and associated components configured to communicate wirelessly according to one or more wireless communication technologies and protocols. In some examples, the one or more antennas can be shared with the charging coil 121 and / or the external coil 108.

[0022] The one or more input devices 187 are devices through which the computing device 110 receives input from a user. The one or more input devices 187 can include physically actuatable user interface elements (e.g., buttons, switches, or dials), a touch screen, a keyboard, a mouse, a pen, and voice input devices, among other input devices.

[0023] The one or more output devices 188 are devices through which the computing device 110 can provide output to a user. The output devices 188 can include, for example, a display 190 and one or more speakers 191, among other output devices.

[0024] It should be appreciated that Figure 1E The arrangement of the computing device or system 110 shown in FIG. 1 is merely illustrative, and aspects of the technology presented herein can be implemented at many different types of systems / devices. For example, the computing device 110 can be a laptop computer, a tablet computer, a mobile phone, a surgical system etc. .

[0025] The OTE sound processing unit 106 includes one or more input devices configured to receive input signals (e.g., sound or data signals). The one or more input devices include one or more sound input devices 118 (e.g., one or more external microphones, audio input ports, coil pickups etc. ), one or more auxiliary input devices 128 (e.g., audio ports such as a direct audio input (DAI), data ports such as a universal serial bus (USB) port, cable ports etc. ), and a wireless transmitter / receiver (transceiver) 120 (e.g., for communicating with an external computing device 110). However, it should be appreciated that the one or more input devices can include additional types of input devices and / or fewer input devices (e.g., the wireless short-range radio transceiver 120 and / or the one or more auxiliary input devices 128 can be omitted).

[0026] The OTE sound processing unit 106 also includes an external coil 108, a charging coil 121, a closely coupled transmitter / receiver (RF transceiver) 122 (sometimes referred to as a radio frequency (RF) transceiver 122), at least one rechargeable battery 132, and an external sound processing module 124. The external sound processing module 124 can include, for example, one or more processors and a memory device (memory) including sound processing logic. The memory device can also include stimulation control logic 185 that, when executed, enables the one or more processors to perform aspects of the presented technology. The memory device can include any one or more of: non-volatile memory (NVM), ferroelectric random access memory (FRAM), read-only memory (ROM), random-access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. The one or more processors are, for example, microprocessors or microcontrollers that execute instructions stored in the memory device for the sound processing logic and the stimulation control logic 185 (with the stimulation generator model 192).

[0027] The implantable component 112 includes an implant body (main module) 134 all configured to be implanted under the skin / tissue (tissue) 115 of the user, a lead region 136, and a cochlear-intra-stimulus assembly 116. The implant body 134 generally includes a hermetically sealed housing 138 in which can be included at least one battery 125, RF interface circuitry 140, and a stimulator unit 142. The implant body 134 also includes an internal / implantable coil 114 that is generally external to the housing 138, but connected to the RF interface circuitry 140 via a hermetic feedthrough (not shown). Figure 1D

[0028] As mentioned, the stimulation assembly 116 is configured to be implanted at least partially in the cochlea of the user. The stimulation assembly 116 includes a plurality of longitudinally spaced-apart intra-cochlear electrical stimulation contacts (electrodes) 144 that collectively form a contact or electrode array 146 for delivering electrical stimulation (current) to the cochlea of the user.

[0029] The stimulation assembly 116 extends through an opening (e.g., a cochleostomy, a round window etc. aperture, etc.) in the cochlea of the user and has a proximal end connected to the stimulator unit 142 via the lead region 136 and a hermetic feedthrough (not shown). Figure 1D The lead region 136 includes a plurality of conductors (wires) that electrically couple the electrodes 144 to the stimulator unit 142. The implantable component 112 also includes an extra-cochlear electrode, sometimes referred to as an extra-cochlear electrode (ECE) 139.

[0030] ​As noted, the cochlear implant system 102 includes an external coil 108 and an implantable coil 114. An external magnet 150 is fixed relative to the external coil 108, and an implantable magnet 152 is fixed relative to the implantable coil 114. The magnets fixed relative to the external coil 108 and the implantable coil 114 facilitate operational alignment of the external coil 108 with the implantable coil 114. This operational alignment of the coils enables the external component 104 to transmit power and optional data to the implantable component 112 via a tightly coupled wireless link 148 formed between the external coil 108 and the implantable coil 114. In certain examples, the tightly coupled wireless link 148 is a radio frequency (RF) link. However, various other types of energy transfer (e.g., infrared (IR), electromagnetic, capacitive, and inductive transfer) can be used to transfer power and / or data from the external component to the implantable component, and thus, Figure 1D Only one exemplary arrangement is shown.

[0031] As noted above, the sound processing unit 106 includes an external sound processing module 124. The external sound processing module 124 is configured to convert received input signals (received at one or more of the input devices) into output signals for stimulating a first ear of a user (i.e., the external sound processing module 124 is configured to perform sound processing on input signals received at the sound processing unit 106). In other words, one or more processors in the external sound processing module 124 are configured to execute sound processing logic in memory to convert received input signals into output signals representing electrical stimulation for delivery to a user. The external sound processing module 124 can further control stimulation provided by the implant 112 using machine learning (e.g., artificial intelligence (AI)) in accordance with the techniques presented herein.

[0032] As noted, Figure 1D Embodiments are shown in which the external sound processing module 124 in the sound processing unit 106 generates the output signals. In alternative embodiments, the sound processing unit 106 can transmit less processed information (e.g., audio data) to the implantable component 112, and sound processing operations (e.g., conversion of sound to output signals) can be performed by processors within the implantable component 112.

[0033] Returning to Figure 1DIn a specific example, the output signal is provided to an RF transceiver 122, which transmits the output signal (e.g., in an encoded manner) percutaneously to the implantable component 112 via an external coil 108 and an implantable coil 114. That is, the output signal is received at the RF interface circuitry 140 via the implantable coil 114 and provided to the stimulator unit 142. The stimulator unit 142 is configured to use the output signal to generate an electrical stimulation signal (e.g., a current signal) for delivery to the user's cochlea. In this way, the cochlear implant system 102 electrically stimulates the user's auditory nerve cells, thereby bypassing the missing or defective hair cells that normally translate acoustic vibrations into neural activity, so that the user perceives one or more components of the received sound signal.

[0034] As detailed above, in external hearing mode, the cochlear implant 112 receives processed sound signals from the sound processing unit 106. However, in invisible hearing mode, the cochlear implant 112 is configured to capture and process sound signals for electrical stimulation of the user's auditory nerve cells. Specifically, as Figure 1D As shown, the cochlear implant 112 includes a plurality of implantable sound sensors 160 and an implantable sound processing module 158. Similar to the external sound processing module 124, the implantable sound processing module 158 may include, for example, one or more processors and a memory device (memory) including sound processing logic. The memory device may include any or more of the following: non-volatile memory (NVM), ferroelectric random access memory (FRAM), read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory, electrical, optical, or other physical / tangible memory storage devices. The one or more processors are, for example, microprocessors or microcontrollers that execute instructions for the sound processing logic stored in the memory device.

[0035] In the invisible hearing mode, the implantable sound sensor 160 is configured to detect / capture signals (e.g., acoustic sound signals, vibrations). etc.), which is provided to the implantable sound processing module 158. The implantable sound processing module 158 is configured to convert the received input signals (received at one or more of the implantable sound sensors 160) into output signals for stimulating the first ear of the user (i.e., the processing module 158 is configured to perform sound processing operations). In other words, one or more processors in the implantable sound processing module 158 are configured to execute sound processing logic in memory to convert the received input signals into output signals 156 that are provided to the stimulator unit 142. The stimulator unit 142 is configured to utilize the output signals 156 to generate electrical stimulation signals (e.g., electrical current signals) for delivery to the cochlea of the user, thereby bypassing missing or defective hair cells that would normally convert acoustic vibrations into neural activity.

[0036] It should be appreciated that the above description of the so-called external hearing mode and the so-called invisible hearing mode is merely illustrative, and that the cochlear implant system 102 can operate differently in different embodiments. For example, in one alternative implementation of the external hearing mode, the cochlear implant 112 can generate stimulation signals for delivery to the user using signals captured by the sound input devices 118 and the implantable sound sensors 160.

[0037] In at least one embodiment, during operation of a hearing device system that includes a cochlear implant, the following sound processing operations are performed, as discussed in further detail below with reference to Figure 2 The sound processing module 124 is configured to convert output signals received from input devices (e.g., one or more sound input devices 118 and / or one or more auxiliary input devices 128) into a set of output signals representing electrical stimulation, as discussed in further detail below with reference to

[0038] Reference is made to Figure 2 , which shows a functional block diagram illustrating an example sound / audio signal processing path of an auditory prosthesis (e.g., the cochlear implant system 102) with which aspects of the technology presented herein can be implemented. For purposes of discussion, the various sound processing operations discussed below can be performed via sound processing logic provided for any combination of external or internal components of a cochlear implant system. Reference is made to Figure 2 the various features illustrated in Figure 2 are discussed as Figures 1A-1D indicated above with respect to various features of the cochlear implant system 102.

[0039] Reference is made to Figure 2 , which considers a sensory / environmental signal or audio signal processing path 251 that can be provided via the sound processing module 124 of the external components 104 and / or via the sound processing module 158 of the implantable components 112. In Figure 2In the example of FIG. 2, the input devices can include two sound input devices, namely first microphone 218A and second microphone 218B, and at least one auxiliary input device 228 (e.g., an audio input port, a cable port, a coil pickup etc. ). If not in electrical form, the input devices can convert the received / input sound signals into electrical signals 253 (referred to herein as electrical sound or sensation signals) that represent the sound / sensation signals received at the input devices. The electrical sound / sensation signals 253 can include electrical sensation signals 253A from microphone 218A, electrical sensation signals 253B from microphone 218B, and electrical sensation signals 253C from auxiliary input 228.

[0040] In Figure 2 , the functional operations enabled by the audio signal processing path (i.e., the operations of the one or more processors in executing the sound processing logic) are generally represented by modules 254, 256, 258, 260, and 262, which collectively make up the audio signal processing path 251. Thus, the audio signal processing path 251 can include a pre-filter bank processing module 254, a filter bank module 256, a post-filter bank processing module 258, a channel selection module 260, and a mapping module 262, each of which is described in greater detail below. The stimulus generator model 192 can be used to generate stimulus signals (e.g., stimulus pulses, analog stimuli, etc.) at any portion of the signal processing path or in place of any portion of the signal processing path. Further, the stimulus generator model can receive audio signals processed prior to the signal processing path or at any point during the signal processing path. The processed signals can provide various features as described below. For example, the stimulus generator model can receive audio signals processed after the filter bank module 256 as described below.

[0041] Consider an example of operation in which the electrical sound signals 253 generated by the input devices are provided to the pre-filter bank processing module 254. The pre-filter bank processing module 254 is configured to combine the electrical sound signals 253 received from the input devices as needed and to prepare / enhance these signals for subsequent processing. The operations performed by the pre-filter bank processing module 254 can include, for example, microphone directional operations, noise reduction operations, input mixing / combining operations, input selection / reduction operations, dynamic range control operations, and / or other types of signal enhancement operations. The operations at the pre-filter bank processing module 254 generate a pre-filter bank output signal 255, which is the basis for further sound processing operations as further described below. The pre-filter bank output signal 255 represents a combination (e.g., mix, selection etc. ) of the input signals received (e.g., mixed, selected etc. ) at the sound input devices at a given point in time.

[0042] In operation, the pre-filter bank processing module 254 generates a pre-filter bank output signal 255, which is provided to a filter bank module 256. The filter bank module 256 generates a suitable set of bandwidth limited channels or frequency partitions, each comprising a spectral component of the received sound / feeling signal. That is, the filter bank module 256 comprises a plurality of bandpass filters that separate the pre-filter bank output signal 255 into a plurality of components / channels, each carrying a frequency sub-band of the original signal (i.e., a frequency component of the received sound / feeling signal).

[0043] The channels created by the filter bank module 256 are sometimes referred to herein as sound processing or bandpass filtered channels, and the sound signal components within each sound processing channel are sometimes referred to herein as bandpass filtered signals or channelized signals. The bandpass filtered or channelized signals created by the filter bank module 256 are processed (e.g., modified / conditioned) as they pass through the audio signal processing path 251. Thus, the bandpass filtered or channelized signals are variously referred to at different stages of the audio signal processing path 251. However, it will be appreciated that references herein to bandpass filtered signals or channelized signals can refer to spectral components of the sound signal received at any point within the audio signal processing path 251 (e.g., pre-processing, processing, selection etc. ).

[0044] At the output of the filter bank module 256, the channelized signals are initially referred to herein as pre-processed signals or filter bank channels 257. The number "n" of filter bank channels 257 generated by the filter bank module 256 can depend on a number of different factors, including but not limited to implant design, number of active electrodes, coding strategy, and / or recipient preference(s). In certain arrangements, twenty-two (22) channelized signals are created, and the audio signal processing path 251 is considered to comprise 22 channels.

[0045] The filter bank channels 257 are provided to a post-filter bank processing module 258. The post-filter bank processing module 258 is configured to perform a number of sound processing operations on the target filter bank channels 257. These sound processing operations include, for example, channelized gain adjustments (e.g., performed via a loudness growth function (LGF) process) in one or more channels for hearing loss compensation (e.g., gain adjustments to one or more discrete frequency ranges of the sound signal, also referred to herein as filter channels), noise reduction operations, speech enhancement operations etc. After performing the sound processing operations, the post-filter bank processing module 258 outputs a plurality of processed channelized signals 259.

[0046] In Figure 2In a specific arrangement of the audio signal processing path 251, the audio signal processing path 251 includes a channel selection module 260. The channel selection module 260 is configured to perform a channel selection process to select which of the “n” channels should be used in the hearing compensation according to one or more selection rules. The signal selected at the channel selection module 260 is indicated by arrow 261 and is referred to herein as the selected channelized signal, or more simply as the selected signal. Figure 2

[0047] In an embodiment of the Figure 2 channel selection module 260 selects a subset “m” of the “n” processed channelized signals 259 used to generate the electrical stimulation delivered to the recipient (i.e., the sound processing channel is reduced from “n” channels to “m” channels). In one specific example, the “m” maximum amplitude channels (maxima) are produced from the “n” available combined channel signals, where “n” and “m” are programmable during initial fitting and / or operation of the prosthesis. In one instance, this specific example can be associated with an advanced combination encoder (ACE), which is generally a type of stimulation encoding strategy, such as optimal pitch and language (OPAL). It should be appreciated that different channel selection methods can be used and are not limited to maxima selection. It should also be appreciated that in certain embodiments, the channel selection module 260 can be omitted. For example, certain arrangements can use continuous interleaved sampling (CIS), a CIS-based, or other non-channel selection sound encoding strategy.

[0048] Figure 2 The audio signal processing path 251 of the example illustrated in FIG. 1 can also include a mapping module 262, which can generate an output signal 263. In one embodiment, the mapping module 262 can be configured to map the selected signal 261 (or the processed channelized signal 259 in embodiments that do not include channel selection) (e.g., via the stimulus generator model 192) such that the output signal 263 corresponds to a set of stimulation control signals (e.g., stimulation commands) representative of properties of an electrical stimulation signal that is to be delivered to the recipient in order to evoke a perception of at least a portion of the received sound signal. For example, this channel mapping can include threshold and comfort level mapping, dynamic range adjustment (e.g., compression), volume adjustment etc. and can encompass selection of various sequential and / or simultaneous stimulation strategies.

[0049] ​In one embodiment, a set of stimulation control signals (stimulation commands) 263 representing electrical stimulation signals can be encoded for transcutaneous transmission (e.g., via an RF link) to the implantable component. Thus, the mapping module 262 can also be referred to as a channel mapping and encoding module, and as an output block configured to convert a plurality of channelized signals into a plurality of stimulation control signals, the implantable component can generate stimulation (current) signals for delivery to the recipient via the stimulator assembly 116 in accordance with the plurality of stimulation control signals via the stimulator unit 142.

[0050] In one embodiment, for example, if the channel selection module 260 is omitted from the audio signal processing path 251, the mapping module 262 can perform a mapping operation that involves mapping a channel envelope to a current level that can be mixed with a stream received from one or more sources. Generally, a channel envelope is a kind of "time envelope" that is extracted from each frequency band (channel) and used to modulate a pulse train delivered to an implanted electrode. Thus, the amplitude of a current pulse can be extracted from a channel envelope, where the channel envelope corresponds to the amplitude of the signal in a given frequency channel.

[0051] Thus, the audio signal processing path 251 generally operates to convert a received sound signal into an output signal 263 that can be used to deliver stimulation to a recipient in a manner that induces a perception of the sound signal.

[0052] As noted above, cochlear implants electrically stimulate the auditory nerve, bypassing damaged sensory receptors and eliciting a pattern of neural activation that represents an acoustic sound. Although cochlear implants restore hearing to people who are severely to completely deaf, many cochlear implant recipients still struggle with complex listening situations, such as speech-in-noise perception and music perception.

[0053] The reason for these difficulties lies in the limited sound information that cochlear implants transmit. Typically, a cochlear implant extracts an envelope in a frequency band corresponding to each implanted electrode, and those envelopes are used to modulate a fixed-rate biphasic pulse train that is transmitted to the electrode. Using only a time envelope in a limited number of frequency bands reduces the temporal and spectral resolution of the acoustic sound. Furthermore, cochlear implants can use techniques such as frequency decomposition in order to provide computational efficiency, which only produces a coarse approximation of the sound. Moreover, the current delivered to the electrode diffuses through the conductive fluid of the cochlea, limiting channel independence. Thus, the cochlear implant-elicited pattern of neural activation is only a coarse approximation of the acoustic hearing-elicited pattern of neural activation.

[0054] According to example embodiments, stimulation control is provided by an implantable device. Stimulation control can be conducted at an external device and can use machine learning (e.g., artificial intelligence (AI)). Example embodiments provide a cochlear implant stimulation strategy that utilizes a computational model of the healthy auditory system and an implanted auditory system to more closely emulate natural acoustic hearing.

[0055] Advances in electrode design and stimulation techniques (e.g., perimodiolar electrode arrays and focused multipolar stimulation) enable finer temporal and spectral resolution in the neural activation pattern generated by cochlear implant stimulation. In addition, a sophisticated model of the auditory system can predict the spike responses of auditory nerve fibers by modeling the middle ear, traveling waves along the basilar membrane, inner hair cell transduction, auditory nerve synapses, and auditory nerve spiking behavior. Furthermore, a computational model of electric hearing can predict the response of the auditory nerve to electrical stimulation by modeling electrode properties, electrode placement, current spread within the cochlea, neural activation, and temporal properties of auditory neurons (e.g., refractoriness, adaptation, facilitation, and accommodation). The computational model of electric hearing can also be personalized for an individual cochlear implant recipient by considering the unique pattern of neural health along the cochlea, ossification or fibrosis within the cochlea, and / or patient-specific etiologic factors.

[0056] Example embodiments minimize the differences between normal hearing and electric hearing computational models to deliver sound information that is more consistent with normal hearing and improve the outcomes of cochlear implant recipients. This type of strategy was previously infeasible because of the massive computational demands of the auditory model, which made real-time application impossible and increased power consumption. The inclusion of the auditory model adds components to the stimulation pattern that are not present in many of the spectral map-based cochlear implant stimulation strategies, including onset enhancement, fundamental frequency modulation, and traveling wave dynamics. Onset enhancement and fundamental frequency modulation have been shown to improve speech perception in cochlear implant recipients when applied independently to pulse trains. Example embodiments encode onset enhancement and fundamental frequency modulation, their interaction consistent with the human auditory system.

[0057] In addition, example embodiments pre-compensate for the temporal properties of neurons (e.g., refractoriness and adaptation) so that the parts of the sound stimulation that are not encoded by neurons in the normal auditory system are not encoded by the cochlear implant processor. This saves power and reduces unnecessary channel interactions by removing redundant pulses.

[0058] In example embodiments, a deep neural network (DNN) or other machine learning model can be trained to generate a stimulation pattern that minimizes the difference between an acoustic auditory neurogram and an electrically evoked neural firing pattern. In embodiments, the computational model can include a model for the auditory periphery (e.g., spiral ganglion neuron activity in the auditory nerve). However, in other embodiments, the computational model can model more central projections (e.g., the auditory brainstem, hypothalamus, or auditory midbrain).

[0059] In some embodiments, the processing power of neural networks is leveraged to deliver higher resolution auditory information tailored to the individual characteristics of a recipient. In example embodiments, a neural network is deployed to deliver electric auditory stimulation that has been trained according to both the recipient's characteristic way of using electric hearing and a standard reference model of "normal" hearing. Furthermore, example embodiments of the present invention can accommodate a wide range of electric stimulation patterns, including focused multi-polar and other sensory electric stimulation therapies that require consideration of recipient-specific electric models with reference to a normal response model for non-electric stimulation.

[0060] Reference Figure 3 , shows a functional block diagram illustrating a method 300 of training a hearing machine learning (ML) model for use with certain techniques presented herein. Initially, a training set of audio signals or audio samples is provided to a hearing computational model 310. The hearing computational model can be a computational model of a normally hearing cochlea. These types of models are often computationally expensive and infeasible to implement on a hearing aid or cochlear implant sound processor.

[0061] The hearing computational model 310 processes the audio signals and produces an output represented as a neural electrogram (e.g., a normally hearing (NH) neural electrogram 320 as shown in Figure 3 Reference Figures 4A-4C , Figure 4A shows a spectrogram 400 for a rising complex tone. The spectrogram plots time along the X-axis, tone frequency along a first Y-axis, and power level (dB) of the tone along a second, opposite Y-axis, where the power level is represented by shading. Figure 4B shows a neural electrogram 410 of inner hair cell voltage for a rising complex tone. The neural electrogram 410 plots time along the X-axis, characteristic frequency of the tone along a first Y-axis, and voltage (in millivolts) of the inner hair cell along a second, opposite Y-axis, where the voltage is represented by shading. Figure 4CA neurogram 420 is shown for the auditory nerve fine structure for a rising complex tone. The neurogram 420 plots time along the X-axis, the characteristic frequency of the tone along a first Y-axis, and the number of spikes along a second, opposite Y-axis, where the number of spikes is represented by shading. The hearing computation model 310 can generate the NH neurogram 320 in the form of the neurogram 410 and / or the neurogram 420 for use with the example embodiments as described below. The hearing computation model can employ any conventional or other model of the auditory system that is capable of predicting the spike response of an auditory nerve fiber by modeling the middle ear, traveling wave along the basilar membrane, inner hair cell transduction, auditory nerve synapses, and auditory nerve spike behavior. The spectrogram 400 is typically used in conventional cochlear implant processors, while the neurograms 410, 420 can be used with the example embodiments and provide additional detail beyond the spectrogram 400.

[0062] The hearing machine learning (ML) model 350 can be trained to perform equivalent functions as the hearing computation model 310 and generate a neurogram (e.g., an ML neurogram 360 as shown in Figure 3 FIG. 4B) from an audio signal. The hearing ML model 350 can include any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feedforward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural network etc. ) to generate a neurogram. For example, the hearing ML model 350 can include a neural network that is substantially similar to the neural network described below (e.g., in Figure 6 ).

[0063] In a training phase, both the hearing computation model 310 and the hearing ML model 350 are provided with a training set of audio signals or audio samples. The hearing ML model processes the audio signals and produces an output that is represented as a neurogram (e.g., an ML neurogram 360 as shown in Figure 3 FIG. 4B) that indicates the firing or activation pattern of neurons in the auditory nerve. The NH neurogram 320 is compared to the ML neurogram 360 by a cost function 330, which provides a difference between these neurograms to train the hearing ML model 350. The cost function can employ any conventional or other cost or error function (e.g., mean absolute error, LI norm (e.g., sum of absolute differences of vector components), L2 norm (or Euclidean distance), weighted etc. ). For example, the cost function can employ a mean absolute error between the data values of the NH neurogram 320 and the ML neurogram 360 (e.g., sum of absolute values of errors (or differences) divided by the sample size etc.). Data values from the neurogram 320, 360 can correspond to the same dimensions (e.g., same neuron, same sampling frequency etc. ). The weights of the hearing ML model 350 are adjusted via any conventional or other training technique (e.g., backpropagation etc. ) to minimize a cost function of the difference (or error) between the quantized ML neurogram 360 and the NH neurogram 320. Training of the hearing ML model 350 is complete once the difference between the ML neurogram 360 and the NH neurogram 320 converges (e.g., the difference remains constant or within a threshold range for a particular period of time or a particular number of training iterations). The hearing ML model 350 can be used to generate a reference neurogram representing normal hearing as described by the example embodiments below.

[0064] Referring to Figure 5 , a functional block diagram illustrating a method 500 of training a sound processor machine learning (ML) model 540 to control stimulation in accordance with certain embodiments is shown. A training set of audio signals or audio samples is provided to a hearing model 510. The audio samples can include speech, music, broadband stimuli, and / or environmental or any other sound. The hearing model 510 can include the hearing computational model 310, or the previously trained hearing machine learning (ML) model 350, to generate a reference neurogram 520 representing normal hearing in substantially the same manner as described above. The audio signals can include microphone signals, outputs of a beamformer that combines multiple microphone signals, and / or audio signals from a telephone or other audio accessory. In addition, various pre-processing (e.g., automatic gain control (AGC), noise reduction etc. ). The hearing model 510 processes the audio signals and produces an output represented as a neurogram (e.g., the reference neurogram 520 as shown in Figure 5 ). The hearing model 510 can generate the reference neurogram 520 (e.g., in the form of the neurogram 410 and / or the neurogram 420). The audio signals can be divided into frames of any desired duration or length, and a neurogram or pattern of neural activation can be generated for each frame.

[0065] The sound processor machine learning (ML) model 540 can be trained to produce stimulation information (e.g., pulse information, analog information, etc.) that provides a stimulation neurogram 560 similar to the reference neurogram 520 representing normal hearing. The sound processor ML model 540 can employ any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feedforward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural networketc. ) to generate a neurogram. For example, the sound processor ML model 540 can employ a neural network as described below (e.g., in Figure 6 ).

[0066] In the training phase, a training set of audio signals or audio samples are also provided to the sound processor ML model 540. The sound processor ML model 540 processes the audio signals and produces information (e.g., pulse information, analog information, etc.) to be provided to the electric stimulation model 550. The information can include controls or characteristics of a stimulation signal (e.g., pulses) that can be used by the stimulator unit 142 to generate a stimulation signal for the implant 112. For example, the information can indicate current levels and / or other characteristics of the electrodes 144 of the implant 112 at corresponding times (e.g., which electrodes are active, current levels for the electrodes, times of activity etc. ).

[0067] The electric stimulation model 550 can be any conventional or other computational or machine learning model (e.g., finite element model, biophysical model, phenomenological model, neural network etc. ) of the neural response to a stimulation signal delivered by a cochlear implant. The electric stimulation model 550 models the effects of electric fields (e.g., current spread etc. ) and neural interfaces (e.g., neural thresholds to activate neurons, refractory times etc. ). Such a model is preferably specific to a particular cochlear implant recipient and includes information about the location and type of electrodes in the recipient’s cochlea, patterns of neural health along the cochlea, fibrosis and ossification within the cochlea, and details about the shape and size of the cochlea and the location of the target neural population. These patient-specific details can be determined through imaging (e.g., clinical CT scans etc. ), electrophysiological measurements (e.g., electrically evoked compound action potentials, EEG, electrocochleograms etc. ), psychophysical measurements (e.g., detection thresholds, amplitude modulation detection thresholds, masking tuning curves etc. ), or combinations thereof. For example, the location of the electrodes can be estimated from radiographic imaging (CT, x-ray, etc.) or from other surgical applications that use implant telemetry (impedance) to track and estimate electrode location during implantation.

[0068] For example, the electric stimulation model 550 can be implemented by a neural network and trained with information and a training set of corresponding known neural responses (e.g., as described below with respect to Figure 6 ) to produce a neural response (e.g., a firing or activation pattern of neurons in the auditory nerve etc.The electrical stimulation model 550 generates an output indicating the firing or activation patterns of neurons in the auditory nerve based on information generated by the sound processor machine learning (ML) model 540. The firing or activation patterns are represented as an electroneurogram (e.g., as shown in the image). Figure 5 The stimulus electroneurogram 560 is shown in the figure. A cost function 530 compares the reference electroneurogram 520 with the stimulus electroneurogram 560, providing the sound processor ML model 540 with the differences between these electroneurograms. The cost function can be any conventional or other cost or error function (e.g., mean absolute error, L1 norm (e.g., the sum of absolute differences of vector components), L2 norm (or Euclidean distance), or a weighted average applied to the difference or error). etc. For example, the cost function could be the mean absolute error between the data values ​​of the reference electroneurogram 520 and the stimulated electroneurogram 560 (e.g., the sum of the absolute values ​​of the errors (or differences) divided by the sample size). etc. Data values ​​from electroneurograms 520 and 560 can correspond to the same dimensions used to apply the cost function (e.g., the same neurons, the same sampling frequency). etc. (For example, via counterpropagation) etc. The weights of the sound processor ML model 540 are adjusted to minimize the cost function of the difference (or error) between the quantized reference electroencephalogram (EEG) 520 and the stimulation EEG 560. Training is complete once the difference between the reference EEG and the stimulation EEG converges (e.g., the difference remains constant or within a threshold range over a specific time period or number of training iterations), and the sound processor ML model 540 can be used in a sound processor of an exemplary embodiment (e.g., its stimulation generator model 192) to process audio signals and generate information to control the stimulator unit 142 to generate and apply stimulation signals for stimulation. Therefore, during the training phase, the sound processor ML model 540 is trained such that the stimulation EEG generated from the information produced by the sound processor ML model is a close approximation of the reference EEG representing normal hearing. The stimulation ML learning model 540 can be deployed to the device by providing the weights of the trained model.

[0069] For example, the sound processor ML model 540 can employ a neural network. Figure 6An exemplary neural network 600 is illustrated. The neural network 600 may include an input layer 610, one or more intermediate layers 620 (e.g., including any hidden layers), and an output layer 630. Each layer includes one or more neurons 650, wherein input layer neurons receive input (e.g., an audio signal or audio sample) and may be associated with weight values. Neurons in the intermediate and output layers are connected to one or more neurons in the preceding layer and receive the outputs of the connected neurons in the preceding layer as inputs. Each connection is associated with weight values, and each neuron produces an output based on a weighted combination of its inputs. The outputs of neurons may also be based on certain types of neural networks (e.g., recursive neural networks). etc. The bias value of ).

[0070] Weight (and bias) values ​​can be adjusted based on various training techniques. For example, a neural network can be machine-learned using a training dataset as input and a corresponding known or reference output, where the neural network attempts to produce the provided output and adjusts the weight (and bias) values ​​using the error between the output (e.g., the difference between the produced output and the known output) (e.g., via backpropagation or other training techniques).

[0071] The reference output corresponds to a reference electroneurogram (EMG) 520 representing normal hearing, relative to the sound processor machine learning (ML) model 540. In this case, the weights of the sound processor ML model 540 are adjusted to provide appropriate information to the electrical stimulation model 550 to generate a stimulated EMG 560 that matches or approximates the reference EMG 520 representing normal hearing. As described above, the difference between the reference EMG and the stimulated EMG is provided to adjust the weights of the sound processor ML model.

[0072] In some embodiments, as described below, feature vectors can be extracted from training set input data and used as input for training, while their known or reference corresponding outputs can be used as known or reference outputs for training. Feature vectors can include any suitable features of the training set input data. For example, features of an audio signal can include fundamental frequency or other frequencies, pitch, amplitude or intensity, spectrogram (magnitude and / or phase), and Mel-frequency cepstral coefficients. etc. .

[0073] The output layer of a neural network indicates the output obtained from the input data (e.g., impulse information). Figure 7 Output layer neurons can also indicate the probability of the output.

[0074] In some embodiments, the signal processing path is divided into independent modules to improve efficiency. (See reference) etc., showing a functional block diagram illustrating a method 700 of training a sound processor machine learning (ML) model using feature extraction applied to input audio signals according to certain embodiments. The example feature extraction process includes a Fast Fourier Transform (FFT) operating on a sliding time window. The FFT essentially transforms the audio signal from the time domain to the frequency domain. A training set of audio signals or audio samples is provided to a hearing model 510. The audio samples can include speech, music, broadband stimuli, and / or environmental or any other sound. The hearing model can include a hearing computation model 310, or a previously trained hearing machine learning (ML) model 350, to generate a reference neurogram 520 representing normal hearing in substantially the same manner as described above. The audio signals can include microphone signals, outputs of a beamformer combining multiple microphone signals, and / or audio signals from a telephone or other audio accessory. In addition, various pre-processing (e.g., automatic gain control (AGC), noise reduction Figure 7 ) can be applied to the audio signals. The hearing model 510 processes the audio signals and produces an output represented as a neurogram (e.g., a reference neurogram 520 as shown in etc. ). The hearing model 510 can generate the reference neurogram 520 in the form of the neurogram 410 and / or the neurogram 420 as described above. The audio signals can be divided into frames of any desired duration or length, and a neurogram or neural activation pattern can be generated for each frame.

[0075] A sound processor machine learning (ML) model 720 can be trained to produce a stimulation signal that provides a neurogram similar to the reference neurogram 520 representing normal hearing. The sound processor ML model 720 can employ any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feedforward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural network Figure 6 ) to generate a neurogram. For example, the sound processor ML model 720 can employ a neural network as described above (e.g., etc. ).

[0076] In the training phase, a training set of audio signals or audio samples is also provided to the feature extraction module 710 to extract features from the audio signals. The features can include any desired features or attributes (e.g., spectrograms, features related to normal hearing etc.). In an embodiment, the feature extraction module can perform a fast Fourier transform (FFT) on the audio signal to extract features therefrom. The FFT essentially transforms the audio signal from the time domain to the frequency domain. The extracted features can include the output of the FFT, magnitudes, phases, frequencies, mel-frequency cepstral coefficients (MFCCs), and / or any other signal features. The extracted features are provided to the sound processor ML model 720. The sound processor ML model 720 processes the extracted features and produces information to be provided to the electric stimulation model 550. For example, as described above, the information can indicate the current levels and / or other characteristics of the electrodes 144 of the implant 112 at corresponding times (e.g., which electrodes are active, current levels for the electrodes, times of activity etc. ). The electric stimulation model can be any conventional or other computational or machine learning model of the neural response to a stimulation signal delivered by a cochlear implant (e.g., finite element model, biophysical model, phenomenological model, neural network Figure 7 ), and is substantially similar to the electric stimulation model described above.

[0077] The electric stimulation model 550 produces an output indicative of a firing or activation pattern of neurons in the auditory nerve based on the information produced by the sound processor machine learning (ML) model 720. The firing or activation pattern is represented as a neural electrogram (e.g., a stimulation neural electrogram 730 as shown in FIG. 7B). The reference neural electrogram 520 is compared to the stimulation neural electrogram 730 by the cost function 530, which determines and provides to the sound processor machine learning (ML) model 720 the difference between these neural electrograms in substantially the same manner as described above. etc. The weights of the sound processor ML model 720 are adjusted (e.g., via backpropagation Figure 8 ) in substantially the same manner as described above to minimize the cost function of the difference (or error) between the quantized reference neural electrogram 520 and the stimulation neural electrogram 730.

[0078] Once the difference between the reference neural electrogram 520 and the stimulation neural electrogram 730 converges (e.g., remains constant or within a threshold range for a particular period of time or a particular number of training iterations), the training is complete, and the feature extraction module 710 and the sound processor ML model 720 can be used in the sound processor (e.g., its stimulation generator model 192) of the example embodiment to extract and process audio signal features and produce information to control the stimulator unit 142 to produce and apply stimulation signals for stimulation.

[0079] Thus, in the training phase, the reference output for the sound processor machine learning (ML) model 720 corresponds to the reference neurogram 520 representing normal hearing. In this case, the weights of the sound processor ML model 720 are adjusted to provide appropriate information to the electric stimulation model 550 to produce a stimulated neurogram 730 of normal hearing that matches or approximates the reference neurogram 520. As described above, the difference between the reference neurogram and the stimulated neurogram is provided for adjusting the weights of the sound processor ML model 720. Thus, the sound processor ML model 720 is trained such that the stimulated neurogram generated from the information produced by the sound processor ML model 720 is a close approximation of the reference neurogram representing normal hearing. The stimulation ML learning model 720 can be deployed to a device by providing the weights of the trained model.

[0080] In some embodiments, a filter bank process can act as a feature extractor and be applied to the audio signal. Referring to etc. , a functional block diagram illustrating a method 800 of training a sound processor machine learning (ML) model with another feature extraction applied to an input audio signal is shown, in accordance with certain embodiments. The example feature extraction process includes a filter bank process. A set of training audio signals or audio samples are provided to a hearing model 510. The audio samples can include speech, music, broadband stimuli, and / or environmental or any other sound. The hearing model can include a hearing computational model 310, or a previously trained hearing machine learning (ML) model 350, to generate a reference neurogram 520 representing normal hearing in substantially the same manner as described above. The audio signals can include a microphone signal, an output of a beamformer that combines multiple microphone signals, and / or an audio signal from a telephone or other audio accessory. In addition, various pre-processing can be applied (e.g., automatic gain control (AGC), noise reduction Figure 8 ). The hearing model 510 processes the audio signals and produces an output represented as a neurogram (e.g., the reference neurogram 520 as shown in etc. ). The hearing model 510 can generate the reference neurogram 520 in the form of the neurogram 410 and / or the neurogram 420 as described above. The audio signals can be divided into frames of any desired duration or length, and a neurogram or neural activation pattern can be generated for each frame.

[0081] A sound processor machine learning (ML) model 820 can be trained to generate stimulus signals that provide an electroneurogram similar to a reference electroneurogram 520 representing normal hearing. The sound processor ML model 820 can employ any conventional or other machine learning model (e.g., mathematical / statistical models; classifiers; decision trees; random forests; feedforward, recursion, convolution, convolutional recursion, deep learning, gating, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural networks). Figure 6 ) to generate an electroneurogram. For example, the sound processor ML model 820 can employ the methods described above (e.g., Figure 2 ) neural network.

[0082] During the training phase, a training set of audio signals or audio samples is also provided to the filter bank unit 810 to extract features from the audio signals. In an embodiment, the filter bank unit generates a suitable set of bandwidth-limited channels or frequency partitions, each including spectral components of the received audio. Various features can be extracted from the resulting channels or partitions and provided to the sound processor machine learning (ML) model 820 (e.g., the output of the filter bank processing, frequency partitions or channels, frequency, phase, amount of partitions or channels, amplitude, etc.). The filter bank unit can perform any conventional or other filter bank process (e.g., corresponding to...). etc. Filter bank module 256 etc. The number of filters in the filter bank can be equal to the number of electrical stimulation channels in the cochlear implant. The extracted features are provided to the sound processor ML model 820. The sound processor ML model 820 processes the extracted features and generates information to be provided to the electrical stimulation model 550. For example, as described above, the information may indicate the current level and / or other characteristics of the electrodes 144 of the implant 112 at corresponding times (e.g., which electrodes are active, the current level used for the electrodes, the duration of activity). etc. The electrical stimulation model 550 can be any conventional or other computational or machine learning model of the neural response to stimulation signals delivered by the cochlear implant (e.g., finite element model, biophysical model, phenomenological model, neural network). Figure 8 And it is basically similar to the above-mentioned electrical stimulation model.

[0083] The electrical stimulation model 550 generates an output indicating the firing or activation patterns of neurons in the auditory nerve based on information produced by the sound processor machine learning (ML) model 820. The firing or activation patterns are represented as an electroneurogram (e.g., as shown in the image). etc.The reference neurogram 520 is compared to the stimulated neurogram 830 by a cost function 530, which determines the difference between these neurograms in substantially the same manner as described above and provides the sound processor ML model 820 with the difference between these neurograms. (For example, via backpropagation Figure 9 ) The weights of the sound processor ML model 820 are adjusted to minimize the cost function of the difference (or error) between the quantized reference neurogram 520 and the stimulated neurogram 830.

[0084] Once the difference between the reference neurogram 520 and the stimulated neurogram 830 converges (e.g., the difference remains constant or within a threshold range for a particular period of time or a particular number of training iterations), the training is complete, and the filter bank module 810 and the sound processor ML model 820 can be used in the sound processor (e.g., its stimulation generator model 192) of the example embodiment to extract and process audio signal features and produce information to control the stimulator unit 142 to produce and apply stimulation signals for stimulation.

[0085] Thus, in the training phase, the reference output for the sound processor machine learning (ML) model 820 corresponds to the reference neurogram 520 representing normal hearing. In this case, the weights of the sound processor ML model 820 are adjusted to provide the appropriate information to the electric stimulation model 550 to produce a stimulated neurogram 830 of normal hearing that matches or approximates the reference neurogram 520. As described above, the difference between the reference neurogram and the stimulated neurogram is provided for adjusting the weights of the sound processor ML model 820. Thus, the sound processor ML model 820 is trained such that the stimulated neurogram generated from the information produced by the sound processor ML model 820 is a close approximation of the reference neurogram representing normal hearing. The stimulation ML learning model 820 can be deployed to a device by providing the trained model’s weights.

[0086] In embodiments, the stimulation generator model 192 can process the neurogram representing normal hearing produced from the hearing model to generate information for the electric stimulation model. In this case, the stimulation generator model 192 is trained to cancel out the effects of the electric stimulation on the recipient (e.g., modeled or introduced by the electric stimulation model) to produce a neurogram that matches or approximates normal hearing.

[0087] Reference etc., showing a functional block diagram illustrating a method 900 of training a sound processor machine learning (ML) model with input audio signals using electroencephalography, in accordance with certain embodiments. A hearing model 510 is provided with a set of training audio signals or audio samples. The audio samples can include speech, music, broadband stimuli, and / or environmental or any other sound. The hearing model can include a hearing computation model 310, or a previously trained hearing machine learning (ML) model 350, to generate a reference electroencephalogram 520 representing normal hearing in substantially the same manner as described above. The audio signals can include microphone signals, outputs of a beamformer that combines multiple microphone signals, and / or audio signals from a telephone or other audio accessory. In addition, various pre-processing (e.g., automatic gain control (AGC), noise reduction Figure 9 ) can be applied. The hearing model 510 processes the audio signals and produces an output represented as an electroencephalogram (e.g., the reference electroencephalogram 520 as shown in etc. ) that indicates a firing or activation pattern of neurons in the auditory nerve. The hearing model 510 can generate the reference electroencephalogram 520 in the form of the electroencephalogram 410 and / or the electroencephalogram 420 as described above. The audio signals can be divided into frames of any desired duration or length, and an electroencephalogram or neural activation pattern can be generated for each frame.

[0088] A sound processor machine learning (ML) model 920 can be trained to produce stimulation signals that provide an electroencephalogram similar to the reference electroencephalogram 520 representing normal hearing. The sound processor ML model 920 can employ any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feedforward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural network Figure 6 ) to generate an electroencephalogram. For example, the sound processor ML model 920 can employ a neural network as described above (e.g., etc. ).

[0089] During the training phase, the sound processor ML model 920 is also provided with the reference electroencephalogram 520 from the hearing model 510. The sound processor ML model 920 processes the reference electroencephalogram and produces information to be provided to an electrical stimulation model 550. For example, as described above, the information can indicate current levels and / or other characteristics of the electrodes 144 of the implant 112 at corresponding times (e.g., which electrodes are active, current levels for the electrodes, times of activity etc. ). The electrical stimulation model 550 can be any conventional or other computational or machine learning model (e.g., finite element model, neural network Figure 9), and is substantially similar to the electrical stimulation model described above.

[0090] The electrical stimulation model 550 produces an output indicative of a firing or activation pattern of neurons in the auditory nerve based on information produced by the sound processor machine learning (ML) model 920. The firing or activation pattern is represented as a neural electrogram (e.g., as shown in etc. The reference neural electrogram 520 is compared to the stimulation neural electrogram 930 by a cost function 530, which determines and provides to the sound processor ML model 920 a difference between these neural electrograms in substantially the same manner as described above. (For example, via backpropagation Figure 10 The weights of the sound processor ML model 920 are adjusted to minimize the cost function of the difference (or error) between the quantized reference neural electrogram 520 and the stimulation neural electrogram 930.

[0091] Once the difference between the reference neural electrogram 520 and the stimulation neural electrogram 930 converges (e.g., the difference remains constant or within a threshold range for a particular period of time or a particular number of training iterations), the training is complete, and the hearing model 510 and the sound processor ML model 920 can be used in the sound processor (e.g., its stimulation generator model 192) of the example embodiment to process audio signals and produce information in substantially the same manner as described above. The information controls the stimulator unit 142 to produce and apply stimulation signals for stimulation.

[0092] Thus, in the training phase, the reference output for the sound processor machine learning (ML) model 920 corresponds to the reference neural electrogram 520 representing normal hearing. In this case, the weights of the sound processor ML model 920 are adjusted to provide the electrical stimulation model 550 with appropriate information to produce a stimulation neural electrogram 930 of normal hearing that matches or approximates the reference neural electrogram 520. As described above, the difference between the reference neural electrogram and the stimulation neural electrogram is provided for adjusting the weights of the sound processor ML model 920. Thus, the sound processor ML model 920 is trained such that the stimulation neural electrogram generated from the sound processor ML model 920 is a close approximation of the reference neural electrogram representing normal hearing. In other words, the sound processor ML model 920 is trained to remove the effects of electrical stimulation on the recipient (e.g., modeled or introduced by the electrical stimulation model) to produce a neural electrogram that matches or approximates normal hearing. The stimulation ML learning model 920 can be deployed to a device by providing the weights of the trained model.

[0093] In some embodiments, in addition to emulating the behavior of a normally hearing cochlea, a noise reduction capability is provided. In this case, during training, a clean audio signal is provided to the hearing model to produce a reference neurogram representing normal hearing, while a noisy audio signal (e.g., adding noise to the clean audio signal) is provided to generate a neurogram for training the stimulus generator model. The stimulus generator model is trained to minimize the difference between the clean reference neurogram and the resulting stimulus neurogram from the noisy signal. The stimulus generator model thus performs noise reduction and cancels out the effects of the electrical stimulation on the recipient (modeled or introduced by the electrical stimulus generator model).

[0094] Referring to etc. , a functional block diagram illustrating a method 1000 of training a sound processor machine learning (ML) model to reduce noise according to certain embodiments is shown. A clean training audio signal or a set of audio samples is provided to a first hearing model 510A. The audio samples can include speech, music, broadband stimuli, and / or environmental or any other sound. The set of clean audio signals is also provided to a mixer 1010, which introduces noise to produce a noisy signal. The noise can include any type of noise (e.g., from the surrounding environment, synthetic noise, noisy human voice noise, reverberation or other convolutional noise, echo, cafe / restaurant noise Figure 10 ). The noisy signal is provided to a second hearing model 510B. The hearing models 510A, 510B can include a hearing computational model 310, or a previously trained hearing machine learning (ML) model 350, to generate a neurogram representing normal hearing in substantially the same manner as described above. The clean audio signal (before the noise is introduced) can include a microphone signal, an output of a beamformer that combines multiple microphone signals, and / or an audio signal from a telephone or other audio accessory. The hearing model 510A processes the clean audio signal and produces an output represented as a neurogram (e.g., a reference neurogram 520 as shown in Figure 10 ). The hearing model 510A can generate the reference neurogram 520 in the form of the neurogram 410 and / or the neurogram 420 as described above. Similarly, the hearing model 510B processes the noisy audio signal and produces an output represented as a neurogram (e.g., a noisy neurogram 1015 as shown in etc. ). The hearing model 510B can generate the noisy neurogram 1015 in the form of the neurogram 410 and / or the neurogram 420 as described above. As described above, the audio signal can be divided into frames of any desired duration or length, and a neurogram or neural activation pattern can be generated for each frame.

[0095] The sound processor machine learning (ML) model 1020 can be trained to produce stimulation signals that provide a neurogram that is similar to the reference neurogram 520 representing normal hearing. The sound processor ML model 1020 can employ any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feed-forward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural network Figure 6 ) to generate a neurogram. For example, the sound processor ML model 1020 can employ a neural network as described above (e.g., etc. ).

[0096] In the training phase, the sound processor ML model 1020 is provided with the noisy neurogram 1015 from the hearing model 510B. The sound processor ML model 1020 processes the noisy neurogram and produces information to be provided to the electric stimulation model 550. For example, as described above, the information can indicate the current levels and / or other characteristics of the electrodes 144 of the implant 112 at corresponding times (e.g., which electrodes are active, current levels for the electrodes, times of activity etc. ). The electric stimulation model 550 can be any conventional or other computational or machine learning model (e.g., finite element model, neural network Figure 10 ) of the neural response to stimulation signals delivered by a cochlear implant, and is substantially similar to the electric stimulation model described above.

[0097] The electric stimulation model 550 produces an output indicating the firing or activation pattern of neurons in the auditory nerve based on the information produced by the sound processor machine learning (ML) model 1020. The firing or activation pattern is represented as a neurogram (e.g., a stimulation neurogram 1030 as shown in etc. ). The reference neurogram 520 (produced from clean audio signals) is compared to the stimulation neurogram 1030 (produced from noisy audio signals) by the cost function 530, which determines and provides to the sound processor ML model 1020 the difference between these neurograms in substantially the same manner as described above. (For example, via backpropagation Figure 1D ) adjusts the weights of the sound processor ML model 1020 to minimize the cost function of the difference (or error) between the quantized reference neurogram 520 and the stimulation neurogram 1030.

[0098] Once the difference between the reference neurogram 520 and the stimulated neurogram 1030 converges (e.g., the difference remains constant or within a threshold range over a particular time period or a particular number of training iterations), training is complete, and the hearing model 510A and the sound processor ML model 1020 can be used in a sound processor (e.g., its stimulation generator model 192) of an example embodiment to process an audio signal and produce information in substantially the same manner as described above. The information controls the stimulator unit 142 to produce and apply a stimulation signal for stimulation.

[0099] Thus, in the training phase, the reference output for the sound processor machine learning (ML) model 1020 corresponds to the reference neurogram 520 that represents normal hearing. In this case, the weights of the sound processor ML model 1020 are adjusted to provide the appropriate information to the electric stimulation model 550 to produce a stimulated neurogram 1030 of normal hearing that matches or closely approximates the reference neurogram 520. As described above, the difference between the reference neurogram and the stimulated neurogram is provided for adjusting the weights of the sound processor ML model 1020. Thus, the sound processor ML model 1020 is trained such that the stimulated neurogram generated from the information produced by the sound processor ML model 1020 based on a noisy audio signal is a close approximation of the reference neurogram of normal hearing for a clean audio signal. In other words, the sound processor ML model 1020 is trained to remove the effects of noise and electric stimulation on the recipient (e.g., modeled or introduced by the electric stimulation model) to produce a neurogram of normal hearing that matches or closely approximates a clean audio signal. The sound processor ML learning model 1020 can be deployed to a device by providing the trained model’s weights.

[0100] Although commercial cochlear implants typically use monopolar stimulation, example embodiments can be used with any stimulation mode. The spatial resolution of the electric stimulation can be controlled, for example, by using different electrode configurations for a given stimulation channel to activate different widths of neural cell regions. For example, monopolar stimulation is an electrode configuration in which, for a given stimulation channel, current is “pulled out” via one of the intracochlear electrodes 144, but current is “pumped in” by an electrode external to the cochlea (sometimes referred to as an extracochlear electrode (ECE) 139 etc. Other types of electrode configurations, such as bipolar, tripolar, focused multipolar (FMP), also known as “phased array” stimulation etc.Typically, the size of the stimulated neural population is reduced by "pulling out" current via one or more of the cochlear electrodes 144, while simultaneously "infusing" current via one or more other adjacent cochlear electrodes. Bipolar, tripolar, focused multipolar, and other electrode configurations that both pull out and infuse current via the cochlear electrodes are generally and collectively referred to herein as "focused" stimulation. Compared to monopolar stimulation, focused stimulation typically exhibits less current diffusion (i.e., a narrower stimulation pattern) and therefore higher spatial resolution. Similarly, other electrode configurations, such as bipolar patterns, virtual channels, wide channels, and defocused multipolar, also exhibit this characteristic. Figure 11A The size of the stimulated nerve group is usually increased by “pulling out” current through multiple adjacent cochlear electrodes.

[0101] Exemplary embodiments may utilize more advanced stimulation models focusing on multipolar stimulation. Given multipolar information derived from channel amplitude (for stimulation channels) by a sound processor machine learning (ML) model, the electrical stimulation model computes an electroneurogram. The stimulation generator model 192 is trained to eliminate the effects of electrical stimulation on the recipient (e.g., modeled or introduced by the electrical stimulation model) to produce an electroneurogram that matches or approximates normal hearing.

[0102] refer to etc. This diagram illustrates a functional block diagram of a method 1100 for training a sound processor machine learning (ML) model for focused multipolar stimulation using an electroneurogram of an input audio signal, according to certain embodiments. The focused multipolar stimulation parameters can be individualized using electrical measurements from the cochlea (e.g., transimpedance matrix or electrically evoked compound action potentials). A set of training audio signals or audio samples is provided to a hearing model 510. The audio samples may include speech, music, broadband stimulation, and / or ambient or any other sound. The hearing model may include a hearing computational model 310, or a previously trained hearing machine learning (ML) model 350, to generate a reference electroneurogram 520 representing normal hearing in substantially the same manner as described above. The audio signals may include microphone signals, the output of a beamformer combining multiple microphone signals, and / or audio signals from a telephone or other audio accessory. Furthermore, various preprocessing techniques (e.g., automatic gain control (AGC), noise reduction, etc.) can be applied. Figure 11A The hearing model 510 processes audio signals and generates a neural electrogram (e.g., as shown in the image). etc.The output of the hearing model 510 (e.g., a reference neurogram 520 as described above with respect to FIG. 5) indicates a firing or activation pattern of neurons in the auditory nerve. The hearing model 510 can generate the reference neurogram 520 in the form of the neurogram 410 and / or the neurogram 420 as described above. The audio signal can be divided into frames of any desired duration or length, and a neurogram or neural activation pattern can be generated for each frame.

[0103] The sound processor machine learning (ML) model 1120 can be trained to produce channel amplitudes to be provided to the focused multipolar pulse generator 1123. The focused multipolar pulse generator produces focused multipolar pulses for providing a neurogram that resembles the reference neurogram 520 representing normal hearing. The sound processor ML model 1120 can employ any conventional or other machine learning model (e.g., mathematical / statistical model; classifier; decision tree; random forest; feedforward, recurrent, convolutional, convolutional recurrent, deep learning, gated, long short-term memory (LSTM), self-attention, encoder / decoder, or other neural network Figure 6 ) to generate a neurogram. For example, the sound processor ML model 1120 can employ a neural network as described above (e.g., with respect to FIG. 5). etc.

[0104] In the training phase, the sound processor ML model 1120 is also provided with the reference neurogram 520 from the hearing model 510. The sound processor ML model 1120 processes the reference neurogram and produces channel amplitudes to be provided to the multipolar pulse generator 1123. The multipolar pulse generator generates multipolar pulse information to be provided to the electric stimulation model 1125. For example, the multipolar pulse information can indicate current levels and / or other characteristics (e.g., which electrodes are active, current levels for the electrodes, times etc. ) for a particular set of electrodes 144 of the implant 112 to stimulate at a corresponding time, and provides finer control to activate smaller groups or sets of neurons. The electric stimulation model 1125 can be any conventional or other computational or machine learning model (e.g., finite element model, neural network Figure 11A ) of the neural response to focused multipolar stimulation pulses delivered by a cochlear implant, and is substantially similar to the electric stimulation model described above.

[0105] The electric stimulation model 1125 produces an output indicating a firing or activation pattern of neurons in the auditory nerve based on the multipolar pulse information produced from the channel amplitudes of the sound processor machine learning (ML) model 1120. The firing or activation pattern is represented as a neurogram (e.g., as described above with respect to FIG. 5). etc. ​The reference neurogram 520 and the stimulated neurogram 1130 are compared by a cost function 530, which determines the difference between these neurograms in substantially the same manner as described above and provides the difference to the sound processor ML model 1120. (For example, via backpropagation Figure 11B ) The weights of the sound processor ML model 1120 are adjusted to minimize the cost function of the difference (or error) between the quantized reference neurogram 520 and the stimulated neurogram 1130.

[0106] Once the difference between the reference neurogram 520 and the stimulated neurogram 1130 converges (e.g., the difference remains constant or within a threshold range for a particular period of time or a particular number of training iterations), the training is complete, and the hearing model 510, the sound processor ML model 1120, and the multi-polar pulse generator 1123 can be used in the sound processor (e.g., its stimulation generator model 192) of the example embodiment to process audio signals and produce channel amplitudes and multi-polar pulse information in substantially the same manner as described above. The multi-polar pulse information controls the stimulator unit 142 to produce and apply focused multi-polar stimulation pulses for stimulation.

[0107] Thus, in the training phase, the reference output for the sound processor machine learning (ML) model 1120 corresponds to the reference neurogram 520 that represents normal hearing. In this case, the weights of the sound processor ML model 1120 are adjusted to provide appropriate channel amplitudes to produce multi-polar pulse information for the electric stimulation model 1125. The electric stimulation model 1125 produces a stimulated neurogram 1130 that matches or approximates normal hearing of the reference neurogram 520. As described above, the difference between the reference neurogram and the stimulated neurogram is provided for adjusting the weights of the sound processor ML model 1120. Thus, the sound processor ML model 1120 is trained such that the stimulated neurogram generated from the multi-polar pulse information derived from the channel amplitudes of the sound processor ML model 1120 is a close approximation of the reference neurogram that represents normal hearing. In other words, the sound processor ML model 1120 is trained to remove the effects (e.g., modeled or introduced by the electric stimulation model) of focused multi-polar stimulation on the recipient to produce a neurogram that matches or approximates normal hearing. The sound processor ML learning model 1120 can be deployed to the device by providing the weights (and other parameters) of the trained model.

[0108] Reference Figure 11B, showing a functional block diagram illustrating a method 1150 of training a sound processor machine learning (ML) model for focused multipolar stimulation using input neurogram of an audio signal in accordance with certain example embodiments. The method 1150 is substantially similar to the method 1100 described above, except that the sound processor ML model 1120 includes a multipolar pulse generator 1123 and directly generates multipolar pulse information.

[0109] In the training phase, the sound processor ML model 1120 is provided with the reference neurogram 520 from the hearing model 510 as described above. The sound processor ML model 1120 processes the reference neurogram and produces focused multipolar pulse information to be provided to the electric stimulation model 1125. The electric stimulation model 1125 produces an output indicative of a firing or activation pattern of neurons in the auditory nerve based on the multipolar pulse information produced by the sound processor ML model 1120. The firing or activation pattern is represented as a stimulation neurogram (e.g., stimulation neurogram 1130 as shown in FIG. 11B). etc.

[0110] The reference neurogram 520 is compared to the stimulation neurogram 1130 by a cost function 530, which determines the difference between these neurograms and provides the difference to the sound processor ML model 1120 in substantially the same manner as described above. (For example, via backpropagation etc. ) The weights of the sound processor ML model 1120 are adjusted to minimize the cost function of the difference (or error) between the quantized reference neurogram 520 and the stimulation neurogram 1130.

[0111] Once the difference between the reference neurogram 520 and the stimulation neurogram 1130 converges (e.g., the difference remains constant or within a threshold range for a particular period of time or a particular number of training iterations), the training is complete, and the hearing model 510 and the sound processor ML model 1120 can be used in the sound processor (e.g., its stimulation generator model 192) of the example embodiments to process audio signals and produce focused multipolar pulse information in substantially the same manner as described above. As described above, the multipolar pulse information controls the stimulator unit 142 to produce and apply focused multipolar stimulation pulses for stimulation. As described above, the stimulation ML learning model 1120 can be deployed to a device by providing the weights of the trained model.

[0112] The example embodiments can be used for any stimulation mode in which various machine learning models (e.g., sound processor machine learning (ML) models 540, 720, 820, 920, 1020, 1120 etc. ) of the example embodiments can be trained to produce pulse information for one or more stimulation modes. For example, training data including stimulation model data (e.g., audio signals, neurograms, features etc. ​) to train a machine learning model, the stimulation model data indicating a type of stimulation model (e.g., monopolar, focused multipolar etc. ). This additional data effectively creates an independent space for each stimulation pattern. The machine learning model can be trained in much the same way as described above to map the input (with stimulation model data) to pulse information corresponding to the space for the stimulation model indicated by the stimulation model data.

[0113] The various machine learning models of the example embodiments (e.g., sound processor machine learning (ML) models 540, 720, 820, 920, 1020, 1120 etc. ) can be pre-trained for a generic or average cochlea (e.g., similar to the hearing machine learning model 350 described above). In this case, these machine learning models are later retrained with recipient-specific data to customize the machine learning models for a particular recipient. This enables faster training of the machine learning models with less training data.

[0114] The generation of information by the sound processor machine learning (ML) models can be performed on an external device (e.g., external component 104 etc. ) and / or on another computing system (e.g., computing device 110 etc. ) in communication with the external device. In cases where the information is generated on the other computing system (e.g., to save processing and / or battery life etc. ), the other computing system can send the information to the external device. The sound processor machine learning models can be trained with training data on the external device, the other computing system, and / or a separate system, and deployed for use on the external device and / or the other computing system. Additionally, the sound processor machine learning models can be dynamically or continuously updated or trained (and deployed) based on new information collected and obtained from the implant.

[0115] The sound processor machine learning (ML) models can be trained in various ways. The sound processor ML models can be trained with training data (e.g., predetermined training data, data from simulated or actual hardware Figure 12 ) on the external device, the other computing system, and / or a separate system, and deployed for use on the external device and / or the other computing system. Additionally, the sound processor ML models can be dynamically or continuously updated or trained (and deployed) based on new information collected from the implant. The training can be performed by the stimulation control logic 185.

[0116] In some embodiments, sound processor machine learning (ML) models can be trained using various training data. For example, the training data can include a variety of audio files that provide various different scenarios. The scenarios can include real-world examples. These various scenarios can be used as training data to train sound processor ML models to be consistent with normal hearing.

[0117] Sound processor machine learning (ML) models can be trained using the entirety or any portion of the training data in substantially the same manner as described above.

[0118] In some embodiments, the technology presented herein can also be implemented by or used in conjunction with vestibular devices (e.g., vestibular implants), visual devices (i.e., bionic eyes), sensors, pacemakers, drug delivery systems, defibrillators, functional electrical stimulation devices, catheters, seizure devices (e.g., devices for monitoring and / or treating epileptic events), sleep apnea devices, electroporation devices Figure 13 implementations, or in conjunction with them. The electrical stimulation models can model the physiological effects (or physiological responses to them) of the determined information, which can be used to train corresponding sound processor machine learning (ML) models to determine stimulation signals (e.g., pulses) that closely approximate or match normal physiological or sensory function (e.g., vision, olfaction, hearing, cardiac function, etc.) in substantially the same manner as described above. Thus, the technology of example embodiments can be applied to any system that provides electrical stimulation to improve or compensate for improper or degraded physiological or sensory function in substantially the same manner as described above.

[0119] Reference is now made to etc. depicted a flowchart of a method 1200 for implementing the technology of the present disclosure. The method 1200 begins with operations at 1205, which can include receiving, at an implantable medical device system, a signal associated with a physiological function. At 1210, the method can include determining, by a machine learning model, information for a stimulation signal to stimulate the physiological function based on the signal. The machine learning model is trained based on modeling physiological effects from the stimulation. At 1215, the method can include controlling stimulation of a recipient of the implantable medical device system based on the determined information. Thus, the method of flowchart 1200 provides a process that can determine and control stimulation based on machine learning.

[0120] Reference is now made to ​wherein a flowchart of a method 1300 for implementing the techniques of the disclosure is depicted. The method 1300 begins with the operation at 1305, which can include determining, by a machine learning model of at least one processor, information for a stimulation signal to stimulate a physiological function based on a signal associated with the physiological function. At 1310, the method can include modeling, via the at least one processor, a physiological effect from the stimulation signal. At 1315, the method can include updating, via the at least one processor, the machine learning model based on a difference between the modeled physiological effect and a reference physiological effect representative of normal physiological function. Thus, the method of flowchart 1300 provides a process for training a machine learning model to control stimulation.

[0121] It should be appreciated that while certain uses of the technology have been explained and discussed above, the disclosed technology can be used with a variety of devices according to many examples of the technology. The above discussion is not intended to be representative of the technology as being suitable only within systems similar to those shown in the figures. In general, additional configurations can be used to practice the processes and systems herein and / or some aspects described can be excluded without departing from the processes and systems disclosed herein.

[0122] The present disclosure describes some aspects of the inventive technology with reference to the accompanying figures, of which only a few possible aspects are illustrated. Other aspects can, however, be embodied in many different forms and should not be interpreted as being limited to the aspects set forth herein. Rather, these aspects are provided as illustrative examples of the disclosure to convey the scope of the aspects possible. As will be apparent, a person having ordinary skill in the art

[0123] It should be appreciated that the various aspects (e.g., portions, components ​ ) described herein with respect to the figures are not intended to limit the systems and processes to the particular aspects described. Thus, additional configurations can be used to practice the methods and systems herein and / or some aspects described can be excluded without departing from the methods and systems disclosed herein.

[0124] According to certain aspects, systems and non-transitory computer-readable storage media are provided. The systems are configured with hardware configured to perform operations similar to the methods of the present disclosure. One or more non-transitory computer-readable storage media include instructions that, when executed by one or more processors, cause the one or more processors to perform operations similar to the methods of the present disclosure.

[0125] Similarly, where steps of a process are disclosed, these steps are described for purposes of illustrating the present methods and systems, and are not intended to limit the present disclosure to a particular sequence of steps. For example, the steps can be performed in different sequences, two or more steps can be performed concurrently, additional steps can be performed, and steps can be excluded without departing from the present disclosure. Further, the disclosed processes can be repeated.

[0126] Although specific aspects are described herein, the scope of the technology is not limited to these specific aspects. Those skilled in the art will recognize other aspects or improvements subsumed by the general scope of the described technology. Accordingly, the specific structures, acts, or media are merely exemplary aspects and not limiting of the scope of the technology as described.

[0127] It will also be appreciated that embodiments presented herein are not mutually exclusive of one another, and that various embodiments can be combined with another embodiment in any of a variety of different ways.

Claims

1. A method comprising: receiving, at an implantable medical device system, a signal associated with a physiological function; determining, by a machine learning model, information associated with a stimulation signal for stimulating the physiological function based on the signal, wherein the machine learning model is trained based on modeling of a physiological effect from the stimulation; and controlling the stimulation of a recipient of the implantable medical device system based on the determined information.

2. The method of claim 1, wherein the machine learning model comprises a neural network.

3. The method of claim 1 or 2, wherein the physiological function comprises hearing.

4. The method of claim 3, wherein the machine learning model is trained to minimize a difference between a normal activation pattern of neurons in an auditory nerve and a stimulation activation pattern generated based on the determined information.

5. The method of claim 4, wherein the machine learning model is trained until a cost function indicative of an error between the normal activation pattern and the stimulation activation pattern converges.

6. The method of claim 4, wherein the stimulation activation pattern is generated from the modeling of the physiological effect of the determined information.

7. The method of claim 4, wherein the physiological effect comprises an effect of a stimulation signal based on the determined information on a cochlea.

8. The method of claim 4, wherein the normal activation pattern and the stimulation activation pattern are each represented by a neurogram indicative of an inner hair cell voltage.

9. The method of claim 4, wherein the normal activation pattern and the stimulation activation pattern are each represented by a neurogram indicative of spikes in an auditory nerve fine structure.

10. The method of claim 4, wherein the normal activation pattern and the stimulation activation pattern are each represented by a neurogram indicative of a central process of an auditory system.

11. The method of claim 2, further comprising: extracting one or more features from the signal, wherein the machine learning model determines the information based on the extracted features.

12. The method of claim 11, wherein extracting the one or more features comprises: extracting the one or more features from a fast Fourier transform (FFT) of the signal.

13. The method of claim 11, wherein extracting the one or more features comprises: extracting the one or more features from a filter bank of the signal.

14. The method of claim 1, wherein the machine learning model determines the information based on an activation pattern of neurons in an auditory nerve from an audio signal.

15. The method of claim 1, wherein the machine learning model determines the information based on an activation pattern of neurons in an auditory nerve from an audio signal combined with noise.

16. The method of claim 1, wherein the stimulation signal corresponds to monopolar stimulation.

17. The method of claim 1, wherein the stimulation signal corresponds to focused multipolar stimulation.

18. The method of claim 1, wherein the machine learning model is pre-trained on average physiological function and retrained with recipient-specific data.

19. The method of claim 1, wherein the modeling of the physiological effect is recipient-specific.

20. One or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors, cause the one or more processors to: receive, at an implantable medical device system, a signal associated with a physiological function; determine, by a machine learning model, information for a stimulation signal to stimulate the physiological function based on the signal, wherein the machine learning model is trained based on modeling of a physiological effect from the stimulation; and control the stimulation of a recipient of the implantable medical device system based on the determined information.

21. The one or more non-transitory computer-readable storage media of claim 20, wherein the machine learning model comprises a neural network.

22. The one or more non-transitory computer-readable storage media of claim 20 or 21, wherein the physiological function comprises hearing.

23. The one or more non-transitory computer-readable storage media of claim 22, wherein the machine learning model is trained to minimize a difference between a normal activation pattern of neurons in an auditory nerve and a stimulation activation pattern generated based on the determined information.

24. The one or more non-transitory computer-readable storage media of claim 23, wherein the machine learning model is trained until a cost function indicative of an error between the normal activation pattern and the stimulation activation pattern converges.

25. The one or more non-transitory computer-readable storage media of claim 23, wherein the stimulation activation pattern is generated from the modeling of the physiological effect of the determined information.

26. The one or more non-transitory computer-readable storage media of claim 23, wherein the physiological effect comprises an effect of a stimulation signal generated based on the determined information on a cochlea.

27. The one or more non-transitory computer-readable storage media of claim 23, wherein the normal activation pattern and the stimulation activation pattern are each represented by neurogram indicative of one of an inner hair cell voltage and a spike in an auditory nerve fine structure.

28. The one or more non-transitory computer-readable storage media of claim 20 or 21, further comprising: extracting one or more features from the signal, wherein the machine learning model determines the information based on the extracted features.

29. The one or more non-transitory computer-readable storage media of claim 20 or 21, wherein the machine learning model determines the information based on an activation pattern of neurons in an auditory nerve from an audio signal.

30. The one or more non-transitory computer-readable storage media of claim 20 or 21, wherein the machine learning model determines the information based on a pattern of activation of neurons in an auditory nerve from an audio signal combined with noise.

31. The one or more non-transitory computer-readable storage media of claim 20 or 21, wherein the stimulation signal corresponds to focused multipolar stimulation.

32. The one or more non-transitory computer-readable storage media of any of claims 20 or 21, wherein the modeling of the physiological effect is specific to the recipient.

33. An implantable medical device system, the implantable medical device system comprising: a memory to store data; and one or more processors, wherein the one or more processors are configured to: receive a signal associated with a physiological function; determine, by a machine learning model, information for a stimulation signal to stimulate the physiological function based on the signal, wherein the machine learning model is trained based on modeling of a physiological effect from the stimulation; and control the stimulation of a recipient of the implantable medical device system based on the determined information.

34. The implantable medical device system of claim 33, wherein the machine learning model comprises a neural network.

35. The implantable medical device system of claim 33 or 34, wherein the physiological function comprises hearing.

36. The implantable medical device system of claim 33 or 34, wherein the machine learning model is trained to minimize a difference between a normal pattern of activation of neurons in an auditory nerve and a stimulation activation pattern generated based on the determined information.

37. The implantable medical device system of claim 36, wherein the machine learning model is trained until a cost function indicative of an error between the normal activation pattern and the stimulation activation pattern converges.

38. The implantable medical device system of claim 36, wherein the stimulation activation pattern is generated from the modeling of the physiological effect of the determined information.

39. The implantable medical device system of claim 36, wherein the physiological effect comprises an effect of a stimulation signal generated based on the determined information on a cochlea.

40. The implantable medical device system of claim 36, wherein the normal activation pattern and the stimulation activation pattern are each represented by neurogram indicative of one of an inner hair cell voltage and a spike in auditory nerve fine structure.

41. The implantable medical device system of claim 33 or 34, further comprising: extracting one or more features from the signal, wherein the machine learning model determines the information based on the extracted features.

42. The implantable medical device system of claim 33 or 34, wherein the machine learning model determines the information based on a pattern of activation of neurons in an auditory nerve from an audio signal.

43. The implantable medical device system of claim 33 or 34, wherein the machine learning model determines the information based on a pattern of activation of neurons in an auditory nerve from an audio signal combined with noise.

44. The implantable medical device system of claim 33 or 34, wherein the stimulation signal corresponds to focused multipolar stimulation.

45. The implantable medical device system of claim 33 or 34, wherein the modeling of the physiological effect is specific to the recipient.

46. A method comprising: determining, by a machine learning model of at least one processor, information for a stimulation signal to stimulate a physiological function based on a signal associated with the physiological function; modeling, via the at least one processor, a physiological effect from the stimulation signal; and updating, via the at least one processor, the machine learning model based on a difference between the modeled physiological effect and a reference physiological effect representative of normal physiological function.

47. The method of claim 46, wherein the machine learning model comprises a neural network.

48. The method of claim 46 or 47, wherein the physiological function comprises hearing.

49. The method of claim 46 or 47, wherein the machine learning model is trained to minimize a difference between a normal pattern of activation of neurons in an auditory nerve and a stimulation pattern of activation generated based on the determined information.

50. The method of claim 49, wherein the machine learning model is trained until a cost function indicative of an error between the normal pattern of activation and the stimulation pattern of activation converges.

51. The method of claim 49, wherein the stimulation pattern of activation is generated from the modeling of the physiological effect from the stimulation signal.

52. The method of claim 49, wherein the physiological effect comprises an effect of a stimulation signal based on the determined information on a cochlea.

53. The method of claim 49, wherein the normal pattern of activation and the stimulation pattern of activation are each represented by neurogram indicative of one of an inner hair cell voltage and a spike in auditory nerve fine structure.

54. The method of claim 46 or 47, wherein the machine learning model determines the information based on a pattern of activation of neurons in an auditory nerve from an audio signal.

55. The method of claim 46 or 47, wherein the machine learning model determines the information based on a pattern of activation of neurons in an auditory nerve from an audio signal combined with noise.

56. The method of claim 46 or 47, wherein the stimulation signal corresponds to focused multipolar stimulation.

57. The method of claim 46 or 47, wherein the modeling of the physiological effect is specific to a user.

Images