Evaluate responses to sensory events and perform processing actions based on them

Abstract

Examples disclosed herein are related to monitoring and processing sensory conditions that affect an individual. Sensors and intelligence integrated into a sensory prosthesis (e.g., an auditory prosthesis) can automatically obtain objective data about the ability of one or more senses of an individual in daily activities. Processing actions can be taken based on the objective data. Further disclosed herein are technologies related to reducing the collection of irrelevant sensory inputs and automatically transmitting relevant data to a caregiver device.

Description

[0001] This application was filed as a PCT international patent application on June 15, 2020, and claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 866,127 filed on June 25, 2019, the entire disclosure of which is incorporated herein by reference.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is related to, but does not claim the benefit of priority to, U.S. Provisional Patent Application No. 62 / 750,394, entitled “Passive Fitting Techniques”; U.S. Provisional Patent Application No. 62 / 703,373, entitled “Habilitation and / or rehabilitation methods and systems”; and U.S. Provisional Patent Application No. 62 / 862,181, entitled “Improving Musical Perception of a Recipient of an Auditory Device.” These related applications are hereby incorporated by reference in their entirety for any and all purposes. Background Art

[0004] Medical devices having one or more implantable components, generally referred to herein as implantable medical devices, have provided a wide range of therapeutic benefits to recipients in recent decades. In particular, partially or fully implantable medical devices, such as hearing prostheses (e.g., bone conduction devices, mechanical stimulators, cochlear implants, etc.), implantable pacemakers, defibrillators, functional electrical stimulation devices, and other implantable medical devices, have successfully performed life-saving and / or lifestyle-enhancing functions and / or recipient monitoring for several years.

[0005] In recent years, there has been an increase in the variety of implantable medical devices and the range of functions achieved thereby. For example, many implantable medical devices now typically include one or more instruments, devices, sensors, processors, controllers, or other functional mechanical or electrical components that are permanently or temporarily implanted into a recipient. These functional devices are typically used to diagnose, prevent, monitor, treat, or manage disease / injury or its symptoms, or to investigate, replace, or modify anatomical structures or physiological processes. Many of these functional devices use power and / or data received from external devices that are part of the implantable medical device or operate in conjunction with the implantable medical device. Summary of the invention

[0006] In an example, there is a hearing prosthesis device, including: an implantable or wearable housing; a stimulator, at least partially disposed within the housing, and configured to deliver energy to tissue of a receptor of the hearing prosthesis device to evoke an auditory perception in the receptor; a plurality of sensors; and a processor configured to perform operations. The operations include: receiving event sensor data from the plurality of sensors; detecting an occurrence of an auditory event near the hearing prosthesis device using the event sensor data; detecting a response of the receptor to the occurrence of the auditory event using response sensor data; characterizing the response of the receptor using the response sensor data; and taking a processing action related to the receptor based on the characterized response.

[0007] In an example, the plurality of sensors include a microphone. In an example, detecting an occurrence includes: receiving an audio signal from a microphone; determining whether the audio signal includes speech asking a question; and in response to determining that the audio signal includes speech asking a question, determining that an auditory event has occurred near the hearing prosthesis. In an example, detecting a response of a receptor includes detecting a speech response of the receptor. In an example, characterizing the response of a receptor includes: determining the length of time taken from being asked a question to the receptor starting to provide a speech response; and determining the length of the speech response provided by the receptor. In an example, detecting a response of a receptor includes determining a behavior of the receptor when being asked a question. In an example, determining the behavior of the receptor includes measuring an amount of stillness or a duration of stillness of the receptor when being asked a question. In an example, measuring an amount or duration of stillness of the receptor includes: measuring muscle movement of the receptor using an electromyogram; measuring muscle movement of the receptor using a camera; measuring muscle movement of the receptor using an accelerometer; or measuring muscle movement of the receptor using a gyroscope. In an example, detecting a speech response includes: detecting a vocalization of the receptor using an implantable microphone; detecting a vocalization of the receptor using an external microphone; detecting a change in a heart rate characteristic of an individual preparing to speak; detecting an inhalation characteristic of an individual preparing to speak; detecting a cortical cascade of the receptor in response to a question. In the example, the auditory event is: a vocal activity of singing, a vocal activity with sound, a vocal activity of singing, a vocal activity of imitation, a vocal activity of whispering, or a musical activity. In the example, the reaction is a vocal reaction, a motor reaction, or a physiological reaction. In the example, the operation also includes: classifying the auditory event as a relevant sensory event or an irrelevant sensory event. In the example, the response of the receptor is detected in response to the auditory event being classified as a relevant sensory event. In the example, the plurality of sensors includes at least one sensor selected from the group including the following items: a microphone, a body noise sensor, a motion sensor, an implanted electrode sensor. In the example, the plurality of sensors includes a motion sensor. In the example, the motion sensor is an accelerometer or a gyroscope. In the example, taking a processing action includes: modifying one or more settings of the auditory prosthesis; reporting the performance quality of the auditory prosthesis regarding the auditory event; suggesting a corrective action; or providing a measure of the ability of the receptor to participate in the auditory event. In the example, the auditory prosthesis also includes a memory storing instructions, which, when executed by one or more processors, causes the processor to perform an operation, thereby configuring the one or more processors to perform the operation. In an example, a stimulator includes: a stimulator unit disposed within a housing; a stimulator assembly disposed at least partially outside the housing; and an array of electrode contacts disposed on the stimulator assembly.

[0008] In an example, there is a non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to: receive event sensor data from a plurality of sensors proximate to a receptor of a sensory prosthesis; detect an occurrence of a sensory event proximate to the sensory prosthesis using the event sensor data; classify the sensory event as a relevant sensory event or an unrelated sensory event; in response to the sensory event being a relevant sensory event, detect a response of the receptor to the occurrence of the sensory event using response sensor data; characterize the response of the receptor using the response sensor data; and take a processing action related to the receptor based on the characterization.

[0009] In an example, a relevant sensory event is an event that is expected to elicit a characteristic response in a receptor. In an example, characterizing a response of a receptor includes determining whether sensor data indicates that the receptor is performing a characteristic response. In an example, classifying a sensory event as a relevant sensory event or an irrelevant sensory event includes analyzing a location specified by the sensor data. In an example, classifying a sensory event as a relevant sensory event or an irrelevant sensory event includes comparing a threshold to the sensor data; in response to the sensor data satisfying the threshold, classifying the sensory event as a relevant sensory event; and in response to the sensor data not satisfying the threshold, classifying the sensory event as an irrelevant sensory event. In an example, the instructions further cause the processor to: in response to the sensory event being an irrelevant sensory event, avoid detecting a response of the receptor to the occurrence of the sensory event. In an example, taking a processing action includes modifying a processing operation of the sensory prosthesis by changing one or more settings of the sensory prosthesis. In an example, characterizing a response includes marking a response as appropriate for the sensory event. In an example, the sensory prosthesis is a hearing prosthesis comprising a non-transitory computer-readable medium.

[0010] In an example, there is a system comprising: a non-transitory computer-readable medium, the non-transitory computer-readable medium comprising instructions, which when executed by one or more processors, causes the one or more processors to: selectively operate in a monitoring mode or a non-monitoring mode. When operating in the monitoring mode: the instructions cause the one or more processors to receive sensor data from a plurality of sensors; detect the occurrence of a sensory event near a sensory prosthesis; characterize the sensory event; use the sensor data to detect a response of a receptor to the occurrence of the sensory event; and store sensory event data about the sensory event and response data about the response of the receptor; and transition to the non-monitoring mode. In an example, in response to transitioning from the monitoring mode to the non-monitoring mode, an analysis based on the sensory event data is transmitted to a caregiver device.

[0011] In an example, selectively operating in a monitoring mode or a non-monitoring mode includes: operating in a non-monitoring mode in response to proximity between a sensory prosthesis and a caregiver device. In an example, selectively operating in a monitoring mode or a non-monitoring mode includes: operating in a non-monitoring mode based on a time of day. In an example, the instructions, when executed by one or more processors, cause the one or more processors to avoid detecting an occurrence of a sensory event near the sensory prosthesis when in the non-monitoring mode. In an example, the analysis includes a metric describing an appropriateness of a response to the sensory event. In an example, transmitting the analysis to the caregiver device includes: directly, automatically and wirelessly transmitting the analysis from the sensory prosthesis to the caregiver device. In an example, the system also includes a first sensory prosthesis, the first sensory prosthesis including a first sensory prosthesis sensor; and a non-transitory computer-readable medium. In an example, the system also includes an auxiliary device, the auxiliary device including an auxiliary device sensor. In an example, the instructions, when executed by the one or more processors, further cause the one or more processors to: aggregate sensor output data from the first sensory prosthesis sensor and the auxiliary device sensor to form aggregate data. In an example, detecting an occurrence of a sensory event near a receptor of the sensory prosthesis includes using the aggregate data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The same number represents the same element or the same type of element in all drawings.

[0013] Figure 1 A system is shown with which one or more of the techniques described herein may be implemented.

[0014] Depend on FIG. 2A to FIG. 2E FIG. 2 , as composed, illustrates a first example process for monitoring an individual's sensory abilities and taking processing actions based thereon.

[0015] Figure 3 A second example process for monitoring an individual's sensory abilities and taking processing actions based thereon is shown.

[0016] Figure 4 An example machine learning framework that can be used with the examples in this article is shown.

[0017] Figure 5 An example cochlear implant system that may benefit from use of the examples disclosed herein is shown.

[0018] Figure 6 Examples of transcutaneous bone conduction devices that may benefit from use of the examples disclosed herein are shown.

[0019] Figure 7 An example of a transcutaneous bone conduction device having a passive implantable component that may benefit from use of the examples disclosed herein is shown.

[0020] Figure 8 An example of a retinal prosthesis system that may benefit from use of the examples disclosed herein is shown.

[0021] Fig. 9 An example of a suitable computing system is shown upon which one or more of the disclosed examples may be implemented. DETAILED DESCRIPTION

[0022] To provide effective treatment interventions for individuals affected by a loss of sensory abilities (such as hearing or vision loss), it is useful to appropriately monitor the progression (or decline) of the individual's sensory abilities. Traditionally, individuals (or their caregivers) are given subjective verbal or written questionnaires regarding the individual's sensory abilities relative to those of people with typical sensory abilities. For example, where the relevant sense is hearing, a functional hearing index may be used to determine the relevant abilities. The functional hearing index is a measure of an individual's hearing performance in an everyday setting.

[0023] However, analysis of an individual's sensory abilities has traditionally been subjective and intermittent, which can limit the effectiveness of monitoring regimes and potentially produce low-quality data. Improvements in data collection about an individual's sensory functioning lead to improvements in the treatment of an individual's sensory condition, as treatment interventions are based on this data.

[0024] Examples disclosed herein relate to improving the monitoring and treatment of sensory conditions affecting individuals, including individuals who are already recipients of one or more sensory prostheses (e.g., auditory or visual prostheses). Example techniques disclosed herein include using sensors and intelligence integrated into the sensory prosthesis of the recipient to automatically obtain objective data about the performance of one or more senses of the recipient during daily activities. Further example techniques include techniques for reducing the collection of irrelevant sensory input (e.g., thereby improving the ability of the sensory prosthesis or another device to provide high-quality output on which processing can be based) and techniques for automatically transmitting relevant data to a caregiver device.

[0025] In an example, a sensory event occurs in the vicinity of a receptor of a sensory prosthesis. Sensors of the sensory prosthesis obtain event data that can be used to detect the occurrence of the sensory event. Next, response data from the sensors is obtained and processed to characterize the response of the receptor to the sensory event. The resulting characterization is used to take a processing action on the receptor. For example, a setting of the sensory prosthesis can be automatically modified to affect the processing being performed by the receptor.

[0026] As a specific example, the technology described herein can be applied to any of a variety of sensory events. In particular, with respect to auditory events, the event can be a conversation in a noisy restaurant, and the characteristics can indicate that the receiver of the hearing prosthesis has difficulty hearing the other party's voice. The settings of the hearing prosthesis can be modified to try to improve the performance of the hearing prosthesis in a noisy environment. As another example, the auditory event can be a positive or inspiring voice directed to the receiver. Such a voice can have distinctive features (e.g., a characteristic frequency or pitch) that can be detected. In response to such a voice, the hearing prosthesis can determine whether the receptor is appropriately affected by the voice (e.g., the signal processing module can evaluate whether the receptor responds in a pleasant manner). Similar evaluations can also be used for angry or annoying voices directed to the receptor. The settings of the hearing prosthesis can be modified to improve the reproduction of specific characteristic frequencies or tones of specific voices.

[0027] As can be seen from the auditory event example above, how the receptors respond to vocalizations is a useful indicator of whether the current processing methods are effective and should be changed. Therefore, the sensors of the hearing prosthesis can be used to detect and characterize auditory events, which include speech in categories that are expected to elicit characteristic responses (e.g., answering a question, joining in laughter, crying in response to shouting). Classification can then be used to assess whether the receptors provide an appropriate response to the auditory event. More generally, how the receptors respond to sensory events is a useful indicator of whether the processing methods are effective and should be changed.

[0028] With respect to classification, it can be determined whether there are corresponding differences in the responses provided by the recipient to auditory events belonging to a particular category or subcategory of auditory events (e.g., different types of questions determined by different pitches, frequencies, or pitch characteristics). Similarly, in some examples, the hearing prosthesis evaluates whether there are corresponding differences in the responses provided by the recipient in response to auditory events delivered with other sensory stimuli (e.g., one's own voice, a voice stream with video, or a voice stream without video). Such classification can be used to identify specific abilities and deficiencies in the ability of the hearing prosthesis to elicit auditory perception in the recipient.

[0029] The hearing prosthesis can detect speech using a microphone (e.g., an implanted microphone or an external microphone of a sound processor). The detected speech can be analyzed to determine whether the detected speech is a type of speech that warrants a receptor response. For example, an integrated sound classifier or other intelligence can analyze an auditory event to determine whether the auditory event can be characterized as a question that warrants a receptor response. For example, in English, interrogative sentences can often be characterized by a rising intonation, which can be detected by an integrated sound classifier or the intelligence of the hearing prosthesis. Interrogative sentences can also have other features that can be detected in addition to or in place of intonation. In an example, an accessory device (e.g., a mobile device or server such as a phone or tablet) can be used to classify auditory events. Classification or processing actions based on the classification can be provided to the hearing prosthesis. Additionally or alternatively, the features or processing actions can be presented to the receptor or caregiver.

[0030] After or during a sensory event, speech or physiological signals from the receptor can be detected, classified, and recorded. Physiological responses can include changes in cardiac and respiratory activity (e.g., changes in rate, changes in rate, intervals between events, or occurrence of arrhythmias) or changes in neural activity (e.g., measured by external or implanted sensors such as montages of electrodes, microphones, or accelerometers). The sensor used to detect such speech or physiological responses can be a component of the sensory prosthesis (e.g., an implanted portion thereof or an external portion thereof, such as a sound processor) or a connected device (e.g., a phone or wearable). Movement and vocal responses can also be detected, classified, and recorded.

[0031] With regard to sounds and other audible responses, a signal processing module (e.g., a hardware module or a software module) can process audio signals captured via one or more sensors (e.g., external or implanted microphones) and can be used to assess whether the recipient has appropriately responded to a sensory event (e.g., appropriately responded to a question). For example, an audio signal can capture the recipient's vocalization in response to a sensory event such as a voice response. The module can measure the length of the voice response (or other vocalization) provided by the recipient and assess whether the length corresponds to the length of the voice given by a typical response (e.g., a response to a question). In addition, the module can determine how long it took the recipient to vocalize a sensory event directed to the recipient. The data can also be related to calculating the sensory ability of the recipient (e.g., the functional hearing ability of the recipient). The module can also be used to assess the characteristics of the response (e.g., intonation, frequency, pitch, etc.) to determine whether the response has the characteristics of the expected response.

[0032] With respect to motion responses, the signal processing module can process motion data obtained from one or more motion sensors (e.g., accelerometers located in the sensory prosthesis itself or a companion device such as a smartphone or smartwatch). The motion data can be used, for example, to assess whether the recipient is responding appropriately to the question. For example, the module can measure the amount and duration of time that the recipient is stationary while listening to the question, and the amount and duration of time that the recipient is moving while responding to the question.

[0033] In addition, sensors can be used to determine the ability of the recipient to appropriately participate in mirroring behavior. For example, mirroring can include the recipient imitating playful words or actions directed at the recipient by the caregiver, or the recipient's answer contains words similar to the corresponding question. The signal processing module can be used to detect whether the recipient appropriately participates in mirroring using audio captured via a sensor (e.g., an external or implanted microphone). In particular, the system can detect (e.g., over an extended duration) whether any tone, frequency, other characteristics, or a combination thereof, is absent in the recipient's mimicry.

[0034] In addition to or in lieu of using the sensors described herein, the signal processing module can process electrophysiological signals obtained via one or more sensors (e.g., clip-on or implanted or external electrodes) and can be used to assess whether the receptor responds appropriately to auditory events. For example, the module can measure the duration of stillness and the duration of response of the receptor while listening to a question via electromyography of the facial muscles. The module can measure the cortical cascades during central integration and processing of heard questions and during preparation of spoken responses.

[0035] In addition to or instead of using the sensors described herein, the signal processing module can process audio-physiological signals captured via sensors such as implanted microphones and can be used to assess whether the subject is responding appropriately to the problem. For example, the module can measure changes in heart rate or respiratory intake as the subject prepares to respond to a sensory event.

[0036] In some examples, multiple sensory prostheses can collaborate to monitor the sensory capabilities of a recipient. For example, bilateral hearing prostheses can be used in tandem to generate data to determine the hearing capabilities of a recipient. When combined, the bilateral prostheses can pool their sensor data to feed a single signal processing module. In some examples, the bilateral prostheses can independently evaluate sensory events and responses before later combining the evaluation into a single receptor recording. In some examples, the prostheses can divide tasks (e.g., one prosthesis analyzes sensory events while another prosthesis monitors the response of the receptor). In addition, in response to changes in hardware configuration (e.g., bilateral, unilateral left, or unilateral right prosthesis configuration), it can be determined whether there are corresponding differences in the responses provided by the receptor. In particular, it can be determined whether the sensory function of the receptor is better with a unilateral prosthesis or a bilateral prosthesis.

[0037] Advantageously, the method described herein can advantageously protect the privacy of the recipient. When a sensory event or response includes speech, the above method can be implemented without analyzing the actual content of the speech (e.g., by measuring the intonation of the speech without performing speech-to-text analysis on the speech and analyzing the resulting text). For example, in some implementations, data about voice quality (e.g., intonation) is processed, and the processing may not involve the semantics of the speech. Of course, in other examples, analyzing the semantic content of the speech may be beneficial in determining whether the recipient has made an appropriate response. In some examples, the analysis of the semantic content can be bound to a user-modifiable option so that the user can choose whether to analyze the semantic content.

[0038] The above techniques can be used to modify the processing of sensory conditions affecting a receptor. For example, the techniques can be used to automatically or manually (e.g., with the help of a clinician) adjust the mapping of a hearing prosthesis, or provide feedback about past adjustments.

[0039] Example System

[0040] Figure 1 A system 100 is shown with which one or more of the techniques described herein may be implemented. As shown, the system 100 includes a sensory prosthesis 110 for a subject (e.g., implanted in or worn by the subject), a computing device 150, and an auxiliary device 162. The computing device 150 is connected to a server 170 via a network 102. The network 102 is a computer network, such as the Internet, that facilitates electronic communication of data between computing devices connected to the network 102.

[0041] Figure 1 Further shown is a sensory event 10 (eg, an auditory or visual event) occurring in the vicinity of the receptor. The receptor provides a response 20.

[0042] Sensory events 10 are events detectable by one or more human senses. In many examples, sensory events 10 are events that are expected (e.g., by a clinician) to be perceived by one or more normally functioning and unaided human senses (e.g., a typical person is able to perceive sound without amplification equipment). Although sensory events 10 can generally be perceived by different human senses (e.g., a person speaking can generally be seen and heard speaking), sensory events 10 can be limited to specific senses that are being enhanced by the receptor's sensory prosthesis. Sensory events 10 can be events that are expected to elicit a characteristic response in the receptor. In some examples, certain events where a characteristic response from the receptor is not expected are excluded from consideration as sensory events 10. For example, white noise (e.g., fan noise or rain noise) can be a sound that the receptor hears but does not expect a specific response to.

[0043] Sensory event 10 may be an auditory event. An auditory event may be an occurrence that typically causes auditory perception in a human individual. For example, an auditory event may have a volume greater than the threshold of typical human hearing and a frequency within the range of typical human hearing. In another example, sensory event 10 may be a visual event. A visual event may be an occurrence that typically causes visual perception in a human individual. For example, a visual event may be within a spectrum and intensity visible to a typical human. In a further example, a sensory event may be a tactile sensory event, an olfactory sensory event, or a taste sensory event. Sensory events are not necessarily limited to the five traditional senses, and may include, for example, balance sensation, thermal sensation, proprioception, and nociception, among others.

[0044] In the case where the sensory event 10 is an auditory event, the auditory event may include one or more of the following auditory activities: vocal activity of singing, vocal activity of voice, vocal activity of singing, vocal activity of imitation, vocal activity of whispering, and musical activity. In an example, the sensory event 10 is an auditory event determined to be likely to cause a startle response in the recipient (e.g., by having a specific loudness above the background level). The startle response of the recipient is an example of response 20.

[0045] Response 20 is the behavior of the receptor that is temporally close to the occurrence of sensory event 10. For example, temporal proximity may be the amount of time that the receptor's response is expected to have occurred (if any response actually occurred). The amount of time may vary depending on the type of sensory event (e.g., one may be expected to respond more quickly to a shocking event than to a question). While in many examples, response 20 is the receptor's reaction to sensory event 10, response 20 may include actions of the receptor that are not triggered by sensory event 10. In other words, the receptor may not be able to respond to sensory event 10, and even if it is unable to respond, it may be considered a response 20 to sensory event 10. Detecting such non-responsive response 20 may be used to modify processing actions. Non-responsiveness when a response is appropriate may indicate a gap between the expected processing result and the actual processing result.

[0046] Response 20 may take any of a variety of forms. For example, response 20 may include one or more of the following responses: a vocal response (e.g., the receptor's production or failure to produce a vocalization in response to sensory event 10), a motor response (e.g., the degree to which the receptor moves or does not move in response to sensory event 10), or a physiological response (e.g., a typically involuntary physiological response that a receptor may make in response to a sensory event, such as increased blood flow, heart rate, or breathing rate).

[0047] Motor reactions can include large motor reactions, such as jumping, frightening, or dancing. Motor reactions can be directional, such as the receptor looking or pointing in a specific direction. Motor reactions can also include subtle reactions, such as making faces (e.g., smiling). Motor reactions can also be a lack of movement or a change in movement (e.g., a relative decrease or increase in the amount of movement of the receptor), which can indicate that the receptor is listening to a person or otherwise paying attention to a sensory event. Motor reactions can be detected based on motion data (e.g., obtained from accelerometers and gyroscopes) and electrode data (e.g., obtained from electrodes monitoring muscle movement or brain waves) and other sources. Predetermined thresholds can be used so that when specific data meets one or more thresholds, it can be determined that a movement reaction (or a specific type of movement reaction) has occurred. Additionally or alternatively, machine learning can be used. For example, a machine learning framework can be trained to determine the occurrence of a motor reaction (or a specific type of motor reaction) based on input data.

[0048] The vocalization reaction may include the recipient vocalizing in response. The vocalization may include, for example, speaking, singing, laughing, or imitation (e.g., imitation of sounds, people, animals, or objects), etc. The vocalization may include babbling or vocalization (e.g., making Ling Six sounds or forming syllables without necessarily forming words). The vocalization may include the repetition of sounds, words, or sentences. The vocalization may include answering questions or having a conversation. The vocalization reaction may be detected primarily based on audio data obtained from an implantable or external microphone. Detection may include analyzing the characteristics of the sound (e.g., pitch, frequency, intensity, etc.) to determine whether and how the receptor responds. The audio may be processed to facilitate the detection of the sound. For example, speech to text processing may be performed on the speech, and the resulting text may be analyzed to determine whether the receptor correctly responds to the sensory event 10 (e.g., by using natural language processing). Mimicry may be detected by comparing patterns in the characteristics of the response 20 with the characteristics of the sensory event 10.

[0049] The response 20 may include a recognition response. A recognition response may be a response that demonstrates that the receptor understands or recognizes the sensory event 10. A recognition response may be an indication that the receptor understands or is able to tell the difference between different sensory events. For example, a recognition response may be discerning the difference between speaking or singing, understanding a word or phrase, recognizing a person by their voice, and being able to recognize emotion in a person's voice. Recognition responses may be detected or inferred in a variety of ways. In an example, a recognition response is detected by analyzing the receptor's brain waves (e.g., measuring cortical cascades) for patterns that indicate recognition. In other examples, recognition responses may be inferred based on movement (e.g., facial movement or larger muscle movement) or vocal responses.

[0050] In some examples, the above reactions can form complex reactions, such as reactions that include both sound and motor reactions. In an example, a complex reaction can include the recipient following instructions and performing one or more actions in response. In some examples, the response 20 (particularly complex responses) can be detected based on responses from people other than the recipient (e.g., a caregiver giving instructions). For example, words of encouragement or disappointment can be used to infer whether the recipient's response 20 is appropriate or inappropriate. Such speech content can be determined by natural language processing (e.g., analyzing the content) or by emotion detection.

[0051] Example System - Sensory Prosthesis

[0052] The sensory prosthesis 110 is a device associated with one or more senses of a subject. For example, the sensory prosthesis 110 can be a prosthesis associated with one or more of the five traditional senses (vision, hearing, touch, taste, and smell) and / or one or more of other senses. For ease of understanding, in many of the disclosed examples, the sensory prosthesis 110 is a hearing prosthesis medical device configured to treat hearing loss of the subject. Although many examples specific to hearing prostheses can be applicable to other types of sensory prostheses. Where the sensory prosthesis 110 is an auditory prosthesis, the sensory prosthesis 110 may take a variety of forms, including a cochlear implant, an electroacoustic device, a percutaneous bone conduction device, a passive transcutaneous bone conduction device, an active transcutaneous bone conduction device, a middle ear device, a fully implantable hearing device, a mostly implantable hearing device, an auditory brainstem implant device, a hearing aid, a tooth-anchored hearing device, a personal sound amplification product, other hearing prostheses, and combinations of the foregoing (e.g., a binaural system including a prosthesis for a first ear of the recipient and a prosthesis of the same or different type for a second ear). In an example, the sensory prosthesis 110 may be or include features associated with a vestibular implant and a bionic eye. Figure 5 (cochlear implant shown), Figure 6 (showing a percutaneous bone conduction device), Figure 7 (transcutaneous bone conduction device shown) and Figure 8 An example implementation of the sensory prosthesis 110 is described in more detail in (retinal prosthesis shown). The technology disclosed herein may also be relevant to applications with devices and systems for, for example, sleep apnea management, tinnitus management, and epilepsy management. The technology disclosed herein may be used with sensory devices such as consumer hearing devices (e.g., hearing aids or personal sound amplification products).

[0053] As shown, the sensory prosthesis 110 includes a housing 112, a stimulator 120, one or more sensors 130, one or more processors 140, and a memory 142. In many examples, the sensory prosthesis 110 may include Figure 1 More or fewer components may be used than shown.

[0054] The housing 112 may include a wearable housing (e.g., wearable on a subject's head or a subject's wrist via a strap, a band, a magnetic connection, or another fastening technique). In some examples, the sensory prosthesis 110 may include multiple cooperating components disposed in separate housings. For example, the sensory prosthesis 110 may include an external component (e.g., having components for receiving and processing sensory data) that is configured to communicate with an implantable component (e.g., having components for delivering stimuli to induce sensory rules in a subject). In the case where the housing 112 is an implantable housing, the housing 112 may be made of a biocompatible material and hermetically sealed inside to prevent intrusion of bodily fluids.

[0055] Stimulator 120 is one or more parts of the sensory prosthesis 110 that provides stimulation to the receptor. For example, stimulator 120 can receive a stimulation control signal from another part of the sensory prosthesis, and produce stimulation based on it. Stimulator can apply stimulation to the receptor to cause sensory perception. According to the type of sensory prosthesis, stimulation can take any one of a variety of forms. In many examples, stimulation includes electrical stimulation or mechanical stimulation. Stimulator 120 can stimulate the nerve cells of the receptor (for example, to bypass the normal sensory phenomenon converted into neural activity to cause the loss or defective cell of sensory perception in the receptor) in a way that makes the receptor perceive one or more components of the sensory input data. As shown in the figure, stimulator 120 may include stimulator unit 122 and stimulator assembly 124.

[0056] The stimulator unit 122 may be a portion of the stimulator 120 that generates stimulation. For example, where the stimulator 120 is an electrical stimulator, the stimulator unit 122 may generate electrical stimulation signals. Where the stimulator 120 is a mechanical stimulator, the stimulator unit 122 may be or include an actuator configured to generate vibrations.

[0057] The stimulator assembly 124 may be the portion of the stimulator 120 through which the stimulation is applied to the recipient. For example, where the stimulator 120 is an electrical stimulator for a cochlear implant, the stimulator assembly 124 may be an elongated lead having an array of electrode contacts disposed thereon for delivering the electrical stimulation to the cochlea of ​​the recipient. Where the stimulator 120 is a mechanical stimulator, the stimulator assembly 124 may be a plate, a column, or another component to conduct the vibrations from the stimulator unit 122 to a desired portion of the recipient's anatomy.

[0058] Sensors 130 are one or more components of the sensory prosthesis 110 that generate signals based on sensed events (e.g., data about the environment surrounding the sensory prosthesis 110, the sensory prosthesis 110 itself, or a receptor). In many examples, the sensors 130 are configured to obtain data for generating stimulation via the stimulator 120. In the example shown, the sensors 130 may include one or more of: a microphone 132, a motion sensor 136, and an electrode sensor 138.

[0059] The one or more microphones 132 may include one or more microphones implanted in the recipient or a microphone external to the recipient. The microphone 132 may be a transducer that converts acoustic energy into an electrical signal. One or more of the microphones 132 may be configured to receive sounds generated outside the recipient. One or more of the microphones may include a body noise sensor 134 or be configured as a body noise sensor 134. The body noise sensor 134 is a sensor 130 configured to sense body noise generated by the recipient. The body noise that may be measured by the body noise sensor 134 may include, for example, sounds such as breathing sounds, blood flow sounds, heartbeat sounds, or stomach sounds.

[0060] The one or more motion sensors 136 may be transducers that convert motion into electrical signals. The motion sensors 136 may include, for example, accelerometers and gyroscope sensors.

[0061] The one or more electrode sensors 138 may be one or more electrodes configured to detect electrical signals. In some examples, the electrode sensors 138 may be electrodes of the stimulator assembly 124 that are configured to not only deliver stimulation but also detect electrical signals. The electrode sensors 138 may include internal or external electrode sensors. In an example, the electrode sensors 138 are wearable electrodes, such as via a headband.

[0062] The sensor 130 may include one or more other sensors, such as one or more position sensors, a telecoil, a camera, a pupillometer, a biosensor (e.g., a heart rate or blood pressure sensor), an otoacoustic emission sensor (e.g., configured to provide an otoacoustic emission signal), an EEG (electroencephalogram) sensor (e.g., configured to provide an EEG signal), one or more light sensors configured to provide a signal related to light levels. The sensor 130 may include components disposed within the housing of the sensory prosthesis 110 and a device electrically coupled to the sensory prosthesis 110 (e.g., via a wired or wireless connection).

[0063] In an example, the sensors 130 include one or more remote devices connected to the sensory prosthesis 110 via an FM (frequency modulation) connection, such as a remote microphone (e.g., Cochlear True Wireless Microphone 2+), a television audio streaming device or a phone clipping device, and other devices with FM transmission capabilities. The sensors 130 may also include sensors that obtain data about the use of the sensory prosthesis 110, such as software sensors operating on the sensory prosthesis 110 that track data such as when the sensory prosthesis 110 is worn by a subject, when the sensory prosthesis 110 (e.g., an external portion thereof) is removed from the subject, when one or more sensory prosthesis settings are modified, the current scene mode in which the sensory prosthesis 110 is operating (e.g., as determined by a scene classifier), and how long the sensory prosthesis 110 is operated using specific settings.

[0064] In an example, the sensor 130 may include a scene classifier. A scene classifier is software that obtains data about the environment surrounding the sensory prosthesis (e.g., from one or more of the sensors 130) and determines a classification of the environment. The classification may be used to determine settings that are appropriate for the environment. For example, where the sensory prosthesis 110 is an auditory prosthesis, the scene classifier may obtain data about the acoustic environment surrounding the auditory prosthesis and classify the acoustic environment into one or more of the following possible classifications: speech, noise, and music, among other classifications. The sensory prosthesis 110 may then use the classification to automatically change the sensory prosthesis settings to adapt to the environment. For example, in response to the scene classifier determining that the acoustic environment surrounding the auditory prosthesis is windy, a wind noise scene may be selected that modifies the settings of the auditory prosthesis to reduce wind noise. In another example, the scene classifier may determine that there is music nearby and automatically modify the auditory prosthesis settings to improve music reproduction. An example scene classifier is described in US 2017 / 0359659, entitled “Advanced Scene Classification for Prosthesis,” filed on June 9, 2016, which is incorporated herein by reference in its entirety for any and all purposes.

[0065] The sensor 130 may generate sensor data. Sensor data is data generated by a sensor of the sensor 130. Depending on the configuration of the sensor 130 that generates the sensor data, the sensor data may take any of a variety of different forms. Furthermore, the form and characteristics of the sensor data may change as the sensor data is used and moved throughout the system 100. For example, the sensor data may begin as a real-time analog signal that is converted within the sensor 130 into a real-time digital signal that is then transmitted to an application in real-time as data packets to be sent in batches (e.g., non-real-time) to the server 170. Furthermore, the sensor data may be processed as the sensor data is used and moved throughout the system 100. For example, the sensor data may be converted into a standardized format and have associated metadata (e.g., timestamps, sensor identifiers, etc.) attached.

[0066] The one or more processors 140 are one or more hardware or software processing units (eg, central processing units) that can obtain and execute instructions. The one or more processors 140 can communicate with the other components of the sensory prosthesis 110 and control their performance.

[0067] The memory 142 is one or more software or hardware based computer readable storage media operable to store information accessible by the one or more processors 140. For more details on the memory, see Fig. 9 As shown, memory 142 may store instructions 144 and sensory prosthesis settings 146 .

[0068] Instructions 144 are processor-executable program instructions that, when executed by one or more processors 140, cause one or more processors 140 to perform actions or operations, such as those described in the procedures herein. Instructions 144 may configure one or more processors 140 to perform operations.

[0069] The sensory prosthesis settings 146 are one or more parameters having values ​​that affect how the sensory prosthesis 110 operates. For example, the sensory prosthesis 110 receives sensory input from the environment (e.g., using a microphone of the sensory prosthesis 110 to obtain audio input), converts the sensory input into stimulation signals, and uses the stimulation signals to generate stimulation (e.g., vibration or electrical stimulation) to induce sensory perception in a receptor.

[0070] Sensory prosthesis settings 146 may include a map with minimum and maximum stimulation levels for stimulation channels. Sensory prosthesis 110 then uses the map to control the amount of stimulation to be provided. For example, where sensory prosthesis 110 is a cochlear implant, the map affects which electrodes of the cochlear implant are stimulated and in what amounts based on the sound input received. In some examples, sensory prosthesis settings 146 include two or more predefined groupings of settings that can be selected by the recipient. One of the two or more predefined groupings of settings may be a default setting.

[0071] The sensory prosthesis settings 146 may also include sensory processing settings that modify the sensory input before it is converted into a stimulation signal. In the case of a hearing prosthesis, for example, such settings may include specific audio equalizer settings that may increase or decrease the intensity of sounds of various frequencies. In an example, the sensory prosthesis settings 146 may include a minimum threshold for a received sensory input to cause stimulation, a maximum threshold for preventing stimulation from being above a level that would cause discomfort, a gain parameter, an intensity parameter (e.g., loudness), and a compression parameter. The sensory prosthesis settings 146 may include settings that affect the dynamic range of stimulation produced by the sensory prosthesis 110. As described above, many of the sensory prosthesis settings 146 affect the physical operation of the sensory prosthesis 110, such as how the sensory prosthesis 110 provides stimulation to a receptor in response to sound input received from the environment. Thus, modifying the sensory prosthesis settings 146 may modify the processing provided by the sensory prosthesis 110. Examples of settings, setting modifications, and preprocessing for hearing prostheses are described in U.S. Patents 9,473,852 and 9,338,567, which are incorporated herein by reference for any and all purposes.

[0072] Example System - Computing Device

[0073] Computing device 150 is a computing device associated with a recipient of sensory prosthesis 110. In many examples, computing device 150 is a phone, tablet, smart watch, or heart rate monitor, but may take other forms. Although primarily described in the context of a recipient, computing device 150 may be a computing device owned or primarily used by a parent or caregiver of the recipient. In the example shown, computing device 150 includes sensory prosthesis application 152. Computing device 150 may communicate with server 170, such as via sensory prosthesis application 152 communicating over network 102.

[0074] In an example, the computing device 150 includes a sensory prosthesis application 152 that operates on the computing device 150 and cooperates with the sensory prosthesis 110. The sensory prosthesis application 152 is a computer program stored as computer-executable instructions in a memory of the computing device 150 that, when executed, performs one or more tasks associated with the sensory prosthesis 110. For example, the sensory prosthesis application 152 may control the sensory prosthesis 110 (e.g., by modifying the sensory prosthesis settings 146 automatically or based on input received from a subject at the computing device 150), monitor the use of the sensory prosthesis 110, and obtain data from the sensory prosthesis 110. The computing device 150 may connect to the sensory prosthesis 110 using, for example, a wireless radio frequency communication protocol (e.g., Bluetooth). The sensory prosthesis application 152 may transmit or receive data from the sensory prosthesis 110 over such a connection. The sensory prosthesis application 152 may be used to stream audio to the sensory prosthesis 110, such as from a microphone of the computing device 150 or from an application (e.g., a video or audio application) running on the computing device 150. In other examples, another application running on the computing device 150 may stream the sensory data to the sensory prosthesis 110. In an example, the sensory prosthesis application 152 acts as a data source by obtaining data about the sensory prosthesis 110 (eg, the sensor 130).

[0075] In some examples, one or more of sensors 130 may be a component of computing device 150. For example, computing device 150 may include a hardware sensor or a software sensor, such as software that obtains data from one or more data streams (e.g., audio or video streamed from computing device 150 to sensory prosthesis 110).

[0076] Example System - Auxiliary Equipment

[0077] The auxiliary device 160 is a device separate from the sensory prosthesis 110 that can provide sensor data for performing the processes and operations described herein. In an example, the auxiliary device 160 is an additional sensory prosthesis 110 from which data can be obtained. In other examples, the auxiliary device can be a mobile phone, a tablet, a smart watch, a heart rate monitor, a wearable EEG, a smart ring, or other device with one or more sensors. The sensor can be as described above with respect to the sensor 130 of the sensory prosthesis 110. In some examples, the auxiliary device 160 can obtain data from the auxiliary device sensor and transmit the data to one or more of the other devices or components of the system 100 for processing.

[0078] Example System - Server

[0079] The server 170 is a server computing device that is remote from the sensory prosthesis 110 and the computing device 150. The server 170 is communicatively coupled to the computing device 150 via the network 102. In many examples, the server 170 is communicatively coupled to the sensory prosthesis 110 indirectly through the computing device 150 (e.g., via the sensory prosthesis application 152). In some examples, the server 170 is directly communicatively coupled to the sensory prosthesis 110 (e.g., via a wireless telecommunications data connection of the sensory prosthesis 110). In certain examples, the sensory prosthesis 110 and the computing device 150 may be considered client devices of the server 170. In some examples, the functionality provided by the server 170 or its components may be provided by or located on a device local to the recipient (e.g., the computing device 150 or the sensory prosthesis 110). The sensory prosthesis application 152 may be a client application configured to interact with the server 170. The server 170 may include a processing unit and a memory that is configured to communicate with the sensory prosthesis 110. Fig. 9 Server 170 may also include instructions executable to perform one or more operations described herein.

[0080] Example Process

[0081] Figure 2 and Figure 3 An example process for monitoring an individual's sensory capabilities and taking processing actions based thereon is shown. One or more aspects of the process (e.g., one or more operations thereof) may be implemented as one or more instructions stored as part of instructions 144. For ease of understanding, the process is written with sensory prosthesis 110 performing the operations. However, the process may be modified to be performed, at least in part, by one or more other devices, such as computing device 150, auxiliary device 162, server 170, other devices, or combinations thereof. One or more of the devices may have instructions that, when executed, cause one or more of the operations to be performed.

[0082] Example Process - First Example Process

[0083] Depend on FIG. 2A to FIG. 2E FIG2 , which is composed of FIG2 , shows an example process 200 . In the example, process 200 begins with operation 210 .

[0084] Operation 210 includes receiving event sensor data 212 from one or more of the sensors 130. The event sensor data 212 may be received in any of a variety of ways. The event sensor data 212 may be received directly from the sensors 130 themselves or via an intermediary device. For example, at least some of the event sensor data 212 may be transmitted to the sensory prosthesis 110 via the auxiliary device 162 or the computing device 150 for processing. In some examples, the event sensor data 212 is pushed to the sensory prosthesis 110. In other examples, the sensory prosthesis 110 transmits a request to the sensor 130 or a device associated therewith to obtain the event sensor data 212.

[0085] In some examples, the event sensor data 212 is stored prior to being processed by the sensory prosthesis 110. In some examples, the sensory prosthesis 110 periodically obtains readings from the sensor 130. The sensory prosthesis can collect event sensor data 212 at various frequencies, and the amount of event sensor data 212 can also vary. For example, in some examples, obtaining and transmitting the event sensor data 212 occurs without significant delay (e.g., in real time). In other examples, a device including the sensor 130 can obtain the event sensor data 212 and store it in batches of data, and transmit the event sensor data 212 to the sensory prosthesis 110 less frequently.

[0086] In an example, operation 210 includes receiving a motion sensor signal from one or more motion sensors 136. In an example, receiving the motion sensor signal may include receiving an accelerometer signal from an accelerometer of the sensory prosthesis 110, receiving a gyroscope signal from a gyroscope of the sensory prosthesis 110, receiving an accelerometer signal from an accelerometer of the auxiliary device 162, or receiving a gyroscope signal from a gyroscope of the auxiliary device 162. In an example, operation 210 includes receiving a body noise sensor signal from a body noise sensor 134 (e.g., an implanted microphone 132). In an example, operation 210 includes receiving an implanted or external electrode sensor signal from an implanted or external electrode sensor 138. In an example, operation 210 includes receiving an external microphone signal from an external microphone 132. After operation 210, the process may move to operation 214.

[0087] Operation 214 includes detecting the occurrence of a sensory event 10 proximate to the sensory prosthesis 110 using the event sensor data 212. Detecting the occurrence of the sensory event 10 may include comparing the event sensor data 212 to one or more templates associated with determining the occurrence of the sensory event 10. For example, the templates may include one or more thresholds associated with the event sensor data 212 (e.g., audio data, motion data, or another type of data) that indicate the occurrence of the sensory event 10. In another example, the templates include one or more machine learning frameworks configured to receive at least some of the event sensor data 212 as input and provide as output a prediction of whether the sensory event 10 occurred.

[0088] In an example, the occurrence of a sensory event 10 may be determined based on a determination by a scene classifier of the sensory prosthesis 110 in response to the sensory prosthesis 110 operating in a particular sensory processing mode. In an example where the sensory event 10 is a speech event (e.g., another person speaking to the recipient), the occurrence of the sensory event 10 may be determined in response to the scene classifier operating in the speech processing mode or switching to operating in the speech processing mode. In an example, the occurrence of the sensory event 10 is determined based on a user input. For example, the recipient (or a caregiver or another person) may provide a user input (e.g., directly at the sensory prosthesis 110 or at another device) indicating that the sensory event 10 is occurring or has recently occurred. In some examples, the user input may further specify a type of sensory event (e.g., speech, music, etc.).

[0089] As another example, operation 210 may include determining the occurrence of a sensory event 10 proximate to the subject based on location data of the event sensor data 212. The location data may be used to determine whether the subject is in a location where a particular kind of sensory event 10 is likely to occur. For example, location data indicating that the subject is in a movie theater may increase the likelihood that the subject is experiencing a sensory event that typically occurs during a movie. As another example, location data indicating that the subject is in a school may increase the likelihood that the subject is experiencing a typical classroom sensory event (e.g., a lecture sensory event).

[0090] In an example, the location data may include satellite-based location data (e.g., GPS-based location data) generated by the computing device 150. In another example, the location data is based on nearby wireless broadcasts, such as a WI-FI SSID (Service Set Identifier). Such wireless broadcasts are useful for determining the current location and the type of current location. For example, when the sensory prosthesis 110 is operated in a school, the computing device 150 may detect a WI-FI SSID called "Classroom WI-FI", which may be used as an indication that a school sensory event is occurring or may occur. In another example, the location data may be determined based on the recipient manually or automatically checking in at a location (e.g., using the computing device 150).

[0091] In some examples, detecting the occurrence of sensory event 10 may include determining whether the audio signal includes speech that may require a response. For example, the speech may be the name of the said recipient, a command, part of a conversation, or a question. Detecting whether the speech requires a response may include determining a characteristic of the speech (e.g., frequency, amplitude modulation, and spectral spread of the audio signal) and comparing the characteristic to a threshold. The analysis may further include analyzing the speech using machine learning. Examples related to speech that may require a response as a question are described in more detail in operations 215 to 217.

[0092] In some examples, operation 210 or 214 includes operation 215. Operation 215 includes receiving an audio signal from microphone 132 (e.g., in some examples, sensor 130 may include microphone 132) as part of event sensor data 212. Microphone 132 may be an external or implantable microphone (e.g., of sensory prosthesis 110) that generates an audio signal indicative of the acoustic environment near the subject.

[0093] After operation 215, the process can move to operation 216. Operation 216 includes determining whether the audio signal includes speech asking a question. The audio signal can be analyzed by, for example, sensory prosthesis 110, computing device 150, server 170, another device, or a combination thereof. In an example, the audio signal can be analyzed to determine whether the quality of the audio signal indicates the occurrence of sensory event 10. For example, the frequency, amplitude modulation, and spectral extension of the audio signal can be analyzed (e.g., using template 180) to determine whether a likelihood threshold indicating the possible occurrence of sensory event 10 is met. In particular, an analysis can be performed to determine whether the audio signal includes speech asking a question. In some examples, the analysis of whether the audio signal includes speech asking a question can include determining an increased tone in the audio signal.

[0094] After operation 216, flow may move to operation 217. Operation 217 includes determining that a sensory event 10 (e.g., an auditory event) has occurred near a sensory prosthesis 110 (e.g., an auditory prosthesis). The determination may include setting a flag or otherwise writing data indicating that the sensory event 10 has occurred. In an example, operation 217 may be performed in response to determining in operation 216 that the audio signal includes speech asking a question.

[0095] In some examples, operation 210 or 214 includes operation 218. Operation 218 includes aggregating sensor output data. For example, aggregating sensor output data may include aggregating sensor output data from sensors of multiple different sensory prostheses (e.g., aggregating output data from a first sensory prosthesis sensor and a second sensory prosthesis sensor) to form aggregated data. Detection of an occurrence of a sensory event near a receptor may then include use of the aggregated data. In some examples, event sensor data 212 is aggregated or event sensor data 212 is aggregated data.

[0096] In some examples, operation 214 includes operation 219. Operation 219 includes determining whether sensory event 10 occurred by analyzing or processing event sensor data 212 using a machine learning framework. For example, some or all of event sensor data 212 may be provided as input to a machine learning framework that is configured to process the input and provide as output an indication of whether the input data indicates sensory event 10 occurred. More details on using a machine learning framework will be provided in Figure 4 Described in.

[0097] After operation 214, the flow may move to operation 220 (see Figure 2B). Operation 220 includes classifying the sensory event 10 as a relevant sensory event or an irrelevant sensory event. Advantageously, such classification can facilitate improvements in processing efficiency by allowing irrelevant sensory events 10 to be ignored or by allowing relevant sensory events 10 to be focused. In some examples, a relevant sensory event 10 is a sensory event 10 that may elicit a response 20 from a receptor. In some examples, a relevant sensory event 10 is a sensory event 10 of interest (e.g., for a receptor, a caregiver, or a clinician). For example, a caregiver may be interested in sensory events 10 that occur when the caregiver is not present (e.g., as may occur in a school or daycare). Thus, a relevant sensory event 10 may be a sensory event 10 that occurs in a place far from the caregiver (e.g., as may be determined by geographic isolation or based on the distance between the caregiver's device and the sensory prosthesis 110). As another example, a receptor may test an updated setting configured to improve the performance of a sensory prosthesis 110 relative to a classroom sensory event, so a relevant sensory event 10 may be a sensory event 10 that occurs in a classroom. In some examples, relevance may be customized (e.g., by a receptor, a caregiver, or a clinician). For example, the relevant sensory events may be tailored to be appropriate for the age of the recipient. For example, the relevance for an infant (e.g., a sound event that may elicit a startle reflex in an infant) may be different than the relevance for a teenager (e.g., a teenager may react differently to speech than an infant). Classification may be performed in any of a variety of ways, such as based on location (see, e.g., operation 221), thresholds (see, e.g., operation 222), machine learning frameworks (see, e.g., operation 227), other ways, or combinations thereof.

[0098] In some examples, operation 220 includes operation 221. Operation 221 includes analyzing the location. For example, the location can be determined or inferred from event sensor data 212 (e.g., using satellite-based location data or nearby wireless broadcasts as described above). The location of the sensory prosthesis 110 can be compared to a predetermined location. In response to satisfying the comparison, the sensory event 10 can be classified as a related sensory event. In response to not satisfying the comparison, the sensory event 10 can be classified as an unrelated sensory event. The comparison can be satisfied in any of a variety of ways, such as a distance between the location of the sensory prosthesis 110 and the predetermined location is less than or equal to a threshold distance. In another example, the comparison can be satisfied based on a specific wireless broadcast (e.g., a WI-FI SSID) that can be detected or undetectable by the sensory prosthesis 110 or another device associated with the receptor.

[0099] In some examples, operation 220 includes operation 222. Operation 222 includes comparing one or more thresholds to some or all of event sensor data 212. The thresholds may be related predetermined thresholds. For example, one or more thresholds may be the amount of a characteristic of a sensory event before a typical person is able to perceive or is expected to respond to the sensory event 10. In another example, the threshold is the amount of a characteristic of a sensory event 10 before a receptor is able to sense or is expected to respond to the sensory event 10 (e.g., the receptor has a higher or lower ability to sense sensory events than a typical person). The threshold may be an intensity level (e.g., decibels or lumens) before an expected response. In these examples, the decibel level of event sensor data 212 is compared to a decibel level that is a typical human hearing threshold. Operation 222 may include comparing to a plurality of different thresholds in one or more thresholds before operation 222 is completed.

[0100] In response to the event sensor data 212 satisfying one or more thresholds, the flow may move to operation 224. Operation 224 includes classifying the sensation event 10 as a relevant sensation event 10 in response to the event sensor data 212 satisfying the one or more thresholds. In response to the event sensor data 212 not satisfying the one or more thresholds, the flow may move to operation 226. Operation 226 includes classifying the sensation event 10 as an unrelated sensation event in response to the sensor data failing to satisfy the one or more thresholds. Classifying the sensation event 10 as relevant or unrelated may include setting a flag or otherwise storing data indicating the relevance of the event. In some examples, data regarding unrelated events is discarded.

[0101] In some examples, operation 220 includes operation 227. Operation 227 includes determining the relevance of sensory event 10 to a machine learning framework. For example, some or all of event sensor data 212 may be provided as input to a machine learning framework that is configured to process the input and provide as output an indication that the input data indicates whether sensory event 10 is relevant or irrelevant. More details on using a machine learning framework will be provided in Figure 4 Described in.

[0102] After operation 220 and in response to the sensory event being an irrelevant sensory event, the flow may move to operation 228. Operation 228 includes avoiding detecting the receptor's response 20 to the occurrence of the irrelevant sensory event 10. Such avoiding detection may conserve battery and computing resources of the sensory prosthesis 110. After operation 228, the flow may return to operation 210 (see Figure 2A ).

[0103] After operation 220 and in response to the sensory event being a relevant sensory event, flow may move to operation 230 .

[0104] Operation 230 includes detecting a response 20 of a receptor to an occurrence of a sensory event 10 using response sensor data 231. Response sensor data 231 is data obtained from one or more of sensors 130 that can be used to detect whether a receptor has a response 20 to a sensory event 10. In some examples, response sensor data 231 is or includes the same event sensor data 212 used to detect the occurrence of sensory event 10. For example, event sensor data 212 may include sufficient data (e.g., data covering a sufficiently long time span) to cover the occurrence of sensory event 10 and the response 20. In some examples, additional data is collected from sensors 130 to cover the time of the receptor's response 20. The additional data may be added to event sensor data 212 to form response sensor data 231. In some examples, when at least some of event sensor data 212 is used as response sensor data 231, portions of event sensor data 212 may be discarded. In addition, response sensor data 231 used to detect a response 20 may be different from event sensor data 212. For example, a motion sensor that tracks the motion of a receptor may be useful in determining a receptor's response 20 to a sensory event 10, but is unlikely to determine whether a sensory event 10 occurred (unless, for example, the sensory event 10 was a tactile sensory event 10). Response sensor data 231 may be obtained in a manner similar to the collection of event sensor data 212 described above.

[0105] In some examples, operation 230 includes operation 232. Operation 232 includes detecting a speech response of the recipient. Detecting a speech response may include detecting a vocalization of the recipient. The vocalization may be detected in any of a variety of ways. For example, an implanted or external microphone 132 may be used to detect the vocalization. The audio data from the microphone 132 may be analyzed (e.g., the characteristics of the audio data may be compared to one or more thresholds of frequency, pitch, volume, or other characteristics) to determine whether the recipient is making a vocal response. In addition to or in lieu of using audio data to identify a speech response, operation 232 may include detecting changes in heart rate characteristics of a recipient who is preparing to speak or currently speaking. For example, an inhalation feature of a recipient who is preparing to speak or currently speaking may be detected. As another example, a cortical cascade of a recipient who is preparing to speak or currently speaking may be detected. In addition, the cortical cascade may indicate a recipient response problem (e.g., a problem detected in operation 216).

[0106] In some examples, operation 230 includes operation 234. When sensory event 10 includes a question being asked (e.g., a question detected in operation 216), operation 230 may include operation 234. Operation 234 includes determining the behavior of the recipient when the question is asked. In some examples, operation 234 includes operation 235. Operation 235 may include measuring the amount or duration of stillness by the recipient. In some examples, measuring the duration or amount of stillness by the recipient may include measuring muscle movement of the recipient. For example, measuring muscle movement may be performed using an electromyogram, a camera, an accelerometer, a gyroscope, or another sensor. The stillness of the recipient may be used to infer that the recipient is listening to the question (and therefore is able to hear the question and is making an appropriate response).

[0107] In some examples, operation 230 includes operation 236. Operation 236 includes comparing response sensor data 231 to one or more predetermined thresholds. In response to response sensor data 231 satisfying one or more thresholds, it can be determined that response 20 occurred. In response to response sensor data 231 failing to satisfy one or more thresholds, it can be determined that response 20 did not occur.

[0108] In some examples, operation 230 includes operation 238. Operation 238 includes using a machine learning framework to determine whether response 20 occurred. For example, some or all of response sensor data 231 can be provided as input to a machine learning framework that is configured to process the input and provide as output an indication of whether the input data indicates that response 20 occurred. More details on using a machine learning framework will be provided in Figure 4 Described in.

[0109] After operation 230, the flow may move to operation 240 (see Figure 2D ). Operation 240 includes the subject characterizing the response 20 using the event sensor data 212. In an example, characterizing the response 20 may include marking the response 20 as appropriate for the sensory event 10. As previously described, the response 20 may be a lack of an appropriate response. For example, if the subject's movement is relatively unchanged before, during, and after approaching the subject to ask a question, the response may be determined to be inappropriate. An appropriate response 20 may be the subject remaining still or exhibiting reduced movement during or after the question being asked. Characterizing the response 20 as appropriate or inappropriate may include comparing the actual response to an expected appropriate response from a typical individual or subject.

[0110] In some examples, operation 240 includes operation 242. Operation 242 includes determining the length of time it takes for the receptor to begin providing a voice response in response to sensory event 10. For example, the length of time taken can be the time it takes from the sensory event 10 to occur (e.g., the question is asked) to the receptor to begin providing a voice response. In an example, the receptor beginning to provide a voice response can be detected in response to detecting a change in heart rate, a change in cortical activity, or an inhalation. In response to the length of time being less than a threshold amount of time, the response 20 can be characterized as being appropriate. Otherwise, the response 20 can be characterized as being inappropriate.

[0111] In some examples, operation 240 includes operation 244. Operation 244 includes determining the length of the voice response provided by the subject. The length of the voice response provided by the subject can be determined using data obtained from an internal or external microphone, for example. In response to the length of the voice response being less than a threshold amount of time, response 20 can be characterized as being appropriate. Otherwise, response 20 can be characterized as being inappropriate.

[0112] In some examples, operation 240 includes operation 246. Operation 246 includes determining whether response sensor data 231 indicates that the receptor is performing a characteristic reaction. This may include comparing response sensor data 231 to a predetermined threshold associated with the characteristic reaction. For example, there may be an amount of movement below which the receptor has an appropriate hearing response to hearing speech sensory event 10. There may be a detectable amount of activity in response sensor data 231 that indicates that the receptor is moving (e.g., dancing or performing another action) in response to sensory event 10.

[0113] In some examples, characterizing response 20 includes characterizing the response using a machine learning framework. For example, some or all of response sensor data 231 can be provided as input to a machine learning framework that is configured to process the input and provide a characterization of response 20 as output. More details on using a machine learning framework will be provided in Figure 4 Described in.

[0114] After operation 240, the flow may move to operation 250 (see Figure 2E ). Operation 250 includes taking a processing action 251 associated with the receptor. Processing action 251 may involve improving the receptor's ability to respond to sensory events 10, such as by improving the ability of sensory prosthesis 110 to induce sensory perception in the receptor. In an example, processing action 251 is an action related to treating a medical condition associated with one or more senses of the receptor. Based on the characterization of response 20, various processing actions 251 may be performed, selected, determined, or recommended.

[0115] In an example, processing action 251 includes reporting the performance quality of sensory prosthesis 110 with respect to sensory event 10 , such as to a clinician (e.g., to help guide treatment) or a caregiver (e.g., to help assure a caregiver that sensory prosthesis 110 is functioning as intended).

[0116] The processing action 251 may also include using the characterization of the response 20 as input to a rehabilitation program. The rehabilitation program may take any of a variety of forms and may generally involve rehabilitating the receptor to more appropriately respond to the sensory event 10. An example of a rehabilitation program is the BRING BACK THE BEAT smartphone application developed by COCHLEAR LTD., which involves improving the receptor's ability to respond to musical sensory events.

[0117] In some examples, operation 250 includes operation 252. Operation 252 includes modifying one or more settings 253 of the sensory prosthesis 110. In this operation 252, modifying the settings 253 may result in a modification of the processing operation of the sensory prosthesis. Based on the characterization of the response 20 (e.g., whether the response 20 is appropriate or inappropriate), it can be determined that the sensory prosthesis settings 146 suboptimally induce sensory perception in the subject with respect to a particular sensor event, and one or more changes in the sensory prosthesis settings 146 may improve the performance of the sensory prosthesis 110 with respect to the sensory event 10. Based on this determination, information about the one or more changes may be provided to the subject, a caregiver, or a clinician (e.g., by way of a report). In some examples, the one or more changes are automatically adopted by the sensory prosthesis 110 itself. The sensory prosthesis settings 146 are then changed, which modifies the ongoing operation of the sensory prosthesis 110. In some examples, the sensory prosthesis settings 146 are changed for a particular scene. For example, the sensory prosthesis settings 146 associated with a speech mode (e.g., as determined by a scene classifier of the sensory prosthesis 110) are changed, but not changed in other modes (e.g., music or wind instrument modes). This scene-specific approach can be advantageous because changes that improve the ability of the sensory prosthesis 110 to produce sensory perceptions of one sensory event 10 that are understandable to the recipient can have concomitant deleterious effects on the ability of the sensory prosthesis 110 to produce perceptions of other sensory events 10 that are understandable to the recipient.

[0118] In some examples, operation 250 includes operation 254. Operation 254 includes reporting a performance quality of the sensory prosthesis 110 with respect to the sensory events 10. The performance quality may include whether the receptors respond appropriately to the sensory events 10, which may serve as a proxy indication of whether the sensory prosthesis 110 accurately elicits the sensory perception.

[0119] In some examples, operation 250 includes operation 256. Operation 256 includes suggesting a corrective action. In an example, processing action 251 includes recommending a corrective action for increasing the likelihood that the recipient will respond appropriately to sensory event 10. In an example, the corrective action may be a reconfiguration or reprogramming of sensory prosthesis 110. In another embodiment, the corrective action includes modifying the treatment provided to the recipient. For example, in the case where sensory prosthesis 110 is a unilateral hearing prosthesis, modifying the treatment may include advancing to a bilateral prosthesis.

[0120] In some examples, operation 250 includes operation 258. Operation 258 includes providing a metric that estimates the ability of the receptor to participate in the sensory event 10. The metric may include, for example, the total number of appropriate responses 20 to the sensory event 10, the total number of inappropriate responses 20 to the sensory event 10, the total number of responses 20 per day, the rate of inappropriate responses 20, and the average level of responses 20. In addition, trends of these metrics can be calculated. These trends can reveal useful information about the sensory health of the receptor, such as determining the improvement or decline in the ability of the receptor to respond to the sensory event 10. Such trend information is useful in determining the processing selection of the receptor. The metric can be segmented based on the quality of the sensory event (if such quality is detected or known). For example, in the case where the sensory event 10 is a musical event, the quality can include the main frequency, rhythm, volume, genre, singer (e.g., male singer, female singer, age, language spoken, accent, etc.). Segmentation of the sensory event 10 based on quality can further enhance the processing action by allowing conclusions to be drawn based on how the receptor is able to perceive a particular type of sensory event. For example, a metric may indicate that a receptor is better able to perceive speech in a one-on-one setting than in a classroom.

[0121] After operation 250 , the flow may return to operation 210 .

[0122] Example Process - Second Example Process

[0123] Figure 3 An example process 300 for monitoring the sensory capabilities of an individual and taking processing actions based thereon is shown. As shown, process 300 includes some of the same operations as process 200. Generally, process 300 also includes details about the operation of sensory prosthesis 110 in monitoring mode 30 or non-monitoring mode 40, and further provides analysis to a caregiver device. In the example, process 300 begins with operation 310.

[0124] Operation 310 includes selectively operating the sensory prosthesis 110 in a monitoring mode 30 or a non-monitoring mode 40. The monitoring mode 30 is characterized in that the sensory prosthesis 110 monitors sensory events 10 and monitors the response 20 of the recipient. The non-monitoring mode 40 is characterized in that the sensory prosthesis 110 does not monitor sensory events 10 or does not monitor the response 20 of the recipient. The sensory prosthesis 110 can selectively operate in the modes 30, 40 based on various conditions. In an example, the modes 30, 40 can be manually switched by the recipient or a caregiver.

[0125] In some examples, operation 310 includes operation 312. Operation 312 includes operating in non-monitoring mode 40 based on proximity. The proximity can be the proximity between the recipient and the caregiver. Operating in non-monitoring mode 40 is advantageous when the recipient is close to the caregiver because the caregiver can personally monitor the degree of response of the recipient to the sensory event 10. In this case, the response 20 may not need to be automatically monitored by the sensory prosthesis 110. In contrast, when the recipient is away from the caregiver (e.g., at school), the caregiver may prefer to monitor the response 20. By selectively operating in modes 30, 40 (e.g., based on proximity), the sensory prosthesis 110 can save computing resources and privacy by avoiding collecting and processing unnecessary data.

[0126] In other examples, the proximity can be the proximity between the subject and another person or location. The proximity can be between the subject and a particular region of interest. For example, the subject can test new settings for the sensory prosthesis 110 to improve the ability of the sensory prosthesis 110 to induce sensory perception in areas with high levels of noise (e.g., background noise in the case where the sensory prosthesis 110 is an auditory prosthesis). To facilitate data collection for the test, the region of interest can be set (e.g., by the subject, a caregiver, or a clinician) to an area that is likely to have high noise levels.

[0127] Proximity can be determined or inferred in a variety of ways. In an example, proximity is determined based on user input (e.g., receiving user input indicating that the receptor is proximate to the region of interest). In another example, proximity is determined based on proximity between the sensory prosthesis 110 and another device, such as a caregiver device (e.g., computing device 150) determined based on, for example, nearby wireless signals. In another example, proximity is determined based on the location of the sensory prosthesis 110 (e.g., determined based on satellite-based location data or nearby wireless signals).

[0128] In some examples, operation 310 includes operation 314. Operation 314 includes operating in non-monitoring mode 40 based on time. For example, the time may be the current time of day. The recipient, caregiver, or clinician may set one or more time ranges for operating in modes 30, 40. For example, where the recipient is a child, it may be advantageous to monitor responses 20 to sensory events 10 in the morning when the child is more likely to be alert. This may improve the ability to monitor data to ensure that missed responses 20 are due to operation of the sensory prosthesis 110 and not due to recipient fatigue.

[0129] After operation 310, flow may move to operation 320. Operation 320 may include determining whether the sensory prosthesis 110 is operating in the monitoring mode 30 or the non-monitoring mode 40. In response to the sensory prosthesis 110 operating in the non-monitoring mode 40, flow may move to operation 330.

[0130] Operation 330 includes avoiding detecting the occurrence of a sensory event 10 proximate to the sensory prosthesis. Operation 330 can take various forms. This can include avoiding obtaining event sensor data 212, avoiding processing event sensor data 212, or avoiding taking action after processing event sensor data 212. In other examples, operation 330 can include not obtaining response sensor data 231, not processing response sensor data 231, or not taking action after processing response sensor data 231. Advantageously, avoiding detection can save battery life of the sensory prosthesis 110 (where the sensory prosthesis 110 is powered by a battery) and can save computing resources. In an example, operation 330 can be performed while in non-monitoring mode 40. After operation 330, the process can return to operation 310.

[0131] In response to operating in monitoring mode 30, the flow may move to operation 210 (e.g., when operating in monitoring mode 30, the flow may move to operation 210), operation 220, operation 230, and operation 240. Operations 210, 220, 230, and 240 may be as described above with respect to FIG. 2. After operation 240, the flow may move to operation 340.

[0132] Operation 340 includes storing sensory event data 342 regarding sensory event 10 and response data 344 regarding receptor's response 20. In some examples, sensory event data 342 may include event sensor data 212, and response data 344 may include response sensor data 231. In other examples, sensory event data 342 is data regarding a determined sensory event 10 (e.g., characterizing event sensor data 212) and response sensor data 231 may include characterizing response data 231. For example, data 342, 344 may include metrics regarding sensory event 10 and response 20, respectively (e.g., as described above).

[0133] In an example, storing data 342 , 344 may include storing data 342 , 344 as part of analysis 362 of sensory events 10 and responses 20 . In an example, operation 340 includes generating analysis 362 .

[0134] The collected and processed data about sensory events 10 and data associated with responses 20 may be aggregated on sensory prosthesis 110, computing device 150 associated with a recipient (e.g., a mobile phone), on server 170, at another device, or a combination thereof, and used to form an analysis 262 of the recipient's ability to perceive or respond to sensory events 10. In an example, analysis 262 is data representing conclusions or inferences generated based on data about sensory events 10 and responses 20. Analysis 262 may be generated based on a comparison of responses 20 over time, such as how a recipient's responses 20 change over time.

[0135] In an example, analysis 262 is generated using a decision tree or decision rule that describes an inference or conclusion to be drawn based on the data. For example, a decision rule may specify that in response to a response 20 indicating that the receptor did not perceive a sensory event 10 having certain characteristics, the receptor has difficulty perceiving sensory events 10 having those characteristics in analysis 262. Analysis 262 may further include the metrics described above. After operation 340, the flow may move to operation 350.

[0136] Operation 350 includes transitioning to the non-monitoring mode 40. The transition may be based on, for example, a manual transition by a user from the non-monitoring mode 40 to the monitoring mode 30 (e.g., via user input). The transition may be performed using or based on one or more of the techniques described above with respect to operation 310. For example, in response to the sensory prosthesis 110 being in proximity to the caregiver device, the sensory prosthesis 110 may automatically transition from the monitoring mode 30 to the non-monitoring mode. After operation 350, the flow may move to operation 360.

[0137] Operation 360 includes transmitting an analysis 362 based on the sensory event data 342 to a caregiver device. In some examples, operation 360 is performed in response to switching from monitoring mode 30 to non-monitoring mode 40. Analysis 362 can be automatically transmitted to the caregiver device. Operation 360 can include initiating a data connection with the caregiver device via a wired or wireless connection. After operation 360, the process can move to operation 250 as described above.

[0138] After operation 250 , the flow may return to operation 310 .

[0139] Machine Learning Frameworks

[0140] Figure 4 An example machine learning framework 400 is shown that can be used with the examples herein.

[0141] The machine learning framework 400 is software and related data that implements machine learning capabilities. In the example shown, the machine learning framework 400 includes two main components: a machine learning model 410 and a machine learning interface 420. One or more aspects of the machine learning framework 400 can be implemented with a machine learning toolkit, such as: TENSORFLOW by GOOGLE, Inc. of Mountain View, California; OPENAI GYM by OPENAI, Inc. of San Francisco, California; or MICROSOFT AZURE MACHINE LEARNING by MICROSOFT, Inc. of Redmond, Washington.

[0142] The machine learning model 410 is a structured representation of learning, such as how to achieve learning and what has been learned. For example, where the machine learning model 410 includes a neural network, the machine learning model 410 may define a representation of the neural network (e.g., nodes of the neural network, connections between nodes, associated weights, and other data), such as via one or more matrices or other data structures. In another example, where the machine learning model 410 includes a decision tree, the machine learning model 410 may define the decision tree (e.g., nodes of the decision tree and connections between them). The machine learning model 410 may include multiple different types of machine learning techniques. For example, the machine learning model 410 may define multiple different neural networks, decision trees, and other machine learning techniques and connections between them. For example, the output of a first neural network may flow to the input of a second neural network, where the output from the second neural network flows into the decision tree to produce a final output.

[0143] The machine learning interface 420 defines a software interface used in conjunction with the machine learning model 410. For example, the machine learning interface 420 may define functions, procedures, and interfaces for providing input to the machine learning model 410, receiving output from the machine learning model 410, training, and maintaining the machine learning model 410.

[0144] In some examples, the machine learning interface 420 requires preprocessing of the input data. In other examples, the machine learning interface 420 can be configured to perform preprocessing. Preprocessing can include, for example, placing the input data in a specific format used by the machine learning model 410. For example, the machine learning model 410 can be configured to process input data in a vector format, and the data provided for processing can be converted to such a format via preprocessing. In an example, the interface provides functionality to convert the provided data into a useful format and then provide the converted data as input to the machine learning model 410.

[0145] The machine learning interface 420 may define a training process 430 for preparing the machine learning model 410 for use. The machine learning model 410 may be trained or otherwise configured to receive data (e.g., event sensor data 212 from the sensor 130) as input and provide output based thereon. For example, the machine learning model 410 may be trained to receive accelerometer and gyroscope signals as input and provide an indication of whether the signals indicate a sensory event 10 as output. The training process 430 may begin at operation 432.

[0146] Operation 432 includes obtaining training data. Training data is typically a collection of human or machine curated data with known training inputs and expected training outputs that can be used to train the machine learning model 410. In the examples herein, the training data may include curated event sensor data 212 or manually created training data from many different individuals and the actual or expected output of the machine learning model 410 for the data. After operation 432, the process may move to operation 434.

[0147] Operation 434 includes processing the training data. Processing the training data includes providing the training data as input to the machine learning model 410. In an example, the training data may be provided as input to the machine learning model 410 using an associated machine learning interface 420. The machine learning model 410 then processes the input training data to produce an output.

[0148] After operation 434, the flow can move to operation 436. Operation 436 includes obtaining output from the machine learning model 410. This can include receiving output from the function. After operation 436, the flow can move to operation 438.

[0149] Operation 438 includes calculating a loss value. The loss value may be calculated using a loss function, for example, based on a comparison between an actual output of the machine learning model 410 and an expected output (e.g., a training output corresponding to a provided training input). Properties of the machine learning model 410 (e.g., weights of connections in the machine learning model) are modified based on the loss value, thereby training the model.

[0150] If the loss value is not small enough (e.g., does not meet the threshold), the process can return to operation 432 to further train the machine learning model 410. For a certain amount of training data, the training process continues until the loss value is small enough. If the loss value is small enough (e.g., less than or equal to a predetermined threshold), the process can move to operation 440.

[0151] Operation 440 includes completing the training. In some examples, completing the training includes providing the machine learning framework 400 for use in production. For example, the machine learning framework 400 with the trained machine learning model 410 can be stored on the sensory prosthesis 110, the computing device 150, the server 170, or stored at another location for use (e.g., for detecting sensory events 10). In some examples, before providing the machine learning framework 400 for use, the trained machine learning model 410 is validated using validation input-output data (e.g., data with expected outputs corresponding to specific inputs that are different from the training data), and after successful validation, the machine learning framework 400 is provided for use.

[0152] Example sensory prosthesis

[0153] As previously mentioned, sensory prosthesis 110 may take any of a variety of forms. Figures 5 to 8 Examples of these forms are described in more detail. For example, the sensory prosthesis 110 can be an auditory prosthesis, such as Figure 5 The cochlear implant system 510, as Figure 6 The transcutaneous bone conduction device 600, or Figure 7 As another example, the sensory prosthesis 110 may be a retinal prosthesis, such as Figure 8 The sensory prosthesis 110 may take other forms. These various sensory prostheses 110 may benefit from use with the systems and processes described above.

[0154] Example sensory prosthesis - cochlear implant system

[0155] Figure 5An example cochlear implant system 510 that may benefit from the use of the techniques disclosed herein is shown. Cochlear implant system 510 includes an implantable component 544 that typically has an internal receiver / transceiver unit 532, a stimulator unit 520, and an elongated lead 518. The internal receiver / transceiver unit 532 allows the cochlear implant system 510 to receive signals from and / or transmit signals to an external device 550. The external device 550 may be a button sound processor worn on the head that includes a receiver / transceiver coil 530 and a sound processing component. Alternatively, the external device 550 may be simply a transmitter / transceiver coil that communicates with a behind-the-ear device that includes a sound processing component and a microphone.

[0156] The implantable component 544 includes an internal coil 536 and, preferably, a magnet (not shown) fixed relative to the internal coil 536. The magnet can be embedded in a flexible silicone or other biocompatible encapsulant together with the internal coil 536. The transmitted signal generally corresponds to the external sound 513. The internal receiver / transceiver unit 532 and the stimulator unit 520 are hermetically sealed in a biocompatible housing, sometimes collectively referred to as a stimulator / receiver unit. The included magnet (not shown) can facilitate the operational alignment of the external coil 530 and the internal coil 536 so that the internal coil 536 can receive power and stimulation data from the external coil 530. The external coil 530 is contained in the external portion. The elongated lead 518 has a proximal end connected to the stimulator unit 520 and a distal end 546 implanted in the cochlea 540 of the recipient. The elongated lead 518 extends from the stimulator unit 520 to the cochlea 540 through the mastoid bone 519 of the recipient. The elongated lead 518 is used to provide electrical stimulation based on the stimulation data to the cochlea 540. The stimulation data may be created based on the external sound 513 and based on the sensory prosthesis setting 146 using a sound processing component.

[0157] In some examples, the external coil 530 transmits electrical signals (e.g., power and stimulation data) to the internal coil 536 via a radio frequency (RF) link. The internal coil 536 is typically a wire antenna coil of a single or multiple strands of platinum or gold wire with multiple turns of electrical insulation. The electrical insulation of the internal coil 536 can be provided by flexible silicone molding. Various types of energy transfer, such as infrared (IR), electromagnetic, capacitive, and inductive transfer, can be used to transfer power and / or data from an external device to the cochlear implant. Although the above description has described internal and external coils formed by insulated wires, in many cases, the internal and / or external coils can be implemented by conductive traces.

[0158] Example Sensory Prosthesis - Transcutaneous Bone Conduction Device

[0159] Figure 66 is a view of an example of a transcutaneous bone conduction device 600 that may benefit from the use of the techniques disclosed herein. For example, the sensory prosthesis setting 146 of the device 600 may be customized using one or more aspects of the disclosed techniques. The bone conduction device 600 is positioned behind an outer ear 601 of a device recipient. The bone conduction device 600 includes a sound input element 626 to receive a sound signal 607. The sound input element 626 may be a microphone, a telecoil, or the like. In this example, the sound input element 626 may be located, for example, on or in the bone conduction device 600, or on a cable extending from the bone conduction device 600. In addition, the bone conduction device 600 includes a sound processor (not shown), a vibrating electromagnetic actuator, and / or various other operating components.

[0160] More specifically, the sound input element 626 converts the received sound signals into electrical signals. These electrical signals are processed by the sound processor. The sound processor generates a control signal that causes the actuator to vibrate. In other words, the actuator converts the electrical signal into a mechanical force to impart vibration to the recipient's skull 636. The conversion of the electrical signal into a mechanical force can be based on the sensory prosthesis setting 146, so that different sensory prosthesis settings 146 can result in different mechanical forces being generated from the same sound signal 607.

[0161] The bone conduction device 600 also includes a coupling device 640 to attach the bone conduction device 600 to a recipient. In the example shown, the coupling device 640 is attached to an anchor system (not shown) implanted in the recipient. An example anchoring system (also referred to as a fixation system) may include a percutaneous mount fixed to the skull 636. An abutment extends from the skull 636 through the muscle 634, fat 628, and skin 632 so that the coupling device 640 can be attached thereto. Such a percutaneous mount provides an attachment location for the coupling device 640 that facilitates efficient transmission of mechanical forces.

[0162] Example Sensory Prosthesis - Transcutaneous Bone Conduction Device

[0163] Figure 7 An example of a transcutaneous bone conduction device 700 having a passive implantable component 701 that may benefit from the use of the techniques disclosed herein is shown. The transcutaneous bone conduction device includes an external device 740 and an implantable component 701. The implantable component 701 includes a passive plate 755 mounted on the bone 738 and transcutaneously coupled to a vibration actuator 742 located in a housing 744 of the external device 740. The plate 755 may be in the form of a permanent magnet, or another form that generates or reacts to a magnetic field, or otherwise allows for the establishment of a magnetic attraction between the external device 740 and the implantable component 750 sufficient to hold the external device 740 on the skin 732 of the recipient.

[0164] In the example, the vibration actuator 742 is a component that converts electrical signals into vibrations. In operation, the sound input element 726 converts sound into electrical signals. Specifically, the transcutaneous bone conduction device 700 provides these electrical signals to the vibration actuator 742, or to a sound processor (not shown) that processes the electrical signals, and then provides those processed signals to the vibration actuator 742. The way in which the sound processor processes the electrical signals can be modified based on the sensory prosthesis setting 146. The vibration actuator 742 converts the electrical signals (processed or unprocessed) into vibrations. Because the vibration actuator 742 is mechanically coupled to the plate 746, the vibration is transmitted from the vibration actuator 742 to the plate 746. The implanted plate assembly 752 is part of the implantable component 750 and is made of a ferromagnetic material, which can be in the form of a permanent magnet that generates a magnetic field and / or reacts to a magnetic field, or otherwise allows a magnetic attraction to be established between the external device 740 and the implantable component 750, which is sufficient to keep the external device 740 on the skin 732 of the recipient. Thus, vibrations generated by the vibration actuator 742 of the external device 740 are transmitted from the plate 746 through the skin 732, fat 734 and muscle 736 to the plate 755 of the plate assembly 752. This can be achieved by mechanical conduction of vibrations through tissue, which is generated by the external device 740 in direct contact with the skin 732 and / or by the magnetic field between the two plates 746, 755. These vibrations are transmitted without penetrating the skin 732 through a solid object (e.g., a support).

[0165] As can be seen, in this example, the implanted plate assembly 752 is substantially rigidly attached to the bone fastener 757. However, other bone fasteners may be used instead in this and other embodiments. In this regard, the implantable plate assembly 752 includes a through hole 754 that matches the outer contour of the bone fastener 757. The through hole 754 thus forms a bone fastener interface portion that matches the exposed portion of the bone fastener 757. In an example, the size and dimensions of this portion are determined so that there is at least a sliding fit or an interference fit relative to this portion. The plate screw 756 is used to secure the plate assembly 752 to the bone fastener 757. The head of the plate screw 756 can be larger than the hole through the implantable plate assembly 752, and thus the plate screw 756 positively retains the implantable plate assembly 752 to the bone fastener 757. The portion of the plate screw 756 that interfaces with the bone fastener 757 substantially corresponds to the abutment screw described in more detail below, thereby allowing the plate screw 756 to be easily assembled into existing bone fasteners used in percutaneous bone conduction devices. In an example, the plate screw 756 is configured such that the same tools and procedures used to install and / or remove an adjacent screw from the bone fixation device 757 can be used to install and / or remove the plate screw 756 from the bone fixation device 757. In some examples, there can be a silicone layer 759 disposed between the plate 755 and the bone 738.

[0166] Example Sensory Prosthesis - Retinal Prosthesis

[0167] Figure 8 A retinal prosthesis system 801 is shown that includes a retinal prosthesis 800 and a mobile computing device 803. The retinal prosthesis 800 includes a processing module 825, and a retinal prosthesis sensor-stimulator 890 is positioned near the retina 891 of the recipient. In an example, sensory input (e.g., photons entering the eye) is absorbed by a microelectronic array of the sensor-stimulator 890, which is hybridized to a glass sheet 892 that includes, for example, an embedded microwire array. The glass can have a curved surface that conforms to the inner radius of the retina. The sensor stimulator 890 can include a microelectronic imaging device that can be made of thin silicon that contains integrated circuits that convert incident photons into electrical charges.

[0168] The processing module 825 includes an image processor 823, which is in signal communication with the sensor-stimulator 890 via, for example, a lead 888 extending through a surgical incision 889 formed in the eye wall. In other examples, the processing module 825 can be in wireless communication with the sensor stimulator 890. The image processor 823 processes the input into the sensor stimulator 890 and provides a control signal back to the sensor stimulator 890 so that the device can provide an output to the optic nerve. That is, in an alternative example, the processing is performed by a component close to the sensor-stimulator 890 or integrated with the sensor-stimulator 890. The charge generated by the conversion of the incident photon is converted into a proportional amount of electronic current, which is input to the nearby retinal cell layer. These cells activate and the signal is transmitted to the optic nerve, thereby producing visual perception.

[0169] The processing module 825 may be implanted in the recipient, or may be part of an external device, such as a behind-the-ear (BTE) unit, a pair of glasses, etc. The retinal prosthesis 800 may also include an external light / image capture device (e.g., located in / on a BTE device or a pair of glasses, etc.), while, as described above, in some examples, the light / image is captured by a sensor-stimulator 890 that is implanted in the recipient.

[0170] Similar to the above example, the retinal prosthesis system 801 can be used in a spatial region having at least one controllable network-connected device associated therewith (e.g., located therein). As such, the processing module 825 includes a performance monitoring engine 827 configured to obtain data related to a "sensory result" or "sensory performance" of a receptor of the retinal prosthesis 800 in the spatial region. As used herein, a "sensory result" or "sensory performance" of a receptor of a sensory prosthesis such as the retinal prosthesis 800 is an estimate or measurement of how effectively a stimulus signal delivered to the receptor represents a sensor input captured from the surrounding environment.

[0171] Data representing the performance of the retinal prosthesis 800 in the spatial region is provided to the mobile computing device 803 and analyzed by the network-connected device evaluation engine 862 based on the operational capabilities of at least one controllable network-connected device associated with the spatial region. For example, the network-connected device evaluation engine 862 can determine one or more effects of the controllable network-connected devices on the sensory outcomes of the receptors in the spatial region. The network-connected device evaluation engine 862 is configured to determine one or more operational changes to the at least one controllable network-connected device that is estimated to improve the sensory outcomes of the receptors in the spatial region, and accordingly, initiate one or more operational changes to the at least one controllable network-connected device.

[0172] Example computing system

[0173] Fig. 9 An example of a suitable computing system 900 is shown, by which one or more of the disclosed examples may be implemented. Computing systems, environments, or configurations that may be suitable for use with the examples described herein include, but are not limited to, personal computers, server computers, handheld devices, laptops, multiprocessor systems, microprocessor-based systems, programmable consumer electronics (e.g., smart phones), network PCs, minicomputers, mainframe computers, tablet computers, distributed computing environments including any of the above systems or devices, and the like. The computing system 900 may be a single virtual or physical device operating in a networked environment through a communication link to one or more remote devices. The remote device may be a sensory prosthesis (e.g., sensory prosthesis 110), a personal computer, a server, a router, a network personal computer, a peer device, or other public network node. In an example, the computing device 150 and the server 170 include one or more components or variations of components of the computing system 900. In addition, in some examples, the sensory prosthesis 110 includes one or more components of the computing system 900.

[0174] In its most basic configuration, computing system 900 includes one or more processors 902 and memory 904 .

[0175] The one or more processors 902 include one or more hardware or software processors (eg, central processing units) that can obtain and execute instructions. The one or more processors 902 can communicate with other components of the computing system 900 and control their performance.

[0176] The memory 904 is one or more software or hardware-based computer-readable storage media that can be operated to store information that can be accessed by one or more processors 902. The memory 904 can store, among other things, instructions and other data that can be executed by one or more processors 902 to implement applications or to enable the execution of operations described herein. The memory 904 can be a volatile memory (e.g., RAM), a non-volatile memory (e.g., ROM), or a combination thereof. The memory 904 can include a transient memory or a non-transient memory. The memory 904 can also include one or more removable or non-removable storage devices. In an example, the memory 904 can include RAM, ROM, EEPROM (electronically erasable programmable read-only memory), flash memory, optical disk memory, magnetic memory, solid-state memory, or any other memory medium that can be used to store information for later access. In an example, the memory 904 includes a modulated data signal (e.g., a signal in which one or more of its characteristics are set or changed in a manner to encode information in the signal), such as a carrier or other transmission mechanism, and includes any information delivery medium. By way of example, and not limitation, memory 904 may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media, or combinations thereof.

[0177] In the example shown, system 900 also includes a network adapter 906, one or more input devices 908, and one or more output devices 910. System 900 may include other components such as a system bus, component interfaces, a graphics system, a power source (eg, a battery), and so forth.

[0178] The network adapter 906 is a component of the computing system 900 that provides network access. The network adapter 906 can provide wired or wireless network access and can support one or more of a variety of communication technologies and protocols, such as Ethernet, cellular, Bluetooth, near field communication, and RF (radio frequency), etc. The network adapter 906 may include one or more antennas and associated components configured for wireless communication according to one or more wireless communication technologies and protocols.

[0179] One or more input devices 908 are devices through which the computing system 900 receives input from a user. One or more input devices 908 may include input devices such as physically actuatable user interface elements (e.g., buttons, switches, or dials), touch screens, keyboards, mice, pens, and voice input devices.

[0180] One or more output devices 910 are devices through which the computing system 900 can provide output to a user. The output device 910 may include a display, a speaker, a printer, and other output devices.

[0181] As should be understood, although the specific uses of the technology have been described and discussed above, according to many examples of the technology, the disclosed technology can be used with various devices. The above discussion is not meant to imply that the disclosed technology is only suitable for implementation in a system similar to that shown in the figure. For example, although some of the technologies described herein are mainly described in the context of an auditory prosthesis (e.g., a cochlear implant), the technology disclosed herein is generally applicable to medical devices (e.g., a medical device that provides pain management functions or therapeutic electrical stimulation, such as deep brain stimulation). Generally, additional configurations can be used to practice the processes and systems herein and / or some aspects described can be excluded without departing from the processes and systems disclosed herein. In addition, the technology described herein may be applicable to determining the response of a receptor to other stimuli, such as visual stimuli, tactile stimuli, olfactory stimuli, taste stimuli or other stimuli. Similarly, the devices used herein need not be limited to auditory prostheses, and may be other medical devices configured to support human senses, such as bionic eyes.

[0182] The present disclosure describes some aspects of the present technology with reference to the accompanying drawings, of which only some possible aspects are shown. However, other aspects may be embodied in many different forms and should not be construed as limited to the aspects described herein. Instead, these aspects are provided so that the present disclosure is thorough and complete, and the scope of possible aspects is fully conveyed to those skilled in the art.

[0183] As should be understood, the various aspects (e.g., parts, components, etc.) described with respect to the figures herein are not intended to limit the systems and processes to the specific aspects described. Therefore, additional configurations can be used to practice the methods and systems herein and / or some aspects described can be excluded without departing from the methods and systems disclosed herein.

[0184] Similarly, where steps of a process are disclosed, the steps are described for the purpose of illustrating the present methods and systems, and not for the purpose of limiting the disclosure to a particular sequence of steps. For example, the steps may be performed in a different order, two or more steps may be performed simultaneously, additional steps may be performed, and disclosed steps may be eliminated without departing from the present disclosure. Furthermore, the disclosed process may be repeated.

[0185] Although specific aspects are described herein, the scope of the present technology is not limited to those specific aspects. Those skilled in the art will recognize other aspects or improvements within the scope of the present technology. Therefore, specific structures, actions or media are disclosed only as illustrative aspects. The scope of the present technology is limited by the attached claims and any equivalents therein.

Claims

1. A medical device comprising: implantable or wearable housing; a stimulator disposed at least partially within the housing and configured to deliver energy to tissue of a recipient of the medical device to evoke an auditory perception in the recipient; Multiple sensors; A processor configured to perform operations including: receiving event sensor data from the plurality of sensors; detecting an occurrence of an auditory event in the vicinity of the medical device using the event sensor data, wherein detecting the occurrence of the auditory event includes determining whether the auditory event includes speech that may require a response; determining a response of the receptor to the occurrence of the auditory event using response sensor data, the response comprising a verbal or physiological response; characterizing the response of the receptor using the response sensor data; as well as Based on characterizing the response, a processing action associated with the receptor is initiated.

2. The medical device according to claim 1, The plurality of sensors include: microphone; Wherein detecting the occurrence comprises: receiving an audio signal from the microphone; determining whether the audio signal includes speech asking a question; and In response to determining that the audio signal includes speech asking a question, determining that an auditory event near the medical device has occurred; Wherein determining the response of the receptor comprises: determining a speech response of the receptor; and Wherein characterizing the response of the receptor comprises: determining the length of time from when the question is asked until the subject begins to provide the speech response; and A length of the speech response provided by the receptor is determined.

3. The medical device of claim 2, wherein determining the response of the receptor comprises: Determine the behavior of the receptor when the question is asked.

4. The medical device of claim 3, wherein determining the behavior of the receptor comprises At least one of an amount of immobility or a duration of immobility of the subject while the question is asked is measured.

5. The medical device of claim 4, wherein measuring at least one of the amount of rest or the duration of rest of the subject comprises: measuring muscle movement of the receptors using electromyography; measuring muscle movement of the receptor using a camera; measuring muscle movement of the subject using an accelerometer; or The receptor's muscle movements were measured using a gyroscope.

6. The medical device of claim 2, wherein determining the voice response comprises: determining vocalizations of the subject using an implanted microphone; determining vocalizations of the subject using an external microphone; determining changes in heart rate characteristics of an individual preparing to speak; Determining the inhalation characteristics of an individual preparing to speak; or Determine the cortical cascade of said receptors that respond to said question.

7. The medical device of claim 1, wherein the auditory event comprises: The vocal activity of speaking; Onomatopoeic vocalization activities; the vocal activity of whispering; or Music activities.

8. The medical device of claim 1, wherein the auditory event comprises: The vocal activity of singing.

9. The medical device of claim 1, wherein the auditory event comprises: The vocal activity of singing.

10. The medical device of claim 1, wherein the response is a vocal response, a motor response, or a physiological response.

11. The medical device of claim 1 , wherein the operations further comprise: classifying the auditory event as a relevant sensory event or an irrelevant sensory event, Wherein determining the response of the receptor is performed in response to the auditory event being classified as a relevant sensory event.

12. The medical device of claim 1, wherein the plurality of sensors comprises at least one sensor selected from the group consisting of: a microphone, a body noise sensor, a motion sensor, and an implanted electrode sensor.

13. The medical device according to claim 12, wherein the plurality of sensors includes the motion sensor; and The motion sensor is an accelerometer or a gyroscope.

14. The medical device of claim 1, wherein initiating the processing action comprises: modifying one or more settings of the medical device; reporting a quality of performance of the medical device with respect to the auditory event; Suggest corrective actions; or A metric is provided that estimates the ability of the receptor to participate in the auditory event.

15. The medical device of claim 1, wherein the stimulator comprises: a stimulator unit, disposed within the housing; a stimulator assembly disposed at least partially exterior to the housing; as well as An electrode contact array is arranged on the stimulator assembly.

16. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: receiving sensor data from a plurality of sensors proximate to a receptor of a sensory device; detecting the occurrence of a sensory event in the vicinity of the sensory device using the sensor data; classifying the sensory event as a relevant sensory event or an irrelevant sensory event; in response to the sensory event being a relevant sensory event, detecting a response of the receptor to the occurrence of the sensory event using the response sensor data, wherein detecting the occurrence of the sensory event includes determining whether the sensory event includes speech that may require a response; characterizing the response of the receptor using the response sensor data; and A processing action associated with the receptor is taken based on the characterization.

17. The non-transitory computer readable medium of claim 16, wherein a sensory event of interest is an event that is expected to elicit a characteristic response in the receptor.

18. The non-transitory computer readable medium of claim 17, wherein characterizing the response of the receptor comprises: A determination is made as to whether the sensor data indicates that the receptor is performing the characteristic response.

19. The non-transitory computer-readable medium of claim 16, wherein classifying the sensory event as a relevant sensory event or an irrelevant sensory event comprises: A location specified by the sensor data is analyzed.

20. The non-transitory computer-readable medium of claim 16, wherein classifying the sensory event as a relevant sensory event or an irrelevant sensory event comprises: comparing a threshold value to the sensor data; responsive to the sensor data satisfying the threshold, classifying the sensory event as a relevant sensory event; as well as In response to the sensor data not satisfying the threshold, the sensory event is classified as an irrelevant sensory event.

21. The non-transitory computer readable medium of claim 16, wherein the instructions further cause the processor to: In response to the sensory event being an irrelevant sensory event, determining the response of the receptor to the occurrence of the sensory event is avoided.

22. The non-transitory computer-readable medium of claim 16, wherein taking the processing action comprises: The processing operation of the sensory device is modified by changing one or more settings of the sensory device.

23. The non-transitory computer-readable medium of claim 16, wherein characterizing the response comprises: The response is marked as appropriate for the sensory event.

24. The non-transitory computer readable medium of claim 16, Wherein the sensory device is a medical device comprising the non-transitory computer readable medium.

Images