Leverage prediction and identification techniques used by hearing prostheses

Abstract

A method comprises: receiving a signal comprising speech data, processing the received signal to identify and / or predict one or more words in the speech data, and evoking an auditory perception based on the received signal, wherein the evoked auditory perception comprises one or more modified words based on the identification and / or prediction of the one or more words.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 865,536, filed on June 24, 2019, entitled “PREDICTION AND IDENTIFICATION TECHNIQUES USED WITH A HEARING PROSTHESIS,” the inventor of which is Paul Michael Carter of Macquarie University, Australia, the entire contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Hearing loss can arise from many different causes, and there are generally two types: conductive and sensorineural. Sensorineural hearing loss is due to the loss or destruction of hair cells in the cochlea, which convert sound signals into nerve impulses. Various hearing prostheses are commercially available to provide people with sensorineural hearing loss with the ability to perceive sound. An example of a hearing prosthesis is a cochlear implant. Conductive hearing loss occurs when the normal mechanical pathway that provides sound to the hair cells in the cochlea is blocked, such as damage to the chain of ossicles or the ear canal. People with conductive hearing loss may retain some form of residual hearing because the hair cells in the cochlea may remain intact.

[0004] People with hearing loss usually wear acoustic hearing aids. Traditional hearing aids rely on the principle of air conduction to transmit sound signals to the cochlea. Specifically, hearing aids usually use a device located in the recipient's ear canal or on the outer ear to amplify the sound received by the recipient's outer ear. The amplified sound reaches the cochlea, causing the perilymph to move and stimulate the auditory nerve. Cases of conductive hearing loss are usually treated with the help of bone conduction hearing aids. Unlike traditional hearing aids, these devices use mechanical actuators coupled to the skull to apply amplified sound. Unlike hearing aids that rely mainly on the principle of air conduction, certain types of hearing prostheses, usually called cochlear implants, convert received sounds into electrical stimulation. Electrical stimulation is applied to the cochlea so that the received sound is perceived. Many devices (such as medical devices that are docked with recipients) have structural features and / or functional features, where adjusting these features for individual recipients has practical value. The process of tailoring or customizing or adjusting a device that is docked with a recipient or used by a recipient to adapt to the specific needs or specific needs or specific features of the recipient is usually called adaptation. One type of medical device that may be of practical value when installed in an individual recipient is the cochlear implant described above. That said, there are other types of medical devices (such as other types of hearing prostheses) that may also be of practical value when installed in a recipient. Summary of the invention

[0005] In an example embodiment, a method is provided that includes receiving a signal including speech data, processing the received signal to identify and / or predict one or more words in the speech data, and evoking an auditory perception based on the received signal, wherein the evoked auditory perception includes one or more modified words based on the identification and / or prediction of the one or more words.

[0006] In an example embodiment, a method is provided that includes receiving a signal including speech data, processing the received signal to identify and / or predict one or more words in the speech data, and evoking an auditory perception based on the received signal, wherein the evoked auditory perception is a different auditory perception relative to what would have been the case in the absence of the identification and / or prediction.

[0007] In an example embodiment, a system is provided, comprising: a hearing prosthesis, the hearing prosthesis including a sound capture device and a processing system, wherein the hearing prosthesis is configured to convert sounds captured by the system, and process signals based on the captured sounds using the processing system, and evoke auditory perception based on an output of the processing system based on the signals, wherein the system includes sound prediction and / or word identification functions, such that the system is configured to predict sounds captured by the sound capture device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments will be described below with reference to the accompanying drawings, in which:

[0009] Figure 1 is a perspective view of an example hearing prosthesis to which at least some of the teachings detailed herein are applicable;

[0010] Figure 2A and 2B An example system including a hearing prosthesis and a remote device in the form of a portable handheld device is shown;

[0011] Figure 3 shows a functional block diagram of an example embodiment;

[0012] Figure 4-6 An example flow chart illustrating an example method;

[0013] Figure 7 shows a functional block diagram of an example embodiment;

[0014] Figure 8 An example flow chart illustrating an example method;

[0015] Fig. 9 illustrates an example functional schematic diagram of an example system according to an example embodiment; and

[0016] Fig.10 Another example functional schematic diagram of an example system according to an example embodiment is shown. DETAILED DESCRIPTION

[0017] Figure 1 is a perspective view of a cochlear implant (referred to as cochlear implant 100) implanted in the ear of a recipient, to which some embodiments and / or variations thereof described in detail herein are applicable. Cochlear implant 100 is part of system 10, which in some embodiments may include external components, as described in detail below. In addition, it should be noted that the teachings described in detail herein are also applicable to other types of hearing prostheses, such as, by way of example only and not limitation, bone conduction devices (transcutaneous, active transcutaneous and / or passive transcutaneous), direct acoustic cochlear stimulators, middle ear implants, and conventional hearing aids, among others. In fact, it should be noted that the teachings described in detail herein are also applicable to so-called multimodal devices. In example embodiments, these multimodal devices apply electrical and acoustic stimulation to the recipient. In example embodiments, these multimodal devices evoke auditory perception via electrical hearing and bone conduction hearing. Therefore, any disclosure herein about one of these types of hearing prostheses corresponds to the disclosure of another of these types of hearing prostheses or any medical device for that purpose, unless otherwise stated, or unless its disclosure is incompatible with a given device based on the current state of the art. Thus, in at least some embodiments, the teachings detailed herein may be applied to partially implantable and / or fully implantable medical devices that provide a wide range of therapeutic benefits to a recipient, patient, or other user, including auditory implants with implanted microphones, auditory brain stimulators, visual prostheses (e.g., bionic eyes), sensors, and the like.

[0018] In view of the above, it should be understood that at least some of the embodiments and / or variations thereof described herein relate to body-worn sensory compensation medical devices (e.g., Figure 1 It should be noted that at least some example embodiments of some sensory compensating medical devices relate to devices such as conventional hearing aids that compensate for hearing while retaining some natural hearing ability, and visual prostheses that are suitable for recipients with some natural visual ability and for recipients without natural visual ability. Therefore, the teachings detailed herein are applicable to any type of sensory compensating medical device and can be used in such sensory compensating medical devices in a practical manner. In this regard, the term "sensory compensating medical device" refers to any device used to provide a sensation to a recipient, regardless of whether the applicable natural sensation is only partially impaired or completely impaired, or was actually never present.

[0019] The recipient has an outer ear 101, a middle ear 105, and an inner ear 107. Components of the outer ear 101, the middle ear 105, and the inner ear 107 will be described below, followed by a description of the cochlear implant 100.

[0020] In a fully functional ear, the outer ear 101 includes the auricle 110 and the ear canal 102. Sound pressure or sound waves 103 are collected by the auricle 110 and directed into and through the ear canal 102. The eardrum 104 spans the far end of the ear canal 102, which vibrates in response to the sound waves 103. This vibration is coupled to the oval window or elliptical window 112 through three bones of the middle ear 105 (collectively referred to as the ossicles 106, including the malleus 108, the incus 109, and the stapes 111). The bones 108, 109, and 111 of the middle ear 105 are used to filter and amplify the sound waves 103, so that the oval window 112 is articulated or vibrated in response to the vibration of the eardrum 104. This vibration causes the perilymph fluid in the cochlea 140 to form fluid motion waves. This fluid motion, in turn, activates tiny hair cells (not shown) inside the cochlea 140. Activation of the hair cells results in generation of appropriate nerve impulses which are transmitted via spiral ganglion cells (not shown) and the auditory nerve 114 to the brain (also not shown) where they are perceived as sound.

[0021] As shown, cochlear implant 100 includes one or more components that are temporarily or permanently implanted in a recipient. Figure 1 , cochlear implant 100 is shown with external device 142, which (along with cochlear implant 100) is part of system 10, as described below, which is configured to provide power to the cochlear implant, wherein the implanted cochlear implant includes a battery that is charged by power provided by external device 142.

[0022] exist Figure 1 In the illustrative arrangement of , external device 142 may include a power source (not shown) disposed in behind-the-ear (BTE) unit 126. External device 142 also includes components of a transcutaneous energy transfer link (referred to as an external energy transfer assembly). The transcutaneous energy transfer link is used to transfer power and / or data to cochlear implant 100. Various types of energy transfer, such as infrared (IR), electromagnetic, capacitive, and inductive transfer, may be used to transfer power and / or data from external device 142 to cochlear implant 100. Figure 1 In the illustrative embodiment of the invention, the external energy transfer assembly includes an external coil 130, which forms part of an inductive radio frequency (RF) communication link. The external coil 130 is typically a wire antenna coil composed of multiple turns of electrically insulated single or multiple strands of platinum or gold wire. The external device 142 also includes a magnet (not shown) located within the turns of the external coil 130. It should be understood that Figure 1 The external devices shown in are merely illustrative and embodiments may use other external devices.

[0023] The cochlear implant 100 includes an internal energy transfer assembly 132 that can be positioned in a recess of the temporal bone adjacent to the recipient's auricle 110. As described in detail below, the internal energy transfer assembly 132 is a component of a transcutaneous energy transfer link and receives power and / or data from an external device 142. In this illustrative embodiment, the energy transfer link includes an inductive RF link and the internal energy transfer assembly 132 includes a primary internal coil 136. The internal coil 136 is typically a wire antenna coil composed of multiple turns of electrically insulated single or multiple strands of platinum or gold wire.

[0024] Cochlear implant 100 also includes a main implantable component 120 and an elongated electrode assembly 118. In some embodiments, internal energy transfer assembly 132 and main implantable component 120 are sealed within a biocompatible housing. In some embodiments, main implantable component 120 includes an implantable microphone assembly (not shown) and a sound processing unit (not shown) to convert sound signals received by the implantable microphone in internal energy transfer assembly 132 into data signals. That is, in some alternative embodiments, the implantable microphone assembly can be located in a separate implantable component (e.g., a component having its own housing, etc.), which communicates signals with main implantable component 120 (e.g., by passing a wire between the separate implantable component and main implantable component 120, etc.). In at least some embodiments, the teachings detailed herein and / or variations thereof can be used for any type of implantable microphone device.

[0025] The main implantable component 120 also includes a stimulator unit (also not shown) that generates electrical stimulation signals based on the data signals. The electrical stimulation signals are transmitted to the recipient via the elongated electrode assembly 118.

[0026] The proximal end of the elongated electrode assembly 118 is connected to the main implantable component 120, and the distal end is implanted in the cochlea 140. The electrode assembly 118 extends from the main implantable component 120 to the cochlea 140 through the mastoid bone 119. In some embodiments, the electrode assembly 118 can be implanted at least in the basal region 116, and sometimes further. For example, the electrode assembly 118 can extend toward the top of the cochlea 140 (referred to as the cochlear apex 134). In some cases, the electrode assembly 118 can be inserted into the cochlea 140 via the basal turn 122. In other cases, the basal turn can be formed by the round window 121, the oval window 112, the promontory 123, or by the apical periphery 147 of the cochlea 140.

[0027] The electrode assembly 118 includes an array 146 of longitudinally aligned and distally extending electrodes 148 disposed along its length. As described above, the stimulator unit generates stimulation signals that are applied by the electrodes 148 to the cochlea 140, thereby stimulating the auditory nerve 114.

[0028] Figure 2A An example system 210 is depicted according to an example embodiment, including a hearing prosthesis 100 and a portable body-worn device (e.g., Figure 2A The system includes a portable handheld device (smartphone, watch, pocket device, any body-worn device) 240 shown, which in the exemplary embodiment corresponds to the cochlear implant 100 described in detail above, and the portable body-worn device 240 can be in the form of a mobile computer with a display 242. The system includes a wireless link 230 between the portable handheld device 240 and the hearing prosthesis 100 (in some embodiments, the link can be wired). In the exemplary embodiment, the hearing prosthesis 100 is an implant (functionally composed of Figure 2A Again, note that although the embodiments detailed herein will be described with respect to use with cochlear implants, the teachings herein may be applicable to other types of prostheses.

[0029] In an example embodiment, the system 210 is configured such that the hearing prosthesis 100 and the portable handheld device 240 have a symbiotic relationship. In an example embodiment, the symbiotic relationship is the ability to display data related to one or more functions of the hearing prosthesis 100 and / or affect the operation of the overall system, and at least in some cases, the ability to control the one or more functions and / or affect the operation of the overall system, as will be described in more detail below. In an example embodiment, this can be achieved by the ability of the handheld device 240 to receive data from the hearing prosthesis 100 via a wireless link 230 (although in other example embodiments, other types of links, such as wired links, can be utilized). As will be described in more detail below, this can be achieved by communicating with a geographically remote device that communicates with the hearing prosthesis 100 and / or the portable handheld device 240 via a link (such as, by way of example only and not limitation, an Internet connection or a cell phone connection). In some such example embodiments, the system 210 may also include geographically remote devices. Again, other examples of this are described in more detail below.

[0030] As described above, in an example embodiment, portable handheld device 240 includes a mobile computer and display 242. In an example embodiment, display 242 is a touch screen display. In an example embodiment, portable handheld device 240 also functions as a portable cellular telephone. In this regard, device 240 may be, by way of example only and not limitation, a smartphone, as the term is commonly used. That is, in an example embodiment, portable handheld device 240 includes a smartphone, again as the term is commonly used.

[0031] The term "mobile computer" is intended to be a device capable of human-computer interaction, where it is expected that the computer will be transported away from a fixed location during normal use. Likewise, in the example embodiment, the portable handheld device 240 is a smart phone, as the term is commonly used. However, in other embodiments, less complex (or more complex) mobile computing devices may be utilized to implement the teachings and / or variations thereof detailed herein. In at least some embodiments, any device, system, and / or method capable of implementing the teachings and / or variations thereof detailed herein may be used. (As will be described in detail below, in some cases, the device 240 is not a mobile computer, but rather a remote device (remote from the hearing prosthesis 100). Some of these embodiments will be described below).

[0032] In an example embodiment, the portable handheld device 240 is configured to receive data from the hearing prosthesis and present an interface display from among a plurality of different interface displays on the display based on the received data. Example embodiments will sometimes be described in terms of data received from the hearing prosthesis 100. However, it should be noted that any disclosure that also applies to data sent from the handheld device 240 to the hearing prosthesis is also included in this disclosure unless otherwise stated or incompatible with related art (or vice versa).

[0033] Note that in some embodiments, system 210 is configured such that cochlear implant 100 and portable device 240 have a relationship to each other. By way of example only and not limitation, in an example embodiment, the relationship is the ability of device 240 to function as a remote microphone for prosthesis 100 via wireless link 230. Thus, device 240 may be a remote microphone. That is, in an alternative embodiment, device 240 is a standalone recording / sound capture device. In another example embodiment, device 240 adds processing power and / or software to the overall system. As will be described in detail below, embodiments include word generation and / or sound prediction and / or word identification functionality that may reside on prosthesis 100 and / or device 240. Device 240 may work in conjunction with prosthesis 100 to achieve the objectives of this document.

[0034] Note that in at least some example embodiments, device 240 corresponds to the Apple Watch, which is commercially available in the United States as of June 6, 2018. TM Series 1 or Series 2. In the example embodiment, device 240 corresponds to a Samsung Galaxy Gear 3, available in the United States as of June 6, 2018. TM Gear 2. The device is programmed and configured to communicate with the prosthesis and / or function to implement the teachings detailed herein.

[0035] In an example embodiment, the telecommunications infrastructure may communicate with the hearing prosthesis 100 and / or the device 240. By way of example only and not limitation, a telecoil 249 or some other communication system (Bluetooth, etc.) is used to communicate with the prosthesis and / or the remote device. Figure 2B An example quasi-functional schematic diagram is depicted that depicts communication between an external communication system 249 (eg, a telecoil) and the hearing prosthesis 100 and / or the handheld device 240 via links 277 and 279, respectively (note that Figure 2B Bidirectional communication is depicted between the hearing prosthesis 100 and the external audio source 249 and between the handheld device and the external audio source 249 - in alternative embodiments, the communication is only one-way (eg, from the external audio source 249 to the corresponding device).

[0036] Briefly, in an exemplary embodiment, as described below, cochlear implant 100 and / or device 240 is used to capture the speech / sound of a recipient and / or a person speaking to the recipient. In addition, as described below, implant 100 and / or device 240 can be used to record data that can be used to improve the word prediction / word identification / sound prediction functions described below.

[0037] As will be described in detail below, element 249 can represent the portal of communication with the remote server etc. that can realize some teachings of this paper.Specifically, only as an example and not limitation, via the above-mentioned link with element 249, the voice of capture and / or the data associated therewith and by system 210 or only by the data recorded by implant 100 can be transmitted to the remote system that can be used for realizing some teachings of this paper.For example, element 249 can make it possible to communicate with the system processing suite (comprising recommendation engine etc., as will be described in detail below) described in detail below, and device 240 can be in the input, output and / or input / output suite of system.

[0038] By way of example and not limitation, some embodiments relate to intelligent speech recognition to enhance the performance of some hearing prostheses for users. Some embodiments utilize recognition techniques that enable identification of, for example, words being spoken in incoming speech captured by a hearing prosthesis, and the prosthesis to "clean" it before presenting it to the recipient. In some embodiments, this may include presenting the captured speech without noise (or with any practical background noise), presenting the captured speech without an accent or with an accent familiar to the user (e.g., adding an Australian accent to speech captured from an American or British speaker, or removing an Australian accent from speech used to evoke an American hearing, or adding a British accent, or at least British language features, for non-native English speakers, since many such people learn "British English" rather than "American English", etc.), and / or presenting speech to the recipient at a given volume relative to other volumes, and / or adjusting other parameters in any practical way that may improve speech recognition performance or have practical value in speech identification performance (which may be to make the listening task less strenuous). Some embodiments include translating captured speech from one language to another as needed and / or providing a given sentence or paragraph or phrase in a simpler or more easily understood form.

[0039] Some embodiments include performing speech recognition in real time in a hearing instrument.

[0040] Figure 3 A functional diagram of an example embodiment is shown. Herein, a standard hearing prosthesis is modified in which a standard signal processing path may include a standard sound / speech processor, such as those commercially available in cochlear implants or other types of hearing prostheses. (Note that the teachings herein are applicable to other types of sensory prostheses, such as retinal implants, in which the signal processing path would include, for example, a light processor. Thus, any disclosure herein regarding an auditory prosthesis and / or a sound processor and / or a sound processing path corresponds to disclosure of an alternative embodiment in which the auditory prosthesis is a visual prosthesis, such as a retinal implant, which includes, for example, a light processor, and any disclosure herein regarding a sound processor under a sound processing path corresponds to an alternative embodiment of the light processing path.)

[0041] In this example embodiment, a variable delay device is included in the hearing prosthesis, and the variable delay device is configured to apply a variable delay to the output of the standard signal processing path with respect to the signal flow through the hearing instrument. In an example embodiment, the variable delay device can be a memory unit that stores inputs received from the standard signal processing path and allows these inputs to be retrieved shortly thereafter according to a time frame to be described in detail below. The variable delay can be part of a sound processor and / or signal processor used in the prosthesis, or any system capable of using delays according to at least some example embodiments. A delay circuit can be used. In this example embodiment, the user can control the amount of delay, such as by input to the prosthesis, whether the input corresponds to a time frame or indicates the recipient's final desire, where the prosthesis determines what the delay should be based on the input. As can be seen, the hearing prosthesis is configured to enhance the signal based on input from the signal recognition and analysis block. This will be described in more detail below, but in an example embodiment, it can be a chip or processor or computing device that includes software for performing speech recognition and / or sound recognition, etc. Other details will be described below. In any case, in an example embodiment, a signal recognition and analysis block can be used to determine the amount of delay, and also in accordance with the teachings below, a control signal can be provided to the variable delay block to adjust the delay and / or eliminate the delay. Signal enhancement can correspond to any action described herein on how to modify a signal based on the captured sound or how to replace the signal with another signal, which will be described in more detail below. Digital-to-analog conversion is an optional example, and it is noted that some embodiments herein may be directed to a purely analog system. In fact, a digital storage unit and a microphone and analog-to-digital converter associated therewith are also optional (not shown, but in accordance with the label "digitalization"). The digital storage unit may alternatively be an analog storage unit, and may not exist in some embodiments in an occasional situation. In an example embodiment, the storage unit may be a memory unit or a circuit including a transistor, etc., or a group of chips, etc.

[0042] Continue to refer Figure 3In an embodiment, the output of the signal enhancement (including changing the signal and replacing the signal with a new signal, which will be described in more detail below) is ultimately used to provide an output to the user / recipient. In an example embodiment, this can be an electrical stimulation applied via a cochlear implant electrode array. In another example embodiment, this can be a vibration output output by a bone conduction device, for example. In this regard, the hearing instrument / hearing prosthesis can be an active transcutaneous bone conduction device, a passive transcutaneous bone conduction device, and / or a transcutaneous bone conduction device. In an example embodiment, the output can be a mechanical output from a middle ear implant device, etc., and / or can be a mechanical output that enters a device located inside the cochlea to mechanically stimulate the cochlea. In an example embodiment, the signal output can be provided to a receiver of a conventional hearing aid instead, such as in an ear-type hearing aid or a hearing aid including a speaker located in the ear canal. In addition, as described above, the output can be an output to a retinal implant device. It is critical that, unless otherwise stated, any disclosure herein about the features of a given prosthesis (which has similar features to a different type of prosthesis detailed herein) corresponds to the disclosure of an alternative embodiment of such a prosthesis.

[0043] In an example embodiment, Figure 3 The signal recognition and analysis block may be a word identification and / or word prediction device. In an example embodiment, Figure 3 The signal recognition and analysis block may correspond to a processor or computer chip, or to a computing device configured to identify and / or predict words, and / or may be a component, such as an input and / or output device that is in signal communication with a remote device having the described functionality associated with word recognition and / or word prediction, or may be configured to communicate signals with a remote device having the described functionality associated with word recognition and / or word prediction.

[0044] It should be noted that while many of the embodiments detailed herein involve processing that results in or requires or utilizes processing delays, it should be noted that other embodiments may be implemented without processing delays and / or may be implemented without processing delays in the following circumstances, that is, a statistically average recipient (e.g., a 50th percentile male or female ergonomics born and residing in the United States, Canada, Australia, an EU country, the United Kingdom, France, Germany, Japan, Italy, Australia, New Zealand, or India and aged between 18 and 45 years old - hereinafter, any reference to a statistically average recipient corresponds to a percentile male or female ergonomics born and residing in one of these countries) will note this, and all other conditions are the same. In at least some example embodiments implemented without processing delays, predictions and / or identifications can be based entirely on existing sound streams. In embodiments implemented with delays, by way of example only and not limitation, or in embodiments implemented with at least relatively noticeable delays, predictions can be made based on past and / or future sounds (unless otherwise stated, any disclosure of sounds herein corresponds to disclosure of words and vice versa), so that the predictions are more accurate.

[0045] in short, Figure 4 An example flow chart of an example method (method 400) is shown, which includes method action 410, which includes receiving a signal including voice data. In an example embodiment, Figure 3 A microphone receives a signal. In an example embodiment, the signal is a signal received from a microphone, although the signal may be a different signal based on the signal from the microphone (e.g., as may be the case with respect to preprocessing and / or output of a sound processor, depending on how the teachings herein are implemented, or as regards a remote processing embodiment, where, for example, the hearing prosthesis communicates with a remotely located device and the signal from the microphone is used to generate another signal that is ultimately analyzed or evaluated, although the same signal may be transmitted to a remote component). Additionally, the signal may be received from another device (e.g., a USB port, etc.), where, for example, the speech data is not from live speech, but may be pre-recorded speech, and / or where, for example, speech originates from a remote location and is transmitted electronically to a recipient, such as a television broadcast or radio broadcast, etc., where, for example, the prosthesis is in wired and / or signal communication with an output device that transmits or provides the speech data (e.g., thereby bypassing the microphone). As long as the signal includes speech data, method action 410 covers the signal.

[0046] Method 400 also includes method action 420, which includes: processing the received signal to identify and / or predict one or more words in the speech data. This can be done by any processor configured to do so, such as a processor and / or a computer and / or a computer chip and / or an artificial intelligence device and / or a trained expert system, etc. In an example embodiment, action 420 is performed using a computing device that includes word identification / word recognition software for speech-to-text applications and / or spelling correction applications, etc. (e.g., software used on a smartphone when a person speaks into the smartphone and the smartphone converts the captured speech into text, or a Dragon TM software or any variation thereof). It should be further noted that the method actions disclosed herein may also include utilizing a system that "learns" from the past and / or from user experience, similarly, such as Dragon TM Software system, etc. In addition, as described above, the system may also include word prediction technology. In example embodiments, the device system and / or method for performing method action 420 may be a computing device including software for predicting words (such as software used in a web browser and / or a smart phone, etc.). In at least some example embodiments, any device, system, and / or method capable of implementing word identification and / or word identification and / or word prediction may be used.

[0047] The method 400 also includes a method act 430 , which includes evoking an auditory perception based on the received signal, wherein the evoked auditory perception includes one or more modified words based on the identification and / or prediction of the one or more words.

[0048] In short, it should be noted that in some embodiments, method 400 can be performed entirely in a self-contained hearing prosthesis (e.g., a cochlear implant or any other hearing prosthesis detailed herein). It should also be noted that some embodiments include methods where speech data and features associated with speech are replaced with features associated with light, and the evoked perception is a visual perception that includes one or more modified visuals or images based on identifications and / or predictions, provided that this is achievable in the art.

[0049] Thus, in one example embodiment, processing of method act 420 includes utilizing speech recognition software to identify one or more words.

[0050] Embodiments may include utilizing word prediction techniques, as distinguished from word completion techniques often used in web browsers, smartphones, etc., where, as a user begins typing a word, the browser or device provides words that the user may be attempting to type. Word prediction techniques further refine completion techniques by "predicting" the words that may appear next in the speech. This prediction is based on ongoing speech analysis (via AI or other techniques - embodiments include onboard artificial intelligence devices and systems included in hearing prostheses, and / or embodiments where the artificial intelligence systems and devices are remotely located, which artificial intelligence systems and devices can be used to implement the word recognition and / or word prediction techniques disclosed herein). By identifying and / or analyzing words in the speech, the next word or words can be predicted.

[0051] In some embodiments, the teachings herein can be used to predict with a relatively high level of certainty that the next word will be one of the "n" words. The embodiments of the teachings described in detail herein can respond in different ways. In some embodiments, the words with the highest certainty are selected. In other embodiments, the words are combined so that the prediction level of each frequency window (bin) reflects, for example, the average prediction level. Other embodiments apply a "least common denominator" approach so that, for example, only the overlapping area of ​​the predicted words is considered as the level of the predicted words.

[0052] In some embodiments, due to system delay, the prediction of the next word cannot be performed. In such embodiments, some embodiments, after the next word to be processed by the prosthesis and / or the next 2 or 3 or 4 or 5 or 6 or 7 words or any value or value range therebetween (e.g., 3 to 5, 2 to 6, etc., with an increment of 1), predict one or more words (e.g., 2 or 3 or 4 or 5 or 6 or 7 or 8 or more words or any value or value range therebetween, with an increment of 1) with sufficient accuracy to provide signal processing benefits. In some embodiments, the prediction includes or is limited to "further speech", speech of tonal languages, music, etc. In some embodiments, some predictions are based on the identification of previous words. In some cases, for example, when the prediction of the next or subsequent words is not certain enough, these other predictions still provide noise reduction or other signal processing benefits (e.g., updating / maintaining scene classifiers, displaying past words and possible next words to the recipient through an application, etc.). That is, even if the prediction is not enough to determine, for example, replacing a certain word with a certain other word, the prediction can still be used for other non-understanding-based implementations.

[0053] In some embodiments, speech / word identification / prediction techniques are used to effectively change the auditory perception generated by the voice data relative to the original situation. Therefore, in the example embodiment of method 400, at least one of the following is included: (i) the received signal is modified to produce words that sound different relative to what would be the case without modification, or (ii) a new signal provided to the device that evokes auditory perception is generated, which causes the generation of the following words, which sound different relative to what would be the case when the signal is used by the device. Regarding the first scenario, this can include removing noise from the signal, or increasing or decreasing the amplitude of certain frequencies, smoothing frequencies, increasing sharpness, etc. relative to other frequencies. In the second scenario, a new signal will be generated. As an example only and not a limitation, for example, the received signal includes the phrase "to be or not to be". The part containing "to be or not to be" can be retained in the signal, but a new signal based on the pre-recorded / pre-stored data of the word "not" can be inserted / inserted into the speech processing stream, and then the signal can be restored to the base signal. Some additional features will be described below. In short, it should be noted that the concept of the present invention is that a prosthesis or other system can determine that the word "no" should be between the phrases "yes or" and "yes". Based on such a determination, it can also be determined whether the underlying signal can be modified, or whether it will produce the desired auditory perception of the word "no" in its unmodified form. If it is determined that it will not provide the desired auditory perception, the signal will be replaced by a new signal that is more likely to provide the desired perception. That is, the same operating principles can be used to simply modify the signal to produce words that sound different from what they would otherwise be. The idea is to ensure or increase the likelihood that the auditory perception is the word "no" rather than some other word. This can be better achieved using the word prediction and / or word identification teachings of the present invention.

[0054] Indeed, in some example embodiments, the word recognition system may replace an entire phrase or modify the entire signal to increase the likelihood that the auditory perception is "yes or no." In this regard, the system may have a catalog of known or commonly used phrases that the system may rely on to identify and / or predict words. Furthermore, the act of identifying and / or predicting one or more words in the speech data may include identifying and / or predicting one or more sentences in the speech data, or two or three or four or five or six or seven or eight or nine or 10 or 11 or 12 or 13 or 14 or 15 words, or any value or range of values, in increments of 1, (performed once, rather than, for example, for several minutes, which would be to simply count the number of times the system operates).

[0055] For clarity, in at least some example embodiments, any signal processing scheme or word compensation or word replacement scheme that can improve the auditory perception evoked based on word identification and / or word prediction can be utilized, whether this is noise reduction, gain amplitude modification, or replacing a signal with an entirely new signal that can be artificially generated. In addition, as will be explained below, in some example embodiments, where the recipient has difficulty with certain words, some words can be replaced with other words or other sounds that may be considered "nonsense," except that the word will be a word that is more easily identified by the recipient relative to other words. In this regard, due to the limitations of the auditory prosthesis, the recipient may have difficulty recognizing or identifying certain words, especially when the words are spoken relative to other words. For example, the word "forty" and the word "thirty" sometimes sound the same or difficult to distinguish, depending on the person's understanding of the word. In this regard, the ability of a cochlear implant to evoke auditory perception is limited to certain frequencies, and this limitation may increase the difficulty of distinguishing the two words. Using the word recognition and / or word generation techniques herein, the word "thirty" can be replaced with "3" and "zero" or "ytriht" (thirty spelled backwards), where "forty" is always "forty". The goal is to improve comprehension in this situation. No one except the person using the prosthesis will know the "awkwardness" of auditory perception. But if the modification of the words results in better comprehension, this has practical value.

[0056] As described above, the teachings detailed herein can be used to modify auditory perception in a variety of ways. In this regard, in an example embodiment, a signal is modified to produce words with a different accent relative to what would be the case in the absence of the modification, and / or a new signal provided to a device that evokes an auditory perception is generated that results in the production of words with a different accent relative to what would be the case if the signal was used by the device.

[0057] Likewise, in at least some example embodiments, any approach that has practical value with respect to altering or adjusting auditory perceptions generated based on word recognition and / or word prediction (as opposed to being generated in other ways) may be used.

[0058] In view of the practical value of enhancing or improving the original understanding, in some embodiments, method action 420 of processing the received signal is performed in real time during a conversation between the person whose auditory perception is evoked (the recipient of the hearing prosthesis) and the person generating the speech data (e.g., the person speaking to the recipient).

[0059] According to some embodiments of the teachings detailed herein, speech recognition devices, systems, and methods are utilized by utilizing high processing power and large amounts of memory, and in some embodiments, these devices, systems, and methods are included in a hearing prosthesis, can be located in a device that communicates signals with the hearing prosthesis (e.g., a smartphone, smart watch, or remote computer), or as a non-body-worn component of the hearing prosthesis (e.g., a pocket-sized or belt-mounted signal processor, a concept that has been superseded by behind-the-ear devices as signal processors have become smaller and consume less power over the years - the old large chassis can be used to support larger, more power-hungry processors that are not necessarily compatible with the small chassis of behind-the-ear devices).

[0060] In example embodiments, the teachings herein include performing speech recognition or speech prediction on or related to an entire sentence or at least more than half, 2 / 3, 3 / 4, 85% or 90% (on average (mean, median and / or mode)) of a sentence. In some embodiments, speech recognition may use evaluation / identification of incoming syllables and / or phonemes that are not in random sequence and / or strung together to form words and sentences, some of which are more likely to occur than others. Thus, embodiments may utilize portions of a sentence to provide context for individual words of the sentence. In speech recognition terms, other words in a sentence may increase the likelihood of correctly identifying any particular word. The same effect applies to syllable recognition and word recognition. Embodiments may utilize the fact that syllables are spoken as part of a word to increase the likelihood of correct identification.

[0061] In an example embodiment, the treatment action is associated with a noticeable delay in auditory perception that is noticed by the person whose auditory perception is evoked, relative to what would be the case without the treatment. In contrast, in an example embodiment, the treatment is not associated with a noticeable delay in the hearing prosthesis that is noticed by the person whose auditory perception is evoked. In an example embodiment, the delay is a result of the treatment that performs method action 420 and / or other actions herein. That is, in an alternative embodiment, as described above, the delay is a result of the recipient and / or user or other health care professional intentionally causing the delay in order to improve the effectiveness of the method action of the device used to achieve such delay. Thus, in both cases, there is a correlation between treatment and delay and / or absence of delay.

[0062] In the example embodiment, the delay is a delta compared to what would be the case in the absence of identification and / or prediction. That is, in the example embodiment, this is relative to what would be the case if normal / standard signal processing (e.g., signal processing known in the art, such as for speech processing, without the teachings of the innovations herein regarding speech identification and / or generation) were used to evoke the auditory perception, with all other conditions being the same.

[0063] In an exemplary embodiment, there are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 34, 42, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 58, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 85, 90, 95, 100, 12 0, 140, 160, 180, 200, 225, 250, 250, 275, 300, 350, 400, 450, 500, 550, 600, 700, 800, 900, 1000, 1250, 1500, 1750, 2000, 2250, 2500, 3000, 3500, 4000, 4500, 5000, 5500, 5500, 6000, 7000, 8000, 9000, 10000, or more milliseconds of delay, or any value or range of values ​​therebetween in increments of 0.1 millisecond (e.g., 30.7 to 88.4 milliseconds, 707 to 4444 milliseconds, 804.3 milliseconds, 589.0 milliseconds, etc.).

[0064] The above-mentioned delays that can be introduced have practical value for improving the effectiveness of speech recognition, for example, enabling the remainder of a word or sentence to be spoken before predicting the word or syllable in the sentence. In this regard, in some cases, if a delay is introduced in the sound path, or a delay is used in conjunction with the teachings of this article to achieve or allow for improved sound processing, then the sound processing strategy can perform better than it would have in the absence of the delay / normal processing time.

[0065] This is in contrast to the standard case, where delays are avoided in hearing instruments because delays can interfere with lip synchronization, interrupt the flow of conversation, and distract the user. In fact, modern cochlear implants and hearing aid sound processing strategies and / or other hearing prostheses intentionally avoid introducing noticeable delays between the sound received by the microphone and the sound output by the speaker or electrode array. This is because it is generally believed that if such a delay exceeds tens of milliseconds, the user will be able to perceive the time difference between vision and sound, which can distract or annoy the user. When listening to and observing speech, this is the so-called "lip synchronization" problem - the speech is not synchronized with the speaker's lip movements. This is particularly a problem for hearing-impaired people because they generally rely more on lip reading than ordinary people. Therefore, the art avoids such delays and strives to minimize such delays. On the contrary, the teachings of this article can be applied to situations where the harmful effects of delays can be offset by improvements and understanding or improved auditory experience produced by the detailed teachings of this article.

[0066] Furthermore, there are indeed situations where delays are less harmful, or not harmful at all. This could be when listening to a podcast, the radio, or any type of broadcast speech. In a telephone conversation, it may also be acceptable to introduce a small delay (e.g., less than 1 second) so that multiple syllables can be acquired and / or stored and analyzed to improve the performance of word recognition work. In practice, the disruption caused by delays varies depending on the listening situation, and in any given listening situation, a range of different delays may be used to obtain the most practical speech recognition performance.

[0067] For example, for broadcast video (sound and picture), if the broadcast system knows the amount of delay introduced by the auditory instrument, it can compensate by delaying the broadcast of the picture by the same amount as the delay in the audio processing so that the sound and picture are synchronized. Therefore, the embodiment is extended from being applicable only to the auditory prosthesis sound capture scenario to an overall system that synchronizes the auditory perception evoked based on the teachings detailed in this article with other media. By way of example only and not limitation, there can be a system including a video and audio system, such as a television or video conferencing system, in which a delay is also built into the video to synchronize the auditory perception with the video. The system can communicate with the auditory prosthesis. In at least some example embodiments, if communication is possible between the broadcast device and the auditory instrument, in some cases, it is possible to match the delays of the audio and video data streams to synchronize the two. In other words, the visual data stream can be deliberately delayed to allow the required delay to occur in the audio processing, thereby improving the quality of sound recognition.

[0068] Additionally, the system may have speech recognition and / or prediction features, and in some embodiments, may include sound processing features, wherein the resulting signal is then provided to the hearing prosthesis as if the signal were a normal signal captured during normal use of the prosthesis. In this regard, some embodiments are completely separate from the hearing prosthesis, other than using the hearing prosthesis to ultimately evoke auditory perception.

[0069] A corollary to this is that while the above embodiments are generally directed to integrated sound processing and signal enhancement systems within a hearing prosthesis, signal enhancement may alternatively occur upstream of the standard signal processing path. This may occur in the above systems. In this regard, the signal provided to the hearing prosthesis may be an enhanced signal in which the "words" or sounds provided to the sound processor of the hearing prosthesis have been modified or enhanced.

[0070] Modern televisions may have a variable and / or sometimes manually adjustable delay between the picture and the sound. This is usually to account for the delay in processing digital video images, which is usually longer than the delay in processing digital audio. However, the principle may be the same. In fact, to be clear, some embodiments may be relatively "low-level technology". In some embodiments, a black box may be set between the television and the feed, which may have a memory and / or delay circuit and may delay the signal provided to the television for a specific time, but may also be provided to the auditory prosthesis before the signal is provided to the television. By synchronizing the delay in the prosthesis with the delay in the black box, any harmful effects associated with delays related to sound may be mitigated. In addition, the black box may have processing for identification and / or word prediction, etc. In this example, the output of the black box is output to the prosthesis at the same time as the video signal is output to the television, because the delay associated with such utility value relative to word identification has been used up before the signal reaches the auditory prosthesis. In this regard, the utility value of the delay detailed herein occurs outside the auditory prosthesis / before the signal is provided to the auditory prosthesis. In other words, it is not the operation of the hearing prosthesis that is delayed (beyond the normal range), but rather the operation of the system used with the hearing prosthesis that is delayed.

[0071] Thus, embodiments include introducing a variable delay (up to several seconds in some embodiments) between received sound and the output signal of the hearing instrument, where the amount of delay can be adjusted manually or automatically to accommodate different listening conditions.

[0072] Some embodiments include systems, devices, and methods where the system identifies different sound environments and automatically / based on a predetermined scheme identifies a practical delay time. For example, such identification may include identifying broadcast audio (where a relatively long delay is acceptable), telephone conversations (where a shorter delay is acceptable), face-to-face conversations (where as little delay as possible is often optimal).

[0073] In some embodiments, the identification of these sound environments is performed by the user's own voice. If a voice (not the user) is detected and the user is not speaking, in some embodiments, it is automatically determined that the user is listening to a broadcast signal, and a delay can be set accordingly. If another voice and the user's voice appear in turn, it can be automatically determined that a conversation is likely to be ongoing, and a delay can be set accordingly. In this case, in some embodiments, potential variables are used to determine whether the conversation is conducted by phone or in person. In addition, other detection features can also be used, such as whether a pickup coil or external input is used, which indicates a telephone conversation, so the system can automatically determine whether these items are detected. In other cases, the quality of the telephone signal (e.g., limited bandwidth or wide, or other spectrum characteristics can be evaluated, etc.) can also identify the conversation as a telephone conversation, rather than a face-to-face conversation. In an example embodiment, the user is able to override automatic detection by commands input by the user, etc.

[0074] A corollary of the above is that in some embodiments, the type of conversation may be input into a reputable system or other system via the recipient. In an example embodiment, the prosthesis may be provided with an input system, such as a button, which the recipient may press to indicate the type of conversation. That is, the recipient may affirmatively input information into the prosthesis to indicate the type of conversation. In some embodiments, the input may be a voice input. For example, because the hearing prosthesis includes a microphone and may capture the voice of the recipient, in some embodiments, all the recipient does is state a predetermined preset phrase or the like that may be interpreted by the prosthesis as input information, wherein the prosthesis is configured to recognize that this is input rather than just a statement made in a conversation or the like. In at least some example embodiments, any device, system, and / or method that is capable of providing information about the type of conversation to a hearing prosthesis or other system so that the system can determine the type of delay that should be implemented may be used.

[0075] Figure 5 An example algorithm of an example method (method 500) is shown, method 500 includes method actions 510 and 520, which respectively include performing method actions 410 and 420 detailed above. Method 500 also includes method action 530, which includes evoking an auditory perception based on the received signal, wherein the auditory perception evoked is a different auditory perception relative to what would be the case in the absence of the identification. Herein, in some cases, the method can be implemented by simply utilizing noise cancellation or noise suppression techniques or even beamforming techniques for this. In at least some embodiments, any item that causes the auditory perception to change or be different (relative to the case when there is no identification and / or no generation) can be used.

[0076] refer to Figure 5and method 500, and the embodiments above detailing how the delay of video is longer than the delay of an in-person conversation, an example embodiment includes performing method 500 (or method 400, in some embodiments, or any other method detailed herein), wherein the processing is associated with a noticeable delay in the hearing prosthesis that is noticeable by the person whose auditory perception is evoked, relative to what would be the case if the processing was not performed. The method also includes evaluating the overall context related to the received signal, determining one or more delay parameters that will cause the noticeable delay, and adjusting the processing scheme used by the system so that the noticeable delay falls within the one or more parameters. For example, a more extensive word recognition or word prediction algorithm can be used relative to other algorithms, such as if there will be a longer delay. In addition, more "fail-safe" checks can be run to check whether the prediction / identification is accurate, and if not, another prediction / identification is run. For example, if the predicted or identified word is "run", and the delay time is limited, it may not be completely certain that the word is said in a context where the word would be identified as "fun" instead of "run". However, if there is more time, more context can be provided. In an extreme example, if the conversation is about a person with a broken leg, the method may determine that the word is unlikely to be "run." Of course, this may require more data to determine this, and therefore greater latency. That is, in example embodiments, such as those that rely on artificial intelligence or utilize very powerful processing, the system or method may identify the overall context of the speech and infer that someone is unlikely to be talking about running because of statements that have been made prior to the question being expressed (e.g., I have been unable to leave the house for several days due to illness, followed by a several-minute discussion about the stock market - the system may determine that a sick person is unlikely to say the word "run," etc.).

[0077] In an example embodiment, the one or more parameters include a length of delay, and the length of delay for video-based speech data is longer than the length of delay for real-time speech data from a person speaking to a recipient of a hearing prosthesis for evoking auditory perception.

[0078] Figure 6 An example flow chart of an example method 600 is shown, including a method action 610, which includes performing method 500. Method 600 also includes a method action 620, which includes automatically adjusting a noise control system of a hearing prosthesis that is evoking auditory perception based on the identified and / or predicted one or more words. By way of example only and not limitation, this may be adjusting a noise cancellation system. In an example embodiment, this may be adjusting a beamforming system, etc., so that a microphone of the prosthesis is focused toward the speaker, rather than a more omnidirectional focus that captures more noise.

[0079] In some embodiments, whether and to what extent the operation of the prosthesis is adjusted, and / or the word / signal is modified, or the word / signal is replaced, based on the word identification / prediction technique, can depend on the level of certainty of the word prediction / identification. For example, if the level of certainty is low (which may occur when there is not enough data available to predict the next word, or if there is a variable indicating the percentage probability that the identified word may be incorrect / the word is correct), these techniques may not make many adjustments to the device (if any). In this case, the device should operate according to standard practices, such as implementing noise cancellation according to traditional algorithms, or implementing beamforming according to algorithms that are not based on word identification / prediction. In some embodiments, based on the teachings of this article, the operation of the hearing prosthesis can reflect a combination of standard techniques that are affected / controlled in some parts. For example, word recognition and / or word prediction techniques can be used to achieve a very aggressive level of noise reduction technology with fewer artifacts, such as when the prediction / identification has a high confidence level, which is contrary to the case where the noise reduction system is operating normally. In such an embodiment, the correlation between the predicted word and the result of the aggressive adjustment (rather than the less aggressive adjustment) can enable the use of the result without introducing undesirable human factors. In fact, in this regard, the replacement / substitution / alteration of the signal and / or some other teachings of the present document may be considered to be an extreme form of noise cancellation / reduction, for example, in the sense that if the signal is completely replaced by an artificial signal, the new signal may be noise-free / may be a signal containing only the desired content (which in some embodiments may be just speech).

[0080] In some embodiments, for a system that implements sufficient delay to extend to a syllable, word or word sequence that is being predicted, the accuracy of the prediction can be measured. This can be accomplished by, for example, comparing the predicted syllable, word or word sequence with the actual syllable, word or word sequence that is predicted, both of which are available at any time in a system with such a delay. In addition, for example, for a system that does not have sufficient delay to extend to a syllable, word or word sequence that is being predicted, the system can determine the accuracy of the prediction by monitoring the actual syllable, word or word sequence that is emitted after making the prediction. In some embodiments, in the above two cases, if the correlation between the predicted speech and the actual speech is high, the system will be more likely to replace the actual speech with the predicted speech. In some embodiments, in addition to the accuracy of the prediction measurement, other factors can also be used when deciding when and how to replace the speech with the predicted alternative speech. An example of one of these factors to be considered is the speech perception ability of the listener or recipient of the auditory instrument. In the case of poor speech perception ability of the recipient, the system will increase the possibility of replacing all or part of the syllable, word or word sequence, because the recipient is unlikely to understand the incoming speech without help. Conversely, in situations where the recipient's speech perception is good, the system is less likely to replace all or part of a syllable, word, or sequence of words because the recipient is more likely to understand the incoming speech without system adjustments. Another example of an input that can be considered when determining the likelihood of replacing an actual utterance with a predicted utterance is a "real-time" estimate of how well the user understands a particular incoming speech train. For example, if the user is listening to a podcast, video broadcast, or similar program, this can be estimated by monitoring the number of times the user replays or repeats a particular sequence of the podcast, video broadcast, or similar program. However, if it is detected that the user is talking to another person (e.g., on the phone or in person), an estimate of the user's difficulty in talking can be obtained by the number of times the user asks the speaker to repeat what he said. Another way to determine the degree of difficulty is by monitoring the number of times the user adjusts the hearing instrument controls. If the user adjusts the controls many times, it should be understood that the user is more likely to have difficulty understanding the incoming speech than if he adjusts the hearing aid less often. In some cases, there are other variables that affect the likelihood that the system will replace or not replace a particular syllable, word, or sequence of words, such as the speaker's accent, the level of reverberation in the incoming sound, the spectrum of the incoming sound, and so on. It should also be understood that the likelihood of replacing a particular utterance with a predicted utterance may be dynamic, as some or all of the above and other variables that affect the likelihood may change over time. As these variables change, the likelihood of replacing a syllable, word or sequence or word may also change. The speed at which the likelihood responds to changes in the above variables is another parameter that can be controlled by the system that implements the replacement of syllables, words, or word sequences.Embodiments include devices, systems and / or methods that can detect one or more or all of the above events and evaluate the detected data to make a determination as to whether to implement one or more of the teachings herein, and then do so or instruct another system / control another system to do so.

[0081] Furthermore, in some embodiments, there is an action of automatically adjusting the volume and / or gain of a hearing prosthesis that evokes auditory perception based on the identified and / or predicted word or words. This replaces or compensates for the above-described adjustment / operation.

[0082] In an example embodiment, a signal is modified to produce words that are words that would likely be perceived by the recipient relative to what would be the case if the modification had not occurred, or a new signal provided to the device that is evoking auditory perception is generated that results in the production of words that are different from words that would likely be perceived by the recipient relative to what would be the case if the signal were used by the device.

[0083] Furthermore, in example embodiments, a signal is modified to produce a different meaning than what would be the case if the modification was not made and what the recipient would likely understand, or a new signal provided to the device that is evoking auditory perception is produced that results in the production of a meaning that is different than what would be the case if the signal were used by the device and what would be likely understood by the recipient. This case is a rather "extreme" (in a good way) example of the capabilities of the innovation herein. Here, the recipient gets something that was not originally the case. Note that this is not a word translation, where the understanding would be the same if the person understood the language. This results in a fundamentally different result, which is the highest level of hearing - understanding.

[0084] Consistent with the above teachings, in an example embodiment of the above method, the auditory prosthesis is configured to evoke auditory perception, thereby providing a mixed word to the recipient based on the results of the identification. In this regard, by way of example only and not limitation, if the word identification and / or word prediction system does not have a sufficiently high confidence that the identified and / or predicted word is correct, and / or there are two or more applicable "competing" words as the case may be, a word combination can be created or a mixed word can be provided, wherein the recipient will hear a word that can be interpreted as both (or will not guide the recipient in one way or another based on ergonomic statistics and / or psychoanalytic data), but is understandable in the context. For example, the "north" or "south" in front of Carolina may be a "mouth", such as "Mouth Carolina". The recipient may not know which is which, but at least the recipient will not be misled. In fact, it is entirely possible that it does not matter which one it is. However, this can prevent false information from being provided to the recipient. Alternatively, because the discussion may be about a trip to "Myrtle Beach," the recipient will understand that this refers to South Carolina. That is, an advanced system can be utilized that understands that this should be South Carolina due to the context. This demonstrates the ability to innovate.

[0085] Thus, in example embodiments, in view of the above, a hearing prosthesis (or other portion of a system) may be configured to evaluate a certainty level of an identification and generate an auditory perception based on the evaluation of the certainty level, wherein the generated auditory perception differs depending on the certainty level.

[0086] In an example embodiment of method 500, for example, a signal is modified to produce a sound that is completely different from what it would be in the absence of the modification, and / or a new signal provided to a device for evoking auditory perception is generated that evokes a sound that is completely different from what it would be if the signal were used by the device. In some embodiments, the act of identifying and / or predicting one or more words in the speech data includes: identifying one or more sentences in the speech data, and there is no delay that can be noticed by a person whose auditory perception is evoked. In some embodiments, the act of identifying and / or predicting one or more words in the speech data includes identifying one or more sentences in the speech data, and the delay (if any) that is present is any of the above-mentioned delays.

[0087] In view of the above, it can be seen that in some embodiments, there is a hearing prosthesis that includes a sound capture device and a processing system (as opposed to a hearing prosthesis system, which may include a smartphone or a remote device in signal communication therewith), wherein the hearing prosthesis is configured to convert sounds captured by the system (e.g., sounds captured by the hearing prosthesis), and use the processing system to process the signal based on the captured sound, and evoke auditory perception based on the output of the processing system based on the signal. In this example embodiment, the hearing prosthesis is part of a system that includes sound prediction and / or word identification functionality, such that the device is configured to predict sounds captured by the sound capture device. In this context, it can be part of the hearing prosthesis, or part of another device (e.g., a smartphone) that communicates with the prosthesis. Note that in this embodiment, predictions are made for sounds, not words.

[0088] In one example embodiment, all functionality may be located within or as part of the prosthesis, while in other embodiments, functionality is separate throughout the system. Indeed, in example embodiments, the system is a hearing prosthesis, meaning that the system does not include other components that have this functionality, such as a smartphone or remote processor.

[0089] In example embodiments, the system has a sound prediction function, i.e., a word prediction function, and in some embodiments, it is the hearing prosthesis that has the sound prediction function. In other embodiments, the sound prediction function is located in another device, such as a smartphone, such as a computer that communicates with the hearing prosthesis via a remote server. Furthermore, in example embodiments, the function may be located in the above-mentioned black box that is integrated into the communication system.

[0090] In an example embodiment, the system is configured to evoke auditory perception based on the result of the word prediction function, and the auditory perception evoked is different from what it would be if there were no word prediction function results. This is consistent with the above teachings. In an example embodiment, the sound prediction function is a word prediction function, and the system has such a function, and wherein the word prediction function is the ability of the processing system to predict the words after one or more subsequent words that have not yet been received. As described above, one or more subsequent words can be 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 or 15 subsequent words, or any value between these or a value range of 1 with an increment, assuming that this is allowed in the art. In an example embodiment, the prosthesis is configured to increase the aggressiveness of the noise reduction algorithm based on the prediction of the sound. In an example embodiment, the prosthesis is configured to reduce the aggressiveness of the noise reduction algorithm / increase the regression of the noise reduction algorithm based on the prediction of the sound (as described above, in some embodiments, it can be considered that the system and teachings of this article are used for noise reduction). In an example embodiment, these are done based on the protection of words.

[0091] In short, consistent with the above teachings, in an example embodiment, a system of which a hearing prosthesis is a part (e.g., the prosthesis itself) can be configured to introduce a variable delay between capturing a sound and evoking an auditory perception (whether such delay is automatic / developed by the system or based on input from a user), and the system is configured to utilize a predictive function to predict a word, then compare the predicted word to a subsequently received word, and then adjust the operation of the system, such as adjusting the operation of the hearing prosthesis, to evoke the auditory perception based on the comparison. In addition, the system can be configured to introduce a delay between the time location of capturing a sound and evoking an auditory perception that is an increment of a normal processing delay, and the system is configured to identify a sound environment and vary the delay based on the identified sound environment. In example embodiments, the prosthesis or other parts of the system are configured with a scene classifier, such as the scene classifier disclosed in U.S. Patent Application Publication No. 2017-0359659, entitled "Advanced Scene Classification for Prosthesis" and Von Brasch, published on December 14, 2017, and / or may be a device / system corresponding to the device / system disclosed in the application / the prosthesis and system herein may correspond to a device / system having additional functions as modified herein. These devices and systems may also have their own sound detectors for determining the environment. In example embodiments, self-sound detection is performed according to any one or more of the teachings of U.S. Patent No. 2016 / 0080878, and / or the teachings associated with detecting self-sounds herein are implemented in a manner that triggers the control technology of the application. Therefore, in at least some example embodiments, the prosthesis 100 and / or the device 240 and / or the remote device are configured to or include structures that perform one or more or all of the actions detailed in the patent application. In addition, the embodiments include performing methods corresponding to the execution of one or more method actions detailed in the patent application.

[0092] In example embodiments, own sound detection is performed in accordance with any one or more of the teachings of WO 2015 / 132692, and / or the teachings associated with own sound detection herein are implemented in a manner that triggers the control techniques of the application. Thus, in at least some example embodiments, the prosthesis 100 and / or the apparatus 240 and / or the remote device are configured to or include structure that performs one or more or all of the actions detailed in that patent application. In addition, embodiments include performing methods corresponding to the performance of one or more method actions detailed in that patent application.

[0093] As also described above, the apparatus may use latent variables or other means such as telecoil detectors or the like.

[0094] Some embodiments include speech enhancement algorithms and devices including a component known in the art as a Speech Presence Probability (SPP) estimator. Figure 7 Example embodiments for use with the teachings of this article are provided. In this article, speech enhancement strategies can be improved by obtaining accurate SPPs. When combined with the teachings of this article, any commercially available speech presence probability estimator can be utilized in the embodiments of this article. The SPP estimator can be based on a processor or computer, etc. and / or based on a chip and / or can be based on an artificial intelligence system that is part of a hearing prosthesis or part of an independent component of the entire system (the hearing prosthesis is part of the system). In an example embodiment, the next one or more words or sentences, etc. are predicted to enhance speech. For example, an implementation includes, relative to what would otherwise be the case if no prediction was made, using predictions to enhance the estimation of the SPP. The predicted words / sounds can be used to generate a separate SPP and / or modify an existing SPP, which can be combined with the traditional SPP of the speech enhancement algorithm, provided that this can enhance its operation.

[0095] Figure 7 A block diagram of an example embodiment utilizing the teachings just detailed is provided. Figure 7 , block 700 may correspond to or be similar to Figure 3 Block 720 may correspond to or be similar to Figure 3 Of course, there are other ways to accomplish this, and any device, system, and / or method capable of implementing the teachings detailed herein may be utilized in at least some example embodiments.

[0096] In some embodiments, the above embodiments emphasize the "prediction" of the next word or sentence, or the "prediction" of the next word or sentence after the next word, etc. In the case of a delay, this may be counterintuitive because the next word or sentence has been spoken and is known, and no "prediction" is needed. However, in reality, the prediction of an ideal or perfect word (for example, no interfering noise or spoken with a clear accent) may not be what is actually received. Therefore, in some embodiments, if the ideal word or sentence is correctly predicted, the characteristics of the word can be superimposed on the actual speech to any practical extent.

[0097] Therefore, in an example embodiment, the system includes a speech presence probability estimator, which is implemented in a processor of the hearing prosthesis and / or remotely from the prosthesis, such as in a smartphone or a remote device, based on software or an artificial intelligence component (chip, processor) or a trained expert system, etc. In addition, the system is configured to enhance the functionality of the speech presence probability estimator with a sound prediction function (e.g., a word prediction function, which includes a sentence prediction function and a phrase prediction function).

[0098] Figure 8 Another example flow chart of an example method (method 800) is shown. Method 800 includes method action 810, which includes: receiving data containing speech and identifying one or more words therein. Method 800 also includes method action 820, which includes: associating the identified words with previously identified words (i.e., the words received just before the identified words, if any). Method action 830 is also part of method 800, and includes: predicting the next one or more words based on the one or more words identified. Method 800 also includes method action 840, which includes: enhancing noise reduction or other signal processing techniques based on the predicted next word or words. Repeat method actions 810 to 840 as long as it is desired to use method 800. In this regard, in example embodiments, the systems and devices and prostheses detailed herein can be configured to automatically engage and / or disengage prediction / identification functions and control features / signal manipulation / signal manufacturing associated therewith.

[0099] Embodiments include hearing prosthesis sound processor components (BTE devices / off-the-ear (OTE) devices) that apply intelligence to assist speech recognition / speech prediction. This is not simply classification of incoming sounds - this is an attempt to truly understand the concept of incoming speech. Embodiments can utilize basic features of current words and / or sound predictions and / or word identification in behind-the-ear or in-the-ear hearing instruments and / or OTE instruments.

[0100] In some embodiments, once one or more words or sentences are recognized, the hearing instrument has full control over how the sentence is delivered to the listener. It can be presented in a noise-free, easy-to-understand manner, provided that such a manner is practical.

[0101] In fact, embodiments may include providing an auditory perception of speech based on a fully synthesized signal. This is not simply completely eliminating noise. This is creating a completely new "sound" relative to the captured sound that just contains speech. In at least some embodiments, this is achieved using the teachings detailed herein.

[0102] The teachings detailed herein may have practical value for improving speech perception in people who currently have poor speech comprehension abilities. This may be because they are likely to respond best to simplified sound presentations - something that can be achieved if the meaning of the original speech is understood, but is difficult to do without this knowledge. A significant proportion of cochlear implant recipients fall into this category, for reasons that are currently unclear. The teachings detailed herein may be used to help these people.

[0103] Modern cell phones and communication applications such as Skype use real-time speech coding technology to track pitch and formants - relatively slowly changing parameters in speech. It then reconstructs the speech at the receiving end. An embodiment may utilize this technology to implement the features described above, but it should be noted that this does not attempt to recognize spoken words.

[0104] At least some example embodiments utilize the length of acceptable delay (or tolerable delay) in certain listening situations to improve speech understanding, and in some cases, particularly when using cochlear implant devices, the improvement can be significant.

[0105] In an exemplary embodiment, all other things being equal, utilizing the word prediction and / or word recognition and / or sound prediction techniques detailed herein can improve hearing in a standardized hearing test for cochlear implant recipients (which test was deemed of practical use by Cochlear Limited as of April 26, 2019) by at least 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 175, 200, 250, 300, 350, 400, 450, 500, 600, 700, 800, 900 or 1000% or more, or any value or range of values ​​therebetween, by 1%, all other things being equal.

[0106] It should be noted that in the case where speech has been recognized, in some cases, perception can be gradually improved. In the case where the user's speech perception is very poor, for example, for some cochlear implant recipients, it may be practical to provide the user with an output in a clear voice that is almost completely different from the original speaker's voice. And the teachings detailed in this article can achieve this. In some embodiments, the hearing instrument can provide some clues (such as pitch) obtained from the original speech input, and reconstruct the speech from, for example, an existing sound library that matches the user's needs and speech recognition capabilities. For users with good speech perception, most of the quality of the original speech can be retained and transmitted to the user. The knowledge obtained from speech recognition can be used to "clean up" the speech, such as by removing background noise in the signal or making consonants clearer or making formant transitions (distinguishing vowels) more obvious. Then, the resulting speech output can be based on the original speaker's speech features with or without modification, for example, provided by the built-in intelligence of the software running on the hearing instrument, or provided by a remote device. The degree of modification will match the user's needs and can vary according to the noise level or the difficulties that the user may encounter when recognizing speech. In addition, there are embodiments that assess the likely intelligibility of input speech to a user and change the processing strategy to best suit that level of intelligibility. In some embodiments, this may include changing the degree or proportion of the original speech delivered to the user compared to the degree or proportion of computer-generated speech that is known to be intelligible to the user.

[0107] Embodiments include reducing ambient sounds to just speech. Embodiments also include providing fully synthesized auditory perceptions that bear little, if any, resemblance to actual speech in the surrounding environment. By way of example only and not limitation, some people's speech frequencies or speech patterns or special speaking characteristics may make it difficult for cochlear implant recipients to hear. Embodiments include developing a data set for use with an auditory prosthesis that evokes an auditory perception that bears little, if any, resemblance to the speaker's voice, provided that the auditory perception provides or enables the recipient to better understand speech. Therefore, at least some example embodiments provide an auditory prosthesis that provides a single function of transmitting speech to a recipient at a most basic level or a more basic level relative to the situation when there is no teaching. In some embodiments, at least during the period of time during which the device is used accordingly, the prosthesis is simplified to being just a communication device. Although the above embodiments sometimes focus on developing new signals completely, it should be noted that this is not necessarily the case in all cases, and that intelligent sound and speech recognition can be used in various ways for signal enhancement. Some of these methods are used to synthesize new speech completely, while in other embodiments they are used to modify a given signal. Of course, the final result may not be significantly different for the recipient. Simply stated, according to the teaching detailed herein, both ways or any other ways are covered, provided that this can achieve the objectives and teachings of this invention.

[0108] Embodiments include customizing system operation according to specific needs and / or specific personnel. From the foregoing, it can be readily appreciated that the teachings of the innovative aspects of this document can be combined with existing sound stream components and methods to achieve what a recipient will hear. The sound and / or speech recognizer can be configured to output a computer-generated version of the sound it identifies.

[0109] Embodiments may include determining a given recipient's level of performance relative to a hearing prosthesis. For poor performers, the teachings detailed herein may be used in a very aggressive form so as to focus almost, if not entirely, on speech comprehension. For better performers, a gentler approach may be taken, with more subtle changes, modifications, or replications relative to the resulting auditory perception. Indeed, in some embodiments, the teachings detailed herein may be combined with an onboard training system and / or data recorder and / or data analysis device that can determine the proficiency of a hearing prosthesis recipient in at least a general manner sufficient to affect the operation of the systems herein. Alternatively, the devices, systems, and methods herein may be set up by a health care professional, etc., and adjusted based on the health care professional's analysis of the proficiency of a given recipient.

[0110] Embodiments may balance increased latency with significant disruption to the user experience. This may be accomplished by allowing the recipient some and / or full control over the latency. For example, embodiments may include an arrangement where the system implements a latency selected by the system, the recipient provides input as to whether the latency is acceptable or unacceptable, and the system may iterate or modify the latency until a balance is reached.

[0111] In some example embodiments, the processing delay is always at least greater than 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 55, 60, or 70 milliseconds, or any value or range of values ​​therebetween, in increments of 1 ms. These are, of course, some embodiments, not all.

[0112] Fig. 9 A functional schematic diagram of a system is shown, with which some of the teachings and / or variations thereof detailed herein may be implemented. In this regard, Fig. 9 1 is a schematic diagram showing an example arrangement in which a system 1206 may be used to perform one or more or all of the method actions detailed herein in connection with the use of the hearing prosthesis 100. The system 1206 will be described at least in part with respect to interaction with a recipient. In an example embodiment, the system 1206 is a recipient-controlled system. In an example embodiment, the system 1206 may correspond to the remote device 240, which, as described above, may be a portable handheld device, and / or may be a personal computer, etc. It should also be noted that the system 106 may correspond to any system detailed herein, or any processing system herein.

[0113] In an example embodiment, system 1206 may be a system having additional functionality according to the method actions detailed herein. Fig. 9 In the illustrated embodiment, the hearing prosthesis 100 can be connected to the system 1206 to establish a data communication link 1208 between the hearing prosthesis 100 and the system 1206. Thereafter, the system 1206 is bidirectionally coupled to the hearing prosthesis 100 via the data communication link 1208. In at least some embodiments, any communication link capable of implementing the teachings detailed herein, i.e., communicatively coupling the implant and the system, can be used.

[0114] The system 1206 may include a system controller 1212 and a user interface 1214. The controller 1212 may be any type of device capable of executing instructions, such as a general or special purpose computer, a handheld computer (e.g., a personal digital assistant (PDA)), a digital electronic circuit, an integrated circuit, a specially designed application specific integrated circuit (ASIC), firmware, software, and / or a combination thereof (note that any functional component / function detailed herein may be implemented in any of these devices if such devices are capable of implementing the teachings herein, such as by providing them with a program stored in an on-board memory), and if the controller 1212 is provided with a program and hardware and / or firmware, or provided with utility circuits, the controller 1212 may perform some or all of the method actions herein. As will be described in detail below, in an example embodiment, the controller 1212 is a processor. The controller 1212 may also include an interface for establishing a data communication link 1208 with the hearing prosthesis 100. In embodiments where the controller 1212 includes a computer, the interface may be internal or external to the computer, for example. For example, in an example embodiment, the controller 1206 and the cochlear implant may each include a USB, Firewire, Bluetooth, Wi-Fi, or other communication interface through which a data communication link 1208 may be established. The controller 1212 may also include a storage device for storing information. The storage device may be, for example, a volatile or non-volatile storage device, such as a random access memory, a solid-state storage device, a magnetic storage device, a holographic storage device, etc.

[0115] The user interface 1214 may include a display 1222 and an input interface 1224 (which may be the same in the case of a touch screen of a portable device). The display 1222 may be, for example, any type of display device, such as a display device commonly used in computer systems. In an example embodiment, element 1222 corresponds to a device configured to visually display a plurality of words (including sentences) to the recipient 1202, as described above.

[0116] The system of page 9 may be used in embodiments where not all functions performed are performed by the prosthesis. In some embodiments, the system 1206 may be a smartphone.

[0117] Input interface 1224 can be any type of interface capable of receiving information from a recipient, such as a computer keyboard, a mouse, voice response software, a touch screen (e.g., integrated with display 1222), a microphone (e.g., optionally coupled with voice recognition software, etc.), a retinal controller, a joystick, and any other data input or data presentation format developed now or later. Input interface 1224 can be a touch screen of a smartphone. Note that in an example embodiment, display 1222 and input interface 1224 can be the same component (e.g., in the case of a touch screen). In an example embodiment, input interface 1224 is a device configured to receive input from a recipient, the input indicating a selection of one or more of a plurality of words presented by display 1222.

[0118] Note that in at least some example embodiments, system 1206 is configured to perform one or more or all of the method acts detailed herein, wherein various subcomponents of system 1206 are utilized in a conventional manner with respect to a given method act detailed herein.

[0119] In an example embodiment, the system 1206 described in detail above may automatically perform one or more or all of the actions described in detail herein and / or variations thereof, at least those actions that do not require action by the recipient.

[0120] In this case, it should be noted again Fig. 9 The schematic diagram is functional. In some embodiments, system 1206 is a self-contained device (e.g., a laptop computer, a smart phone, etc.) configured to perform one or more or all of the method actions and / or variations thereof described in detail herein. In an alternative embodiment, system 1206 is a system with components located in different geographical locations. By way of example only and not limitation, user interface 1214 may be located at a recipient (e.g., it may be a portable handheld device 240), and system controller (e.g., processor) 1212 may be located at a location away from the recipient. By way of example only and not limitation, system controller 1212 may communicate with user interface 1214 via the Internet and / or via cellular communication technology, thereby communicating with portable handheld device 240. In fact, in at least some embodiments, system controller 1212 may also communicate with user interface 1214 via the Internet and / or via cellular communication, etc. Again, in an example embodiment, user interface 1214 may be a portable communication device, such as, by way of example only and not limitation, a cellular phone and / or a so-called smart phone. In fact, user interface 1214 may be used as a part of a laptop computer, etc. Any arrangement capable of implementing system 1206 and / or capable of implementing a system that enables the teachings detailed herein and / or variations thereof may be used, at least in some embodiments.

[0121] In view of the above, Fig.10 An example functional schematic diagram is depicted in which a remote device 240 communicates with a geographically remote device / facility 10001 via a link 2230, which may be an Internet link. The geographically remote device / facility 10001 may include a controller 1212, and the remote device 240 may include a user interface 1214. Likewise, it can be seen that a direct link 2999 may exist between the prosthesis 100 and the remote device 10001.

[0122] Accordingly, example embodiments entail performing some or all of the method actions detailed herein, wherein the recipient of the hearing prosthesis, the hearing prosthesis 100, and / or the portable handheld device 240 is remote (e.g., geographically remote) from the location where at least some of the method actions detailed herein are performed. These method actions performed remotely may be any of the word prediction / identification / sound prediction functions described above, etc. (A smartphone or other additional device may perform / be configured to perform one or more of these methods / have these functions.)

[0123] Note that in describing the various teachings herein, various actions and / or capabilities have been attributed to various elements of the system 210. In this regard, any disclosure herein associated with a given function or capability of the hearing prosthesis 100 also corresponds to disclosure of a remote device 240 (e.g., a portable handheld device) having the given function or capability, provided that the art is capable of implementing the given function or capability, and / or corresponds to disclosure of a geographically remote facility 10001 having the given function or capability, provided that the art is capable of implementing the given function or capability. By inference, any disclosure herein associated with a given function or capability of the remote device 240 also corresponds to disclosure of the hearing prosthesis 100 having the given function or capability, provided that the art is capable of implementing the function or capability, and / or corresponds to disclosure of a geographically remote facility 10001 having the given function or capability, provided that the art is also capable of implementing the function or capability. As described above, the system 210 may include the hearing prosthesis 100, the remote device 240, and the geographically remote device 1000.

[0124] Note that any method described in detail herein also corresponds to the disclosure of the device and / or system configured to perform one or more or all method actions associated with the content described in detail herein. In an example embodiment, the device and / or system is configured to perform one or more or all method actions in an automated manner. That is, in an alternative embodiment, the device and / or system is configured to perform one or more or all method actions after being prompted by a person. It should also be noted that any disclosure of the device and / or system described in detail herein corresponds to a method for manufacturing and / or using the device and / or system, including a method for using the device according to the function described in detail herein.

[0125] Note that the embodiments include non-transitory computer-readable media having recorded thereon a computer program for performing one or more or any method actions detailed herein. In fact, in an example embodiment, there is a non-transitory computer-readable medium having recorded thereon a computer program for performing at least a portion of any method actions detailed herein.

[0126] In alternative embodiments, any action disclosed herein as being performed by the prosthesis 100 may be performed by the device 240 and / or a remote system, unless otherwise indicated or unless the art does not allow for this. Thus, in alternative embodiments, any functionality of the prosthesis 100 may reside in the device 240 and / or a remote system. Thus, any disclosure regarding functionality of the prosthesis 100 corresponds to a structure of the device 240 and / or a remote system that is configured to perform that functionality or have functionality or perform that method action.

[0127] In alternative embodiments, any actions disclosed herein as being performed by device 240 may be performed by prosthesis 100 and / or a remote system, unless otherwise noted or unless the art does not allow for this. Thus, in alternative embodiments, any functionality of device 240 may reside in prosthesis 100 and / or a remote system. Thus, any disclosure regarding functionality of device 240 corresponds to the structure of prosthesis 100 and / or a remote system that is configured to perform that functionality or have functionality or perform that method action.

[0128] In alternative embodiments, any actions disclosed herein as being performed by a remote system may be performed by the device 240 and / or the prosthesis 100, unless otherwise specified or unless the art does not allow for this. Thus, as an alternative embodiment, any functionality of the remote system may reside in the device 240 and / or the prosthesis 100. Thus, any disclosure regarding functionality of the remote system corresponds to a structure of the device 240 and / or the prosthesis 100 that is configured to perform that functionality or have functionality or perform that method action.

[0129] In an example embodiment, the processing component detailed herein may include an expert subsystem that includes factual domain knowledge and experience that will implement the word prediction / sound prediction / word identification detailed herein, and the processing component may include or be configured to provide output in the form of any action and / or activity detailed above based on the input. In this regard, the system may be an artificial intelligence system. Sufficient training and validation data ("big data") is provided to the system, and an expert subsystem is included in the overall system. The expert subsystem can incorporate factual domain knowledge and / or experience. This allows the system to require less data relative to the situation where there is no expert to achieve a comparable output, all other things being equal.

[0130] The expert subsystem portion of the system may be a system that can be trained to be updated when needed, where a human expert can determine that an update is practical. Thus, the subsystem is a system that is configured to be trained, for example, by incorporating new rules, algorithms, and modifications, optimizing heuristics and algorithms based on input data to maximize performance. The expert subsystem is configured so that once an update has been verified, the expert subsystem of the system is updated. During operation for the user, the input data is processed and then passed to the rules and algorithms that form part of the system's recommendation engine. In some embodiments, the expert subsystem is used for prediction and identification as detailed above.

[0131] In an example embodiment, the trained and / or partially trained subsystem provides output in the form of identification / generation. In some embodiments, as part of the training process, this information is presented to a professional for review and feedback. The recommendations can then be modified based on the feedback. Data about the user or sound experience is then collected to improve performance / continue training.

[0132] Thus, from the above it can be seen that in an example embodiment, the processing suite is an expert subsystem of the system, which includes code for and / or code from a machine learning algorithm for analyzing a metric, and wherein the machine learning algorithm is a training system based on statistically significant population training.

[0133] According to example embodiments, an example machine learning algorithm may be a DNN algorithm. In at least some example embodiments, the input to the system may be processed by a DNN (or code generated by / from a DNN).

[0134] A "neural network" may serve as a machine learning system. Any disclosure herein of such a "neural network" constitutes disclosure of such a "machine learning system". Although the embodiments herein focus on types of neural networks, it should be noted that other embodiments may utilize other types of machine learning systems. Therefore, any disclosure herein of a neural network constitutes disclosure of any other type of machine learning system that is capable of implementing the teachings detailed herein and variations thereof. Specifically, at least some embodiments according to the teachings detailed herein are embodiments that have learning capabilities without explicit programming. Therefore, with respect to some embodiments, any disclosure herein of a device, a system constitutes disclosure of a device and / or system that has learning capabilities without explicit programming, and any disclosure of a method constitutes an action that enables learning without explicit programming therefor.

[0135] Thus, embodiments include utilizing code of a machine learning algorithm and / or code from a machine learning algorithm to analyze data obtained / input into a system to develop data that can be used to implement applicable teachings herein. Likewise, in example embodiments, the machine learning algorithm can be a DNN, and the code can correspond to a trained DNN and / or can be code from a DNN (more on this below).

[0136] Note that any method action disclosed herein corresponds to the disclosure of a non-transitory computer-readable medium having a program code for performing such method action, provided that the art is capable of implementing such method action. In addition, where the art allows, any method action disclosed herein corresponds to the disclosure of code from a machine learning algorithm and / or code for a machine learning algorithm for performing such algorithm. In this regard, the code may correspond to a trained neural network. That is, as will be described in detail below, a large amount (e.g., a statistically significant amount) of data corresponding to the input of the system and the output of the system (linked to the input) can be "fed" to the neural network and trained so that the system can develop the output using only the input (after the system is trained). The neural network used to complete the latter task is a "trained neural network". That is, in an alternative embodiment, a trained neural network can be used to provide (or extract from) an algorithm that can be used separately from a trainable neural network. In one embodiment, there is a training path that constitutes a machine learning algorithm that starts with no training, and then the machine learning algorithm is trained and "graduated", or matures into usable code - the code of the trained machine learning algorithm. With respect to another path, the code from the trained machine learning algorithm is a "descendant" of the trained machine learning algorithm (or some variant thereof, or a predecessor thereof), which may be considered a mutant offspring or clone thereof. That is, with respect to this second path, in at least some example embodiments, the features of the machine learning algorithm that enable it to learn may not be utilized in the practice of some method actions, and therefore not present in the final system. Instead, only the results of the learning are used.

[0137] In an exemplary embodiment, the code from the machine learning algorithm and / or the code of the machine learning algorithm utilizes non-heuristic processing to develop data that the system can utilize. In this regard, the system for performing one or more method actions detailed herein takes in data and extracts basic signals from it, and uses the basic signals to inform itself. By way of example only and not limitation, the system utilizes algorithms other than first-order linear algorithms and "observes" more than one extracted feature. Instead, the algorithm "pays attention to" multiple features. In addition, the algorithm utilizes a higher-order nonlinear statistical model that is able to learn by itself which features in the input are important for study. As described above, in an exemplary embodiment, DNN is utilized to achieve this. In fact, in an exemplary embodiment, as a basis for implementing the teachings detailed herein, there is an underlying assumption that the features of the sound and / or other inputs in the system that produces hearing loss are too complex to be specified, and DNN is utilized in a way that does not know what the algorithm is based on to make its predictions / in what aspects the algorithm expects to develop its predictions. Further, in an exemplary embodiment, the output is a prediction of an auditory map.

[0138] In at least some example embodiments, the DNN is the result code used to make predictions. During the training phase, a number of training operation algorithms are used, which are removed once the DNN is trained.

[0139] In short, it should be noted that in at least some embodiments, the neural network or other machine learning algorithm used herein does not utilize correlations, or in some embodiments, does not utilize simple correlations, but rather develops relationships. In this regard, the learning model is based on utilizing potential relationships that may not be obvious or even identifiable in the larger scheme of things. In example embodiments, neural networks are developed using MatLAB, Buildo, etc. In at least some of the example embodiments detailed herein, the resulting training system does not focus on specific sound features, but is based on the overall relationships present in the underlying significant samples (e.g., statistically significant samples) provided to the system during the learning process. The system itself can calculate these relationships, and there are no known correlations based on features associated with the relationships calculated by the system.

[0140] The end result is a code that is agnostic to the input features. That is, the code of and / or from the trained neural network is such that one cannot identify what input features the code utilized to develop the product (the output of the system). The final arrangement is a complex arrangement of an unknown number of sound features used to predict the recommendations of the system. The code is written in the language of neural networks and is understood by those of ordinary skill in the art to be different from code that utilizes specific and known features. That is, in the example embodiment, the code looks like a neural network.

[0141] Consistent with a common neural network, there is a hidden layer, and the features of the hidden layer are utilized in the process to predict the hearing impairment of the subject.

[0142] However, it should be noted that in at least some example embodiments, one or more of the method actions detailed herein are not performed using a neural network or some form of machine learning algorithm or code based on a neural network. However, in at least some example embodiments, some method actions are performed by using code written in a neural network language.

[0143] It should be further noted that in at least some example embodiments, data used to train the machine learning systems detailed herein may be developed using a significant population (e.g., a statistically significant population) of people having hearing aids and / or cochlear implants, etc.

[0144] Thus, in view of the above, it should be understood that in at least some embodiments, the code of the machine learning algorithm is a trained neural network.

[0145] Artificial intelligence components and / or DNNs, etc. may be embedded on / in a chip, which may be used in / as part of the system in this article.

[0146] The computer system may be implemented as a personal computer, laptop computer, workstation, handheld computer, or a dedicated device specifically designed to implement some of the teachings herein. It is contemplated that some or all of the sound analysis functionality may be implemented in a wearable computer and / or integrated with a sound capture device, or provided in a device such as a dictaphone, cellular telephone, voice recorder, MP3 recorder / player, Apple Computer Inc.'s iPod, or similar device.

[0147] In at least some example embodiments, as described above, there is a non-transitory computer-readable medium corresponding to a software product. The logical operations of the components described herein may be implemented as (1) a sequence of actions implemented by a microprocessor or a program module running on a microprocessor, and / or (2) machine logic circuits or circuit modules interconnected within a computing device. Implementation is a matter of choice depending on the performance requirements of a particular application. Therefore, the logical operations described herein may be variously referred to as operations, routines, structural devices, actions, or modules. Although embodiments are sometimes discussed as being implemented as software, those skilled in the art will recognize that these operations, structural devices, actions, and modules may be implemented with software, firmware, dedicated digital logic, and any combination thereof.

[0148] Some embodiments include a distributed architecture for implementing some of the features taught herein. In this implementation, client processes involving capturing sound and presenting feedback to the user are provided by client components, while signal processing and analysis are provided by network coupling services. Resources for implementing certain functions can be shared by multiple clients. When computationally intensive processes are offloaded to signal processing services, clients can be implemented with relatively low-cost, lightweight components. The network can include a local area network (LAN), a wide area network (WAN), a public network such as the Internet, or other network implementations.

[0149] It should be further noted that any disclosure of a device and / or system detailed herein also corresponds to a disclosure of providing the device and / or system.

[0150] It should also be noted that any disclosure of any process for manufacturing or providing an apparatus herein corresponds to the device and / or system produced thereby. It should also be noted that any disclosure of any device and / or system herein corresponds to the disclosure of a method for producing or providing or manufacturing such device and / or system.

[0151] Any embodiment or any feature disclosed herein may be combined with any one or more or other embodiments and / or other features disclosed herein, unless explicitly stated and / or unless the art does not allow this. Any embodiment or any feature disclosed herein may be expressly excluded from use with any one or more other embodiments and / or other features disclosed herein, unless explicitly stated that this is a combination and / or unless the art does not allow this exclusion.

[0152] Although various embodiments of the present invention have been described above, it should be understood that these embodiments are given by way of example only and are not restrictive. It will be apparent to those skilled in the relevant art that various changes may be made in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A system for evoking auditory perception, comprising: A controller configured to execute a method, the method comprising: receiving a signal including voice data; processing the received signal to identify and / or predict one or more words in the speech data; and evoking an auditory perception based on the received signal, wherein the evoked auditory perception includes one or more modified words based on the identification and / or prediction of the one or more words, The processing introduces a variable delay between receiving the signal and evoking the auditory perception to accommodate different listening conditions.

2. The system of claim 1, wherein: The processing includes identifying the one or more words using speech recognition software.

3. The system of claim 1, wherein: The signal is modified to produce a word that sounds different than it would have without the modification; and / or A new signal provided to the device for evoking the auditory perception is generated, the new signal causing the generated words to sound different relative to the situation where the signal is used by the device.

4. The system of claim 3, wherein: The signal is modified to produce a word with a different accent than would have occurred without the modification; and / or A new signal provided to the device for evoking the auditory perception is generated, the new signal causing the generated words to have a different accent relative to the situation where the signal is used by the device.

5. The system of claim 1, wherein: The act of identifying and / or predicting one or more words in the speech data includes identifying and / or predicting one or more sentences in the speech data.

6. The system of claim 1, wherein: The auditory perception evoked includes fully synthesized words based on the identification and / or prediction of the one or more words, replacing the words in the absence of the identification and / or prediction.

7. The system of claim 1, wherein: The auditory perception evoked includes only speech that is different from speech in the absence of the identification and / or prediction.

8. A system for evoking auditory perception, comprising: A controller configured to execute a method, the method comprising: receiving a signal including voice data; processing the received signal to identify and / or predict one or more words in the speech data; and evoking an auditory perception based on the received signal, wherein the evoked auditory perception is a different auditory perception relative to a situation without the identification and / or prediction, The processing introduces a variable delay between receiving the signal and evoking the auditory perception to accommodate different listening conditions.

9. The system of claim 8, wherein the method further comprises: Based on the one or more words identified and / or predicted, a noise control system of a hearing prosthesis that is evoking the auditory perception is automatically adjusted.

10. The system of claim 8, wherein the method further comprises: Based on the one or more words identified and / or predicted, a volume of a hearing prosthesis that is evoking the auditory perception is automatically adjusted.

11. The system of claim 8, wherein: The signal is modified to produce a word that is different from the word that would be perceived by the recipient in the absence of the modification; and / or A new signal provided to the device for evoking the auditory perception is generated, the new signal causing the generation of words that are different from the words that would be perceived by the recipient if the signal were used by the device.

12. The system of claim 8, wherein: The signal is modified to produce a meaning that is different from the meaning that would be understood by the recipient without the modification; and / or A new signal provided to the device that evokes the auditory perception is generated, the new signal generating a meaning different from the meaning that would be understood by the recipient if the signal were used by the device.

13. The system of claim 8, wherein: The signal is modified to produce a sound that is radically different than it would have been without the modification; or A new signal provided to the device for evoking the auditory perception is generated, the new signal resulting in the evocation of a sound that is completely different than the situation in which the signal is used by the device.

14. The system of claim 8, wherein the method further comprises: evaluating an overall context associated with said received signal; determining one or more parameters for a delay that will result in the variable delay; as well as A processing scheme used by the process is adjusted so that the variable delay falls within the one or more parameters.

15. The system of claim 14, wherein: The one or more parameters include a delay length; and The delay length is longer for video-based speech data than for real-time speech data from a person speaking to a recipient of a hearing prosthesis used to evoke the auditory perception.

16. The system according to claim 8: The auditory prosthesis is configured to evoke auditory perception such that a blended word is provided to a recipient based on a result of the identification.

17. The system according to claim 8: Wherein the hearing prosthesis is configured to assess a level of certainty of the identification and to generate an auditory perception based on the assessment of the level of certainty, wherein the generated auditory perception differs depending on the level of certainty.

18. The system of claim 8, wherein: The auditory perception evoked is a fully synthetic auditory perception that maximizes understanding of speech at the expense of pitch, distinctiveness, and / or emphasis.

19. A system for evoking auditory perception, comprising: A hearing prosthesis comprising a sound capture device and a processor, wherein the hearing prosthesis is configured to convert sounds captured by the system, and to process signals based on the captured sounds using the processor, and to evoke auditory perception based on output from the processor based on the signals, wherein the system comprising sound prediction and / or word identification functionality, such that the system is configured to predict sounds captured by the sound capture device, The system is configured to introduce a variable delay between sound capture and evocation of the auditory perception to accommodate different listening conditions.

20. The system of claim 19, wherein: Wherein the system has sound prediction functionality, the sound prediction functionality is word prediction functionality.

21. The system of claim 20, wherein: The system is configured to evoke an auditory perception based on an outcome of the word prediction functionality, the auditory perception evoked being different from a situation in which the outcome is absent.

22. The system of claim 19, wherein: The sound prediction functionality is word prediction functionality, and the system has word prediction functionality, and wherein the word prediction functionality is the ability to predict a word following one or more subsequent words that have not yet been received by the processor.

23. The system of claim 19, wherein: The prosthesis is configured to increase the aggressiveness of the noise reduction algorithm based on the prediction of the sound.

24. The system of claim 19, wherein: The system includes a speech presence probability estimator; and The system is configured to use the sound prediction functionality to enhance functionality of the speech presence probability estimator.

25. The system of claim 19, wherein: The system is configured to predict words using the sound prediction functionality, then compare the predicted words to subsequently received words, and then adjust the operation of the system to evoke auditory perception based on the comparison.

26. The system of claim 19, wherein: The system is configured to introduce the variable delay between the temporal location of the sound capture and the evocation of the auditory perception, the variable delay being an increment of a normal processing delay; and The system is configured to identify a sound environment and to vary the variable delay based on the identified sound environment.

Images