Method and system for measuring and tracking ear characteristics

Abstract

The present disclosure relates to methods and systems for measuring and tracking ear characteristics. A method performed by a headset comprising a speaker and an in-ear microphone is provided, the method comprising: performing a calibration of the headset to obtain a baseline measurement; while the headset is being worn by a user, driving the speaker with an audio signal, the speaker being arranged to project sound into an ear canal of an ear of the user; capturing, from the in-ear microphone of the headset, sound from within the ear canal of the ear of the user as a microphone signal; determining a parameter associated with the ear of the user based on at least the captured microphone signal and the baseline measurement; and transmitting a notification relating to one or more characteristics of one or more auditory elements of the ear of the user based on the parameter.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 248,353, filed September 24, 2021, which is incorporated herein by reference. Technical Field

[0002] One aspect of this disclosure relates to a system for measuring and tracking ear characteristics of one or more auditory elements of a user's ear. Other aspects are also described. Background Technology

[0003] Audiologists use specialized equipment to identify, diagnose, and treat ear problems. For example, doctors can use tympanic impedance testing devices to test middle ear activity caused by changes in air pressure within the user's ear canal. These devices include a probe with an amplifier, microphone, and air pump. As the probe is inserted into the user's ear, the microphone measures the sound produced by the amplifier, while the air pump adjusts the air pressure within the user's ear. The input at the microphone is used to create a tympanic impedance map, which doctors can use to determine various ear problems. Summary of the Invention

[0004] One aspect of this disclosure is a method performed by headphones (such as a pair of in-ear headphones (e.g., earbuds), over-ear headphones, or earbuds) arranged for wear by a user (e.g., worn on the user's head) and including a speaker and an in-ear (or error) microphone. The headphones perform calibration to obtain baseline measurements. For example, calibration may be performed while the headphones are in a controlled environment (e.g., before the headphones are worn by the user), such as a case arranged to house the headphones when not worn by the user. While the headphones are being worn by the user, the headphones drive the speaker using an audio signal (e.g., which may include audio content desired by the user, such as music, and / or may include one or more test sounds), the speaker being arranged to project sound into the ear canal of the user's ear. The headphones capture sound from within the ear canal of the user's ear as a microphone signal from the in-ear microphone of the headphones. The headphones determine parameters associated with the user's ear based at least on the captured microphone signal and the baseline measurements. The headset transmits notifications related to one or more characteristics of the user's auditory elements based on this parameter.

[0005] In one aspect, the headphones use the microphone signal to determine a secondary path (or S-path) transfer function representing the response between the speaker and the in-ear microphone (when both components are inserted into the user's ear canal), wherein the parameter may be based on the S-path transfer function. In some aspects, the transfer function may be determined as part of an active noise cancellation (ANC) process performed by the headphones to generate an anti-noise signal for driving the speaker. However, in another aspect, the headphones may not be configured to perform the ANC process (e.g., may not have ANC functionality), in which case the S-path transfer function may not be determined as part of the ANC process (or may be determined to be separate from the ANC process).

[0006] In one aspect, the parameter is determined by measuring the acoustic input impedance of the user's ear canal relative to a baseline measurement using the S-path transfer function and using that acoustic impedance to determine a parameter that is part of an equivalent acoustic-mechanical model of the auditory elements of the user's ear (e.g., at least the user's middle and outer ear). In another aspect, the headset compares the parameter to a corresponding parameter of a predefined equivalent acoustic-mechanical model and identifies the one or more characteristics based on the difference between the parameter and the corresponding parameter of the predefined model. In some aspects, the acoustic input impedance is measured over a period of time, and the headset determines the one or more characteristics based on the change between the parameter and the corresponding parameter over that period of time. In another aspect, the parameter may be a reflection mask of the user's ear canal determined based on the acoustic impedance. In this case, the headset may use the reflection mask to filter one or more microphone signals to produce one or more filtered signals, which are used to determine the severity level of the characteristic.

[0007] In some respects, performing the calibration involves determining one or more impedance calibration parameters for the impedance model, wherein measuring the acoustic input impedance involves estimating the impedance by applying the determined one or more impedance calibration parameters and the S-transfer function to the impedance model. In other respects, the one or more impedance calibration parameters are predefined parameters defined offline in a controlled environment.

[0008] On the other hand, determining the parameter associated with the user's ear includes using the S-path transfer function to measure the acoustic input impedance of the user's ear canal relative to the baseline measurement as the parameter, and determining the one or more characteristics of the auditory elements of the user's ear based on the parameter includes using the acoustic impedance to determine whether there is a blockage in the ear canal of the user's ear and to determine the severity level of the blockage.

[0009] In one aspect, the audio signal includes audio content desired by the user. In another aspect, the audio signal includes (e.g., predefined) test sounds. In some aspects, the calibration is performed on the headphones before using the audio signal to drive the speaker and before capturing the sound from within the user's ear canal. In one aspect, the calibration is performed on the headphones before they are being worn by the user. In some aspects, the calibration is performed on the headphones while they are in a controlled environment. In one aspect, the calibration is performed while the headphones are in a case arranged to contain them when they are not being worn by the user. In another aspect, a machine learning model with input based on the parameter is used to determine one or more of the characteristics.

[0010] According to another aspect of this disclosure, a method performed by the headphones may use one or more microphone signals to determine the characteristics of auditory elements of the user's ear. For example, while the headphones are being worn by the user, the headphones may drive the speaker to project a first sound into the ear canal of the user's ear and receive a first set of microphone signals from a set of in-ear microphones, the first set of microphone signals including the first sound captured by the microphones (e.g., as a direct sound from the speaker) and reflections of the first sound (e.g., reflected from the user's ear canal, eardrum, etc.). The headphones generate at least one reflection parameter of the ear canal of the user's ear based on the first set of microphone signals. The headphones drive the speaker to project a second sound into the ear canal of the user's ear and receive a second set of microphone signals including the second sound and reflections of the second sound. The headphones generate a set of output signals based on the second set of microphone signals and the at least one reflection parameter, wherein the output signals include only the reflection of the second sound. The headphones transmit notifications related to the characteristics of auditory elements of the user's ear based on the set of output signals.

[0011] In one aspect, generating the at least one reflection parameter includes determining, based on a comparison of the set of first microphone signals, whether the spectral content of a frequency band across a plurality of frequency bands of the first set of microphone signals primarily includes the first sound as direct sound from the speaker or includes reflections of the first sound, wherein the at least one reflection parameter is generated in response to the spectral content including the reflections of the first sound. In another aspect, determining whether the spectral content primarily includes the first sound as direct sound from the speaker or includes the reflections of the first sound includes determining, within a certain time period, whether the spectral content across the frequency bands of the first set of microphone signals includes a threshold number of reflections of the first sound.

[0012] In some aspects, each of the at least one reflection parameter is associated with a corresponding frequency band among the plurality of frequency bands, wherein generating the set of output signals includes using the at least one reflection parameter to filter the second sound from the second set of microphone signals by removing spectral content across one or more frequency bands not associated with the at least one reflection parameter. In another aspect, the headphones use the first set of microphone signals to generate a directional beam pattern directed towards the user's ear canal and away from the headphones, wherein the spectral content has the directional beam pattern.

[0013] In one aspect, the headphones determine whether the first sound exceeds a confidence threshold based on the audio content of the first sound; and in response to the first sound exceeding the confidence threshold, transmit a notification related to the characteristic of the auditory element. In some aspects, determining whether the first sound exceeds the confidence threshold includes determining whether the audio content includes at least one of attenuation and pauses.

[0014] In some respects, the characteristic is the blockage within the ear canal of the user's ear, and the ear geometry is the geometry of the ear canal. The headphones use this set of output signals to estimate the geometry of the blockage within the ear canal, and the severity level is based on one or more differences between the geometry of the ear canal and the geometry of the blockage within the ear canal.

[0015] The above overview does not constitute an exhaustive list of all aspects of this disclosure. It is contemplated that this disclosure encompasses all systems and methods that can be practiced by all suitable combinations of the aspects outlined above and those disclosed in the detailed embodiments below and specifically pointed out in the claims. Such combinations may have specific advantages not specifically set forth in the foregoing summary. Attached Figure Description

[0016] Multiple aspects are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals in the drawings indicate similar elements. It should be noted that references to "a" or "an" aspect in this disclosure do not necessarily refer to the same aspect, and each refers to at least one. Furthermore, for the sake of brevity and to reduce the total number of drawings, a single drawing may be used to illustrate features of more than one aspect, and for a particular aspect, not all elements in that drawing may be necessary.

[0017] Figure 1 A system including an output device is shown according to one aspect.

[0018] Figure 2 A block diagram of an output device is shown, which is inserted into a user's ear according to one aspect and configured to determine one or more characteristics of the user's ear.

[0019] Figure 3This is a flowchart of one aspect of the process used to determine the characteristics of a user's ear.

[0020] Figure 4 Another block diagram shows an output device configured to determine the characteristics of a user's ear based on one or more reflection parameters of the ear canal from one or more microphone signals.

[0021] Figure 5 This is a flowchart of one aspect of the process for determining one or more characteristics of a user's ear based on one or more estimated reflection parameters of the user's ear canal.

[0022] Figure 6 Another block diagram shows an output device configured to determine the characteristics of a user's ear based on acoustic input impedance measurements, according to one aspect. Detailed Implementation

[0023] Various aspects of this disclosure will now be explained with reference to the accompanying drawings. Unless the shape, relative position, and other aspects of the components described in any aspect are explicitly defined, the scope of this disclosure is not limited to the components shown, which are for illustrative purposes only. Furthermore, while numerous details have been set forth, it should be understood that some embodiments may be implemented without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of the description. Moreover, unless the meaning is explicitly contrary, all scopes shown herein are to be considered to include the endpoints of each scope.

[0024] Figure 1 A system 1, comprising a source device 2 and an output device 3, is illustrated according to one aspect. In one aspect, the system may include other devices, such as a remote electronic server (not shown), which may be communicatively coupled to the source device, an audio device, or both, and may be configured to perform one or more operations as described herein. In one aspect, the source device 2 and / or the output device 3 may perform at least some of the operations described herein. In another aspect, the system may include fewer devices, such as having only the output device 3. In this case, the output device may perform at least some of the operations described herein.

[0025] In one aspect, source device 2 can be any electronic device (e.g., an electronic device including one or more processors, memory, etc.) capable of performing audio signal processing operations and / or networking operations. Examples of source devices may include desktop computers, smart speakers, home theater systems, or infotainment systems integrated into vehicles. In some aspects, source device can be wireless electronic devices, such as tablet computers, smartphones, etc. In other aspects, source device can be wearable devices, such as smartwatches, and / or head-mounted devices (HMDs), such as smart glasses, etc.

[0026] As shown in the figure, output device 3 is an earphone (e.g., a pair of in-ear headphones or earbuds) positioned adjacent to the user's ear 5 (against or inside the user's ear). Specifically, the device is shown as a right earbud, which is against (or positioned on) the user's right ear and / or at least partially inserted into the ear canal of the user's ear. In one aspect, the earphone may include another earbud positioned against the user's left ear (e.g., a left earbud). In this case, the output device may perform one or more audio signal processing operations in at least one (or both) of the earbuds. When one earbud performs at least some of the operations described herein, the earbud may generate one or more drive signals and transmit at least one drive signal to the other earbud, such that both earbuds can drive one or more corresponding speakers. In another aspect, the two earbuds may perform one or more (e.g., similar) operations to generate respective drive signals.

[0027] In one respect, once the headphones (or in-ear headphones) are placed against a user's ear, they acoustically isolate the user's ear (ear canal) from the surrounding environment, thereby preventing (or reducing) sound leakage into (and out of) the user's ear (and / or ear canal). Specifically, the headphone shell may be placed on, around, and / or inside the user's ear to prevent (or reduce) sound leakage into (and out of) the user's ear (ear canal). For example, when the headphones are in-ear headphones, they may be sealed headphones, wherein the headphone shell includes a portion (such as an ear tip for each of the user's ears), wherein each ear tip is arranged to form a seal (or isolation) within the user's ear canal when inserted into one of the user's ears.

[0028] In some aspects, the output device 3 can be any type of headphone, such as over-ear or full-ear headphones. In the case of full-ear headphones, the output device can be part of the headphone housing arranged to cover at least a portion of the user's ear to (at least partially) isolate the ear (and ear canal) from the surrounding environment. In other aspects, the output device can be any type of electronic device (such as a hearing aid) arranged to be positioned on (or in) the user's ear for sound output. In some aspects, the output device can be any HMD (or part of any HMD), such as smart glasses. For example, the audio output device can be part of a component (e.g., frame) of smart glasses that can be placed in or on the user's ear. In other aspects, the output device can be an HMD that (at least partially) does not cover the user's ear (or ear canal), thus exposing the user's ear to the surrounding environment.

[0029] In one aspect, the output device 3 can be any electronic device configured to output sound, perform networking operations, and / or perform audio signal processing operations, as described herein. For example, the output device can be configured to receive one or more audio signals (e.g., from a source device) and can be configured to use one or more audio signals to drive one or more speakers to output sound directed toward the user's ears and / or ear canals. Further details regarding driving speakers are described herein. In one aspect, the audio output device can be a device not designed to be positioned close to (or against) the user's ears. For example, the output device can be (e.g., a standalone) megaphone, a smart speaker, part of a home theater system, part of an infotainment system integrated into a vehicle, etc. In some aspects, the output device can be part of another electronic device such as a laptop computer, desktop computer, or multimedia device. In another aspect, the output device and the source device can be the same device.

[0030] In one aspect, output device 3 may be configured to wirelessly couple to (or communicatively couple to) one or more other electronic devices (e.g., source device 2), such that both devices are configured to communicate with each other. For example, output device 3 is communicatively coupled to source device via wireless connection 4. For example, a wireless output device (such as wireless headphones or a pair of wireless handsets) may be connected to source device via a Wireless Personal Area Network (WPAN) connection to receive audio streams from audio source device. In one aspect, the WPAN connection may be connected via an Advanced Audio Distribution Profile (A2DP) connection or another audio profile connection using the Bluetooth communication protocol. To stream high-quality audio data, source device packages audio data according to the audio profile (e.g., divides audio data into multiple units for transmission) and stores the packets of audio data in a transmission buffer. The packets are then transmitted to wireless output device via Bluetooth (e.g., using air (or wireless) radio frequency (RF) signals). The received packets are stored in a long buffer in output device to provide continuous audio playback in the event of future packet loss (e.g., during transmission interference). The audio data in the buffer is unpacked and processed for audio output through at least one speaker. This process is repeated when the audio output device needs audio output.

[0031] On the other hand, the wireless connection 4 between the output device and the source device may be via one or more other wireless computer networks. For example, the output device may be communicatively coupled to the source device via any network such as a wireless local area network (WLAN), a wide area network (WAN), a cellular network, etc., to exchange digital (e.g., audio) data. Regarding the cellular network, the output device may be configured to establish a wireless (e.g., cellular) call, wherein the cellular network may include one or more cellular towers, which may be part of a communication network (e.g., a 4G Long Term Evolution (LTE) network) that supports data transmission (and / or voice calls) of electronic devices such as mobile devices (e.g., smartphones).

[0032] On the other hand, output device 3 can be communicatively coupled to source device 2 via other methods. For example, both devices can be coupled via a wired connection. In this case, one end of the wired connection can be (e.g., fixedly) connected to the output device, while the other end can have a connector, such as a media jack or a Universal Serial Bus (USB) connector, that inserts into a socket on the source device. Once connected, the output device can be configured to drive one or more speakers of the output device via the wired connection using one or more audio signals. For example, the source device can transmit the audio signal as digital audio (e.g., PCM digital audio). On the other hand, the audio can be transmitted in an analog format.

[0033] In some respects, output device 3 and source device 2 may be different (separate) electronic devices, as illustrated herein. In other respects, the output device may be part of (or integrated with) the source device. For example, as described herein, at least some components of the output device (such as a controller) may be part of the source device, and / or at least some components of the source may be part of the output device. In this case, each of these devices may be communicatively coupled via traces that are part of one or more printed circuit boards (PCBs) within the device.

[0034] Figure 2 A block diagram is shown of an output device 3, inserted into a user's ear 5 according to one aspect and configured to determine one or more characteristics of one or more auditory elements of the user's ear. Specifically, as shown, at least a portion of the output device is inserted into the ear canal 6 of the user's ear 5. As described herein, when inserted into the ear, the output device may form a seal within the ear canal to prevent air movement (e.g., leakage) between the ear canal 6 and the surrounding environment (e.g., the outside of the ear canal). For example, the output device may be an earplug with an ear tip (e.g., the ear tip coupled to the earplug's housing), arranged to form a seal within the ear canal (e.g., by contact with the inner wall of the ear canal), as described herein. On the other hand, as described herein, the output device 3 may be placed on the user's ear 5, such as over-ear headphones or earmuff headphones.

[0035] As described herein, the output device is configured to determine and transmit (e.g., output) one or more characteristics of an auditory element of a user's ear. Specifically, the output device may perform operations to determine (detect or identify) characteristics of auditory elements (e.g., problems, concerns, difficulties, etc.) that may adversely affect a user's ability to hear sound. In one aspect, the detected characteristic may be a specific type of problem (e.g., clinical impairment) of an auditory element of the user's ear. However, in another aspect, the characteristic described herein may be a general indication of a difficulty or problem of one or more auditory elements, rather than a specific clinically defined problem. As used herein, an auditory element may include portions of the user's middle ear (e.g., the tympanic cavity), such as the malleus, incus, and / or stapes, and / or portions of the user's inner ear (e.g., the cochlea). As another example, an auditory element may include the outer ear (e.g., the auditory canal or ear canal) or the tympanic membrane (or eardrum). In another aspect, an auditory element may represent (or any) physical part of the user's ear.

[0036] In one respect, a characteristic of an auditory element can be a naturally occurring consequence or a result of external forces (e.g., causing impairment or reduction in the (audible) performance of one or more auditory elements). For example, an external force can be an object that interacts with (or comes into contact with) an auditory element, such as an obstruction within the user's ear canal. Because the passage of sound through the user's ear canal is at least partially blocked, an obstruction within the user's ear (e.g., the ear canal) can lead to at least partial hearing loss. Further details regarding obstructions are described herein. In another respect, a characteristic can be an obstacle (or a specific problem) to a particular auditory element.

[0037] Traditionally, when a user experiences hearing (or ear)-related symptoms, they may wait to schedule an appointment with a medical professional (e.g., a doctor) for consultation. For example, once a patient develops hearing and / or ear symptoms (e.g., hearing loss, ear pain, dizziness, headache, etc.), they can contact an audiologist. After scheduling an appointment, the audiologist will perform hearing tests using specialized tools. For instance, to identify the condition of a person's ear, the audiologist might perform a tympanometry test, in which a tympanometry probe is inserted into the person's ear canal, and the probe measures changes in air pressure. Although an audiologist may be able to diagnose the problem, patients may inadvertently worsen their condition by waiting too long, potentially leading to unnecessarily prolonged symptoms.

[0038] To overcome these shortcomings, this disclosure describes an output device configured to detect and track characteristics of one or more auditory elements of a user's ear. Specifically, the output device may be headphones, such as in-ear headphones (e.g., earbuds) including a speaker and an in-ear microphone (or error microphone). When the output device is being worn by a user (e.g., the device is inserted into or onto the user's ear canal), the output device can use audio signals to drive the speaker to project sound into the ear canal and can capture microphone signals including sound from within the ear canal. The output device can determine parameters associated with the user's ear based at least on the captured microphone signals. Based on these parameters, the output device can determine one or more characteristics of the auditory elements of the user's ear. Furthermore, the output device can transmit (present or output) a notification (e.g., audible and / or visual) to the user indicating that a characteristic has been detected. Therefore, this disclosure provides an output device (such as in-ear headphones) that, unlike dedicated medical tools, can notify a user of one or more detected characteristics via the user's ear. In response to the notification, the user can schedule a consultation with a medical professional to receive a medical diagnosis. Therefore, users may be able to receive consultations more quickly than if they are not notified of this feature, which could reduce the amount of time patients have to endure symptoms.

[0039] return Figure 2The output device 3 includes a controller 20, a speaker 22, and an in-ear (or error) microphone 21. In one aspect, the output device may include more (or fewer) elements (or components). For example, the output device may include one or more speakers, microphones, etc. In another aspect, the output device may include other components (not shown), such as a reference microphone, one or more displays, and one or more sensors (e.g., accelerometers, inertial measurement units (IMUs), cameras, etc.).

[0040] Controller 20 may be a dedicated processor such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller is configured to perform audio signal processing operations and / or networking operations. For example, controller 20 may be configured to perform ear characteristic tracking and decision-making operations, as described herein. Alternatively, the controller may be configured to perform other audio signal processing operations. Further details regarding the operations performed by controller 20 are described herein.

[0041] The in-ear (or error) microphone 21 can be any type of microphone (e.g., a differential pressure gradient microelectromechanical system (MEMS) microphone) configured to convert acoustic energy caused by sound waves propagating in an acoustic environment into an input microphone signal. As described herein, the microphone is an in-ear microphone such that when the output device is worn by a user (e.g., worn on the user's head), the microphone can be positioned (or arranged) to sense sound (e.g., air pressure difference) within (or near) the user's ear canal. As shown in the figure, the microphone is located at one end of the output device inserted into the ear canal 6 of the user's ear. Therefore, the in-ear microphone can be positioned inside or at the entrance of the ear canal 6 to capture sound from inside or near the user's ear.

[0042] The loudspeaker 22 may be, for example, an electrically driven driver specifically designed for sound output in a particular frequency band, such as a woofer, tweeter, or midrange driver. In one aspect, the loudspeaker 22 may be a “full-range” (or “full-frequency”) electrically driven driver that reproduces as much of the audible frequency range as possible. In another aspect, when the output device includes one or more loudspeakers, each of these loudspeakers may be of the same type (e.g., each is a full-range driver) or may be of different types. For example, when the output device includes two loudspeakers, one may be a midrange driver and the other a woofer.

[0043] As shown in the figure, speaker 22 is an "internal" speaker configured to project sound into (or toward) the ear canal of the user's ear when the output device is being used (or worn) by the user. Thus, similar to microphone 21, the speaker may be positioned within (or at the entrance to) the ear canal 6 of the user's ear 5 to project sound inward into the ear canal. Alternatively, the speaker may be an "external" speaker arranged to output sound to the surrounding environment. Similarly, the output device may include one or more "reference" microphones, each arranged to sense sound from the surrounding environment. Alternatively, the output device may include one or more of these speakers / microphones (or some speakers / microphones). For example, the output device may include one or more internal speakers (e.g., speaker 22) and one or more external speakers.

[0044] In one aspect, each component (or at least some components) of the output device 3 may be integrated into the output device (e.g., within the housing of the output device), such that the component is part of the device. In another aspect, one or more components may be part of a separate electronic device communicatively coupled to the output device. For example, one or more reference microphones may be part of a source device 2 configured to capture ambient sound using the reference microphones and configured to transmit microphone signals captured by the microphones to the output device (e.g., for audio signal processing operations).

[0045] As described herein, controller 20 is configured to perform ear characteristic tracking and decision-making operations. Specifically, the controller includes one or more operation blocks to perform one or more of these operations. As shown, the controller includes (e.g., a secondary path or S-path) a transfer function estimator 23, an impedance estimator 24, an impedance calibrator 25 (which is optional, as shown by the dashed boundary), a model parameter estimator 26, an ear characteristic tracking and decision-making system 27, and an impedance calibration parameter 28. In one aspect, the controller may have more or fewer operation blocks. For example, the controller may not include an optional impedance calibrator.

[0046] In one aspect, at least some of the operations performed by one or more operation blocks may occur while the output device (e.g., the controller 20 of the output device) is performing one or more audio signal processing operations. For example, the controller is receiving an input audio signal (e.g., as an analog or digital audio signal) and may be using that audio signal to drive a speaker 22, which is arranged to project the sound of the audio signal into the user's ear canal 6. Therefore, the controller may be driving the speaker while the output device 3 is being worn by the user (e.g., at least partially inserted into the user's ear canal).

[0047] In one aspect, the input audio signal may include media content or audio content desired by the user, such as musical works, podcasts, movie soundtracks, etc. In another aspect, the input audio signal may include predefined test sounds (e.g., a sine sweep), which may have been defined in a controlled environment (e.g., a laboratory). In one aspect, the controller may be receiving (or acquiring) the input audio signal from local memory. In another aspect, the controller may be receiving the audio signal from a remote device, such as source device 2. In this case, the output device may stream (e.g., via wireless connection 4) the input audio signal for playback by the output device.

[0048] In one aspect, the audio signal may include sounds from one or more software applications being executed by an audio output device (e.g., the controller 20 of the audio output device). For example, the audio signal may be a downlink signal that is part of a telephone call being conducted by a telephone application (which is being executed by the output device and / or the source device). In this case, the telephone application may be configured to exchange audio data (e.g., via the source device) with another device with which the output device is making a call via a communication network. In another aspect, the audio signal may include other sounds from other software applications, such as audible alarms associated with a virtual personal assistant application. In one aspect, these applications may be executed by the source device while the sounds of the software applications are being transmitted (via wireless connection 4) to the output device for playback.

[0049] In one aspect, the input audio signal may be a single audio signal (e.g., a mono audio signal). In another aspect, the controller may acquire two or more input audio signals, such as acquiring two audio channels (e.g., a stereo signal) for output through two or more speakers. In this case, the output device may include two or more speakers, such as having two or more internal speakers (e.g., speakers such as speaker 22, which are arranged to project sound into the ear canal of the user's ear when the output device is worn by the user). In yet another aspect, at least some of the audio signals may be used to drive one or more external speakers. Further information is described herein regarding other audio signal processing operations that may be performed by the controller.

[0050] The transfer function estimator 23 is configured to receive an input audio signal that is being used (or will be used) to drive the speaker 22 and a microphone signal including sound from within the ear canal 6 captured by the in-ear microphone 21, and is configured to use the input audio signal and the microphone signal to estimate (measure or determine) a transfer function representing the propagation path (or response) between the speaker 22 and the in-ear microphone. Thus, the transfer function is estimated based on the air pressure measured by the in-ear microphone. On the other hand, the transfer function is estimated by measuring the ear canal impedance based on the pressure at the in-ear microphone and one or more characteristics of the speaker (e.g., the speaker's volume velocity). In one aspect, the response and impulse response used herein are equivalent to each other. In some aspects, the impulse response is the response of the in-ear microphone signal in the time domain. On the other hand, the transfer function is the frequency response of the impulse response, and in the frequency domain.

[0051] In one aspect, the transfer function may be determined as part of an active noise cancellation (ANC) process (or function) performed by the controller 20 of the output device to generate an anti-noise signal for driving the speaker 22. For example, the controller may be configured to perform adaptive feedback ANC to adapt an ANC filter (e.g., W(z)) (as shown in the figure) based on an estimate of the secondary path (or S-path) transfer function (S(z)) representing the propagation (or acoustic) path from the speaker 22 and microphone 21, which the ANC process uses to adjust the adaptive digital filter used to generate the anti-noise signal. In one aspect, the S-path transfer function defines how the microphone output signal responds to pulses input to the speaker. In some aspects, the S-path transfer function may be estimated by fixing the transfer function, or it may be adaptively calculated online via an adaptive filter algorithm while the device (e.g., headphones) is being worn by a user. In one aspect, the S-path transfer function may be estimated while the ANC function is activated (e.g., based on user input received by the output device).

[0052] As described so far, the controller can be configured to perform feedback ANC. On the other hand, in addition to (or instead of) performing feedback operation, the controller can perform feedforward ANC functionality, wherein a reference microphone signal captured by a reference microphone (not shown) arranged to capture ambient sound (and error microphone signals) is used to adapt the ANC filter according to the S-path transfer function in order to generate another noise-resistant signal.

[0053] On the other hand, transfer function estimator 23 can estimate the S-path transfer function separately from (or not determined to be part of) the ANC process. For example, the output device may not be configured to perform the ANC process (e.g., the controller may not be configured to perform the process), but can still be configured to estimate the S-path. On the other hand, the S-path transfer function can be estimated as part of the ANC process, which does not generate noise-resistant signals. For example, the controller may (e.g., only) be performing the ANC process to estimate the transfer function.

[0054] Impedance estimator 24 is configured to receive (e.g., S-path) transfer functions from transfer function estimator 23 and is configured to use the S-path transfer function to measure (or estimate) the acoustic input impedance of ear canal 6. For example, the impedance estimator can use the transfer function to model the acoustic input (e.g., eardrum) impedance (e.g., individual ear canal). Specifically, the impedance estimator can be configured to receive impedance calibration parameters (or coefficients) 28 (e.g., which may be stored in the controller's memory) and apply the parameters and the S-path transfer function to the impedance model (e.g., stored in memory) to generate (or estimate) the acoustic input impedance. Specifically, the coefficients can be used to convert the S-path transfer function into acoustic input impedance (e.g., according to the impedance model). In some aspects, the impedance calibration parameters can define a calibration curve (e.g., which may include a curve of one or more coefficients relative to frequency). In this case, the impedance estimator can use the calibration curve (relative to the S-path transfer function) to estimate the acoustic input impedance of sound entering the user's ear canal. In one respect, impedance calibration parameters (which may be one or more frequency-dependent parameters) can be determined (or measured) via an impedance calibration operation (or measurement) performed offline in a controlled environment (such as during product development or during a power-on cycle).

[0055] In one aspect, impedance calibrator 25 may be configured to perform calibration of the output device to obtain baseline (or calibration) measurement results. For example, the impedance calibrator may be configured to perform an offline impedance calibration operation to determine impedance calibration parameters. In another aspect, stored parameters may be received from another electronic device performing the calibration. In yet another aspect, the impedance calibrator is configured to perform (e.g., online) impedance calibration to address drift (or variability) over time between one or more acoustic components (e.g., between a speaker and / or a microphone). In one aspect, variability may occur due to the natural degradation of components over time, and / or may be based on changes in environmental conditions, such as changes in objects or contaminants collected on or near the components over time (e.g., dust collection in speaker ports). For example, the impedance calibrator may be configured to receive an input audio signal and a microphone signal from an in-ear microphone and may be configured to perform a baseline measurement, which may be an impulse response measurement (e.g., a transfer function) of the output sound of the speaker 22 at the location of the in-ear microphone. In one aspect, the baseline measurement may be performed before (or after) the output device is worn by a user. Specifically, the measurement can be performed when the output device (or at least a portion of the output device including a speaker and microphone) is in a controlled environment, such as in a container designed (or arranged) to house the output device when it is not in use by a user. For example, when the output device is headphones (e.g., a pair of earbuds), the container could be a case designed to house the headphones when they are not worn by a user. Therefore, impedance calibration can be performed inside the case to address drift issues. In one aspect, this impedance calibration may involve estimating the S-path transfer function and determining whether one or more of the impedance calibration parameters 28 have changed (e.g., based on parameters determined offline). For example, the estimated S-path transfer function may be associated with a predefined acoustic input impedance that is associated with the volume of the case. A change between the recently determined acoustic input impedance (obtained in this case) and the predefined impedance may be a result of drift, and therefore, the impedance calibrator can adjust one or more of the parameters 28 based on the drift. On the other hand, in addition to (or instead of) adjusting parameters, impedance calibrators can be configured to use calibration to adjust (e.g., in the future) the estimated transfer function to address variations in acoustic components.

[0056] In one aspect, an impedance calibrator can be configured to periodically acquire (or perform) one or more baseline measurements (e.g., impedance recalibration). For example, the calibrator may perform impedance calibration periodically (e.g., once a day, once a week, once a month, etc.) to maintain optimal impedance estimation. In another aspect, the calibrator may perform measurements when it is determined that the output device is in a controlled environment (e.g., based on sensor data captured by one or more sensors of the output device, indicating that the output device has been placed in a storage box, as described herein). In yet another aspect, calibration measurements may be performed based on user input (e.g., based on user commands received by the output device).

[0057] On the other hand, an impedance calibrator can be configured to perform a baseline measurement while the output device is being used by a user (e.g., worn). Specifically, the impedance calibrator can perform this measurement periodically to track changes in impedance over time (e.g., from when the user first purchased the output device to the present time).

[0058] In one aspect, the impedance estimator may be configured to receive a baseline measurement (e.g., impedance calibration of an impedance calibrator) and may be configured to measure (or estimate) the acoustic input impedance of the user's ear canal using a transfer function and relative to the baseline measurement. For example, the impedance estimator may be configured to compensate for variations in impedance (e.g., based on variance, as described herein) by adjusting (or compensating) the impedance calibration parameter 28 based on the impedance calibration and / or the estimated S-path transfer function. Thus, the impedance estimator can use (e.g., one or more) baseline measurements to address variability when estimating the acoustic input impedance of the user's ear canal.

[0059] In one aspect, one or more auditory elements of the physical ear can be modeled (or estimated) as an equivalent acoustic-mechanical model. For example, the equivalent model could be an RLC circuit (or network) in which auditory elements of the physical ear are modeled as one or more parameters (or components) of the RLC network. Specifically, auditory elements such as the outer ear, tympanic membrane, middle ear, and / or inner ear can be modeled as one or more electrical and / or mechanical components (e.g., resistors, capacitors, inductors, dampers, etc.) of the RLC network. In one aspect, one or more of these four parts of the human ear can be composed of one or more components constituting the model. For example, the ear canal of the outer ear can be modeled as a damper, while the cochlea of ​​the inner ear can be modeled as an inductor. In one aspect, the modeled auditory elements can also include characteristics of one or more of these elements. For example, an RLC circuit can model the characteristics of the stapes in the middle ear (e.g., where the stiffness of the stapes can be represented by a capacitor). In another aspect, the equivalent model can consist of any type of electrical and / or mechanical components.

[0060] Model parameter estimator 26 is configured to receive estimated acoustic input impedance from impedance estimator 24 and is configured to use the acoustic input impedance to determine an equivalent acoustic-mechanical model of the user's ear (e.g., at least a portion of the equivalent acoustic-mechanical model). For example, the acoustic input impedance can be used as input data to perform numerical optimization methods (such as nonlinear least-squares parameter fitting) to determine an equivalent model of the user's ear (e.g., one or more parameters). In one aspect, the model parameter estimator can, for example, use the numerical optimization methods described herein to minimize the difference between the measured acoustic input impedance and the impedance produced by the equivalent acoustic-mechanical model (e.g., whose parameters have been modified). In another aspect, the estimator can apply the acoustic impedance to a parametric model configured to produce one or more parameters based on the impedance. In yet another aspect, the estimator can use the impedance to perform a table lookup in a data structure (e.g., stored in the memory of controller 20) that associates one or more parameters of the equivalent model with one or more measured acoustic impedances. Thus, the estimator can compare the measured impedances within the data structure and select a parameter associated with the measured acoustic impedance that matches the estimated impedance of the user's ear. In one respect, the determined parameters can be characteristics (e.g., values) of components (e.g., resistor values, capacitor values, etc.) within an equivalent model.

[0061] On the other hand, the model parameter estimator 26 can be configured to determine (or estimate) one or more other parameters. For example, the estimator can determine a parameter associated with the acoustic seal created by the output device when it is inserted into (or on) the user's ear. Specifically, this parameter can indicate the effectiveness of the seal (e.g., as a value between 0 and 1) (e.g., in preventing sound leakage).

[0062] Ear characteristic tracking and decision-making system 27 is configured to receive one or more estimated parameters (e.g., model component values) from model estimator 26 and is configured to determine, based on the received parameters, whether one or more auditory elements of a user's ear possess one or more characteristics. Specifically, the system may receive an equivalent acoustic-mechanical model of the user's ear (e.g., a user-ear-specific acoustic-mechanical model) from model estimator 26 and may use this model to determine one or more characteristics of the user's ear. For example, the system may compare the received equivalent model with a predefined equivalent model (which may be stored in memory). In one aspect, the predefined model may be a "normal" model of the human ear, whereby the human ear is a healthy ear (e.g., does not possess characteristics associated with the auditory elements of the human ear (or possesses characteristics below a threshold)). In other words, the parameters of the normal model are associated with some auditory elements that do not possess any (e.g., known) characteristics associated with them. In another aspect, the predefined model may be a general model (e.g., a model averaged across one or more other models). On the other hand, the predefined model can be a (e.g., predetermined) model of the user's ear (e.g., possibly determined once during the first power-on cycle of the output device and again when the device is inserted into (or placed on) the user's ear). The system determines whether there are any differences between the estimated equivalent model and the predefined model in order to identify whether the user's ear has any characteristics or problems. For example, the system can compare one or more specific parameters of the equivalent model with corresponding parameters of the predefined model, and based on the differences between the parameters, it can identify one or more characteristics of one or more auditory elements (e.g., which may be associated with the parameters). For example, as described herein, the parameters of the equivalent model associated with the stiffness of the stapes can be (at least) capacitors. Therefore, the system can compare the capacitance value of the capacitors in the equivalent model with predefined capacitance values ​​from a normal model. If the capacitance value differs from the predefined value (e.g., greater than or less than a predefined threshold), the system can determine that (at least) the user's middle ear (e.g., stapes) may have a problem (as a characteristic).

[0063] On one hand, the system can determine one or more characteristics based on whether the estimated model (or one or more estimated parameters) has drifted over a period of time. Specifically, the system can track the estimated parameters to determine whether the parameters have increased (e.g., at a specific rate) over a period of time. If so, the system can determine that a certain characteristic exists in the auditory element associated with this drift.

[0064] In one aspect, upon determining that one or more auditory elements of a user's ear possess one or more characteristics, the system may perform one or more operations. Specifically, the system may transmit a notification to the user, indicating a potential problem with the user's ear. For example, an output device may use a notification audio signal including the notification to drive speaker 22 to notify the user. In another aspect, the system may present (e.g., pop up) the notification on a display screen (such as the display screen of source device 2) communicatively coupled to the output device. In one aspect, in addition to (or instead of) notifying the user of the detected characteristics, the notification may offer the user a recommendation to visit a medical provider (e.g., an audiologist) who can examine the user's ear to provide a medical diagnosis. In another aspect, the system may transmit the notification to another electronic device that can present (or track) the identified one or more characteristics. In some aspects, the system may transmit the notification to one or more applications being executed by one or more electronic devices (e.g., on the output device) that can track and / or output alerts regarding the characteristics. Thus, the system is able to provide the user with a notification to seek a medical provider so that the user can obtain a diagnosis from the medical provider, which can improve the user's hearing health.

[0065] In one aspect, the system can track the characteristics of identified auditory elements over a period of time and can be configured to provide notifications to the user based on changes in those characteristics. For example, the system can determine whether the characteristic persists during the period of time (e.g., whether the identified characteristic is detected at a threshold level during the period of time), and if so, the system can provide a notification indicating that the characteristic persists.

[0066] In some respects, system 27 may determine and track ear characteristics based on machine learning (ML) algorithms executed by controller 20. For example, the system may use an ML algorithm to determine one or more estimated model parameters as input to the algorithm and one or more characteristics of one or more auditory elements of the user as output. In some respects, the system may use any type of ML algorithm to perform this determination (e.g., deep neural networks (DNNs), etc.).

[0067] On the other hand, to track characteristics, the system can track changes in one or more parameters. For example, a parameter may change based on variations in input impedance over a period of time. As described herein, the determined characteristic may be based on whether the estimated parameter exceeds (or falls below) a predefined parameter threshold of a predefined model. The system can track the difference between the estimated parameter and the predefined parameter over that period of time. For example, the system can track the rate of change of the parameter relative to a predefined parameter (e.g., based on changes in input impedance) over that period of time. When the rate of change is determined to exceed a threshold, the system can notify the user.

[0068] On the other hand, the system can notify the user of the severity level (or value) of a determined characteristic based on the rate of change (and / or based on the difference between one or more estimated parameters and predefined parameters). For example, when a parameter is determined to have a specific rate of change, the system can notify the user of the characteristic and its severity level (e.g., low, medium, high). Specifically, the system can determine whether the determined rate of change is greater than a first (e.g., low) threshold and / or whether the determined rate of change is greater than a second (e.g., high) threshold (i.e., greater than the first threshold). If it is below the first threshold, the severity level can be low; if it is above the first threshold but below the second threshold, the severity level can be medium; and if it is above the second threshold, the severity level can be high. More information regarding severity levels is described herein.

[0069] Figure 3 This is a flowchart of one aspect of a process 30 for determining the characteristics of a user's ear. In one aspect, process 30 may be performed by an output device (e.g., headphones) 3 (e.g., a controller 20). Specifically, at least some of the operations described herein may be performed by... Figure 2 At least some of the operation blocks described herein are executed. In one aspect, process 30 may be performed while the output device is playing back audio content (e.g., by driving speaker 22 using an input audio signal).

[0070] At the start of process 30, controller 20 performs calibration of the headphones to obtain baseline measurements (at box 31). Specifically, the controller may determine one or more impedance calibration parameters of the impedance model. For example, the controller may receive one or more impedance calibration parameters 28, which may have been previously determined by offline impedance calibration (e.g., in a controlled environment), as described herein. Alternatively, the obtained parameters may be based on online impedance calibration, where impedance calibrator 25 performs impedance measurements in the storage case of the output device and the parameters may be determined based on those measurements. While the output device is being worn by the user, controller 20 uses an audio signal to drive a speaker arranged to project sound into the ear canal of the user's ear (at box 32). The controller captures sound from within the ear canal of the user's ear as a microphone signal from the in-ear microphone 21 of the output device (at box 33). The controller determines parameters associated with the user's ear based at least on the captured microphone signal and the baseline measurements (at box 34). Specifically, these parameters may be parameters of an equivalent acoustic-mechanical model equivalent to the user's physical ear. For example, the controller can use the microphone signal to determine a (e.g., S-path) transfer function representing the response between the speaker and the in-ear microphone, wherein the determined parameters are based on the S-path transfer function. Specifically, the controller can use the S-path transfer function to estimate (or measure) the acoustic input impedance of the user's ear canal relative to baseline measurements (e.g., impedance calibration parameter 28). Specifically, the acoustic input impedance can be estimated (as described herein) by applying one or more determined impedance calibration parameters and the S-path transfer function to an impedance model. In one aspect, the estimated impedance can be used to determine one or more parameters associated with the user's ear.

[0071] Controller 20 determines one or more characteristics of auditory elements of the user's ear based on determined parameters (in box 35). Specifically, as described herein, system 27 can compare the determined parameters with predefined parameters and, based on the differences (or variations) between the parameters, identify one or more characteristics of auditory elements of the user's ear (e.g., middle ear, inner ear, etc.). In one aspect, the controller transmits a notification related to one or more characteristics of one or more auditory elements of the user's ear based on the parameters. Specifically, as described herein, the controller determines one or more characteristics based on the parameters and transmits a notification, which may include an indication of the characteristic. As described herein, the notification may be an audible notification and / or a visual notification. In some aspects, the notification may be transmitted when determining one or more characteristics, as described herein.

[0072] Variations exist in the execution of process 30. For example, at least some specific operations in these processes may not be performed in the exact order shown and described. This specific operation may not be performed in a consecutive series of operations, and different specific operations may be performed in different aspects. For example, the controller may perform calibration before the remaining operations of process 30. Specifically, the output device may perform calibration before performing the operations in blocks 32-35. On the other hand, calibration may be performed while the device is in a controlled environment, as described herein. In this case, the controller may use signals (e.g., separate from the input signal used to drive the speaker in block 32) to drive the speaker and capture microphone signals from the in-ear microphone, which includes the sound produced by the speaker. The controller may use these signals to estimate the response at the in-ear microphone as a baseline measurement, which can be used to determine parameters, as described herein.

[0073] On the other hand, controller 20 may determine one or more parameters based on one or more estimated impedances. As described herein, the controller may use an audio signal to drive a speaker and project sound into the ear canal of a user's ear. In one aspect, the sound may be output at a certain loudness level. For example, the loudness level may be a user-defined level and / or a predefined level. In one aspect, the controller may determine multiple impedances, where each impedance is determined when the audio output device (e.g., driving a speaker with the same or different audio signal) is at a different loudness level. For example, the controller may determine a first impedance based on a first audio signal output at a first loudness level, and may determine a second impedance based on a second audio signal input at a second (e.g., a different) loudness level. Thus, the controller may observe changes in the estimated impedance at different loudness levels and may determine one or more parameters based on the impedance changes. Therefore, the controller is configured to measure the acoustic impedance at two or more different sound levels to detect a nonlinear response of the user's ear (e.g., the middle ear and / or inner ear).

[0074] In one aspect, the operations for measuring and tracking ear characteristics can be performed by an output device inserted into (or covering) one or more of a user's ears. Specifically, these operations can be performed by a headphone device configured to perform audio signal processing and / or networking operations to stream audio content for playback to the user. In another aspect, these operations can be performed by headphones that isolate the user's ear (ear canal) from the surrounding environment. For example, the headphones can be in-ear headphones, which include ear tips for forming a seal within at least a portion of the user's ear canal upon insertion. In yet another aspect, these operations are best performed by a sealed device that prevents air leakage into (and out of) the user's ear canal. Thus, the operations for measuring and tracking ear characteristics can be performed by any (e.g., consumer) electronic head-mounted device, rather than by a dedicated measuring instrument.

[0075] In one aspect, controller 20 may be configured to perform one or more audio signal processing operations based on elements coupled to the controller. For example, controller 20 may include a pickup beamformer configured to process audio (or microphone) signals generated by two or more external microphones (and / or two or more in-ear microphones) of an output device to form a directional beam pattern (as one or more audio signals) for spatially selective pickup in a specific direction, thereby increasing sensitivity to the location of one or more sound sources. In some aspects, the controller may perform audio processing operations (e.g., perform spectral shaping) on ​​the audio signal containing the directional beam pattern.

[0076] In one aspect, the controller may optionally perform additional DSP operations. For example, the controller may apply one or more scalar gains to the input audio signal and / or perform spectral shaping of the signal by applying one or more filters (e.g., low-pass filters, band-pass filters, high-pass filters, etc.). In another aspect, the controller may perform spatial rendering operations on one or more of these signals (and / or mixed content) by applying spatial filters (such as head-related transfer functions (HRTF)) to generate binaural audio signals for driving one or more speakers (e.g., left and right speakers), as described herein. As another example, when the output device includes one or more loudspeakers, the controller may render the HOA representation of the audio signal and the downlink signal to generate one or more loudspeaker driver signals (e.g., based on a predefined loudspeaker configuration).

[0077] In one respect, the operations performed by the controller may be implemented in software (e.g., as instructions stored in memory and executed by the controller) and / or may be implemented by hardware logic structures as described herein.

[0078] As described herein, controller 20 can be configured to measure and track ear characteristics of one or more auditory elements of a user's ear. In one aspect, characteristics may include obstruction or blockage within the user's ear canal. This obstruction may be caused by a foreign object or by the natural accumulation of earwax (or earwax) produced by the body to protect the ear. While small obstructions may not be noticeable to a person, large obstructions (e.g., a large buildup of earwax) can adversely affect a person's hearing and lead to other health problems (e.g., ear pain, dizziness, etc.). Detecting and tracking these problems can help a person effectively manage characteristics such as ear canal obstruction (e.g., by notifying the user of the characteristic so that the person can schedule a consultation with a healthcare provider), thereby improving the person's overall ear health.

[0079] This disclosure provides an ear characteristic tracking and decision-making system configured to identify ear characteristics and track these characteristics over a period of time. Specifically, to detect and track characteristics such as obstruction, a controller is configured to generate a set of one or more reflection parameters of the ear canal of a user's ear using one or more microphone signals, the set of one or more microphone signals including sound produced by one or more speakers and reflections of sound (e.g., sound reflected from the ear canal of the user's ear, sound reflected from the eardrum, etc.). The reflection parameters may indicate (or represent) the reflection pattern of the ear canal of the user's ear in response to an audible stimulus. For example, when a speaker output propagates through the ear canal to the eardrum, it produces resonance and reflections along the way. Compared to an ear canal with more (or more severe) obstruction, most of the sound energy can propagate further to the eardrum through an ear canal with less (or almost no) obstruction (or object). Therefore, an ear canal with almost no obstruction may produce a specific reflection pattern captured by one or more microphones, while an ear canal with more (or more severe) obstruction may produce a different reflection pattern. Using reflection parameters, the controller can filter the microphone signal and use the filtered (or output) signal to determine whether a blockage has occurred and / or the severity level of the blockage over a given time period. Therefore, the system can track the severity level of identified characteristics to notify the user of the progression of the characteristic's severity. Understanding the severity of a particular characteristic provides additional information that the user can use to help determine whether to seek advice from a medical provider (e.g., an audiologist).

[0080] Figure 4 Another block diagram of an output device configured to determine characteristics of a user's ear based on reflection parameters from one or more microphone signals is shown. In another aspect, the output device may also be configured to determine the severity of the determined characteristics, as described herein. As shown, the output device includes a controller 20, a speaker 22, and one or more in-ear microphones 21. In one aspect, the output device may include more or fewer elements. For example, the output device may include an array of in-ear microphones comprising two or more in-ear microphones from which the controller 20 may receive signals from two or more (or N) microphones.

[0081] As shown in the figure, controller 20 includes several operational blocks, such as a reflection parameter estimator 42, a reflection filter 45, an ear geometry estimator 41, a microphone calibrator 46, and an ear characteristic tracking and decision-making system (or system) 27. In one aspect, the controller may include more or fewer operational blocks, such as having one or more operational modules, such as a microphone calibrator and an ear geometry estimator (which are shown with dashed boundaries).

[0082] As described herein, the controller can perform one or more audio signal processing operations when the output device is used by a user. Specifically, the output device may be worn by the user (or worn on the user's body) while the controller is performing one or more operations, as described herein. For example, similar to Figure 2 The output device can be headphones, such as earplugs inserted into the ear canal 6 of the user's ear 5 when the device is in operation. Alternatively, while the output device is being worn by the user, the controller may be using the input audio signal to drive the speaker 22. Additionally, N in-ear microphones 21 (where N is the number of in-ear microphones in the output device) may be capturing sound from within the user's ear canal. Therefore, the controller 20 can be configured to acquire N microphone signals (at least one microphone signal per microphone), each microphone signal including the captured sound.

[0083] Reflection parameter estimator 42 is configured to receive N microphone signals captured by an in-ear microphone and to use these signals to estimate (or generate) a set of one or more reflection parameters of the user's ear canal. In one aspect, the reflection parameters may be based on band (e.g., sub-band) analysis of the spectral content within one or more of the microphone signals. Specifically, for each of several bands spanning a frequency range (e.g., the range of frequencies audible to humans), the reflection estimator may compare the spectral content of the microphone signals to determine whether the spectral content includes reflections of sound produced by a speaker. These reflections may be sounds reflected from auditory elements of the user's ear (e.g., sounds reflected from the user's ear canal, sounds reflected from the eardrum, etc.). In another aspect, the estimator is determining whether each band primarily includes sound produced by a speaker (e.g., direct sound from a speaker) or reflections from auditory elements. In one aspect, the estimator determines whether the spectral content across the bands within a certain time period (e.g., one second, one minute, one hour, etc.) includes a threshold number of reflections of sound produced by a speaker. If so, it can be determined that the band primarily includes reflections. Therefore, the spectral content across a frequency band can be determined to primarily consist of reflections based on the number of reflections detected within that band. On the other hand, the spectral content may also include (or at least partially include) direct sound generated by the loudspeaker.

[0084] In one aspect, reflection parameters can be generated by calculating a spatial correlation matrix between one or more microphone signals, wherein a frequency band (or sub-band) between microphone signals having spectral content above a threshold (e.g., an energy level above an energy threshold level) can be determined to indicate reflections in that frequency band. In another aspect, the number of reflections within a frequency band can be identified by tracking the spectral content within the band over a period of time (e.g., tracking the number of times the spectral content within a sub-band exceeds a threshold over a period of time). In another aspect, a large number of reflections (e.g., above a threshold, above an average, etc.) and / or having a high energy level can indicate that the reflection originates from a location outside the ear canal (e.g., reflections caused by at least a portion of sound from an output device inserted into (or placed) in the user's ear canal).

[0085] In one aspect, a reflection parameter can indicate whether at least a portion of the microphone signal will be filtered out (e.g., removed) (e.g., when it is determined whether a certain characteristic of the user's ear is present). For example, the reflection parameter can be a binary value that indicates whether spectral content across a corresponding frequency band has (or has been determined to have) reflections from the user's ear canal. For example, when the parameter is high ("1"), the spectral content may be associated with (e.g., predominantly) reflections, while when the parameter is low (e.g., "0"), the spectral content predominantly does not have reflections (or may include reflections below a threshold).

[0086] In one aspect, a reflection parameter estimator can use estimated reflection parameters to generate a reflection pattern (e.g., a band reflection map). Specifically, the estimator can determine the reflection pattern based on an effective rank estimate of the reflection (or reflection parameters) within one or more frequency bands. Thus, the pattern can indicate whether (or which frequency band) one or more frequency bands include a large amount (e.g., above reflection) of reflection (e.g., across a frequency range).

[0087] On the other hand, one or more reflection parameters can be generated by examining a microphone signal in a similar manner, as described herein. For example, a reflection estimator can use a microphone signal to determine reflection parameters across one or more frequency bands (e.g., determining the presence of reflection in a frequency band based on spectral content within that band being above a threshold).

[0088] In one aspect, the reflection estimator 42 may estimate reflection parameters (and / or reflection patterns) based on an MLML algorithm. Specifically, the estimator may include an ML algorithm that receives one or more in-ear microphone signals as input and can generate (or predict) multiple (sub-band) reflections associated with the one or more microphone signals. Based on this number, the reflection estimator may generate one or more reflection parameters. In one aspect, the reflection estimator may estimate parameters for each of several frequency bands spanning a frequency range (e.g., the range of frequencies audible to humans). In another aspect, any method may be used to determine the reflection parameters of the user's ear canal based on one or more microphone signals.

[0089] In one aspect, in addition to (or instead of) determining whether a reflection is being received (or detected) in one or more microphone signals, the reflection parameter estimator can be configured to determine (or estimate) the direction of arrival (DOA) of the received reflection. Specifically, the estimator can be configured to track reflections originating from a specific area within the user's ear (e.g., reflections occurring inside or from the ear canal), rather than reflections originating from inside the ear canal, such as direct sound (or other reflections) that may occur elsewhere (e.g., based on the direct energy path generated by the amplifier when it projects sound into the ear canal). In another aspect, these other reflections may be due to sound being reflected from a portion of the headphones (e.g., the front portion) (such as from the ear tip of the headphones that are inserted into and form a seal in the user's ear). Therefore, the reflection parameter estimator can be configured to use one or more microphone signals to generate a directional beam pattern that points (or toward) the user's ear canal and away from the output device. Specifically, the estimator can estimate the band direction of arrival (DOA) parameter by analyzing one or more frequency bands that include the strongest signal (e.g., having the highest spectral content relative to other frequency bands). In one aspect, these frequency bands (e.g., those with the strongest signal) may represent the direct sound path between the loudspeaker and one or more microphones. Therefore, the estimator can estimate the DOA parameters for each frequency band associated with reflections from within the ear canal (e.g., within a specific area pointed to by the beam pattern). In another aspect, the parameter estimator can generate a DOA mask that includes the DOA parameters for each frequency band from which reflections originate (e.g., the DOA parameters for each frequency band) (e.g., thereby filtering out frequency bands associated with the direct path between the loudspeaker and microphones, as well as other reflections not originating from the region of interest).

[0090] In one aspect, the reflection parameter estimator 42 is configured to combine (or add) DOA parameters to the reflection parameters. In another aspect, parameters can be combined by multiplying parameters in the frequency domain. In yet another aspect, the parameter estimator can combine reflection patterns from one or more reflection patterns and a DOA mask of one or more DOA parameters to create a reflection mask. In one aspect, one or more estimated reflection parameters can define the reflection mask. Further information regarding filtered signals is described herein.

[0091] In one aspect, reflection filter 45 is configured to receive estimated reflection parameters from reflection parameter estimator 42 and receive (or acquire) N microphone signals, and is configured to filter the signals from the N microphones using the reflection parameters to produce one or more output signals. Specifically, one or more estimated reflection parameters can be used to define (or generate) a reflection filter to filter out spectral content (e.g., in the frequency domain) from the microphone signals that is not associated with the reflection parameters. Specifically, the reflection filter can remove spectral content associated with reflection parameters having low values, thereby indicating that the spectral content does not include (e.g., does not primarily include) reflections. In one aspect, a reflection filter can be defined using a created reflection mask, which can be a binary mask, wherein reflections of the microphone signals (e.g., as output signals) can be filtered out when processed using the mask. For example, the output (filtered) signal may include energy reflected only from the sound output of a speaker, which has been reflected back from inside the ear canal and received by the in-ear microphone. Reflections that may originate from areas outside the ear canal and / or direct acoustic energy from the speaker can be filtered out by the reflection mask. In one respect, the reflection filter 45 can produce the same number of filtered signals (N signals) as the number of microphone signals.

[0092] In one aspect, a microphone signal filtered by a reflection filter can be obtained after a microphone signal used to generate one or more reflection parameters. For example, the reflection parameters can be estimated over a period of time (e.g., one hour, one day, etc.). Once estimated, the reflection filter 45 can receive the reflection parameters and obtain a microphone signal obtained after the period of time for filtering subsequent microphone signals.

[0093] In one aspect, the reflection filter 45 can be dynamically updated (or changed) based on changes in reflections within the user's ear canal. For example, the reflection parameter estimator can be configured to adjust the reflection parameters based on changes in the spectral content captured within one or more microphone signals. For example, the reflection parameter estimator can monitor one or more frequency bands to determine whether reflections in those bands exceed a threshold, as described above. When it is determined that the number of reflections in a frequency band has decreased (e.g., from a time point prior to estimating the reflection parameters), the parameter estimator can adjust the parameters (e.g., from high to low values). Thus, the reflection filter can be dynamically updated as the reflection parameters change.

[0094] In one aspect, the ear geometry estimator 41 is configured to estimate the geometry (e.g., size, shape, length, etc.) of a user's ear (e.g., ear canal). In another aspect, the geometry estimator may be configured to receive one or more reflection parameters (e.g., reflection masks) estimated by the reflection parameter estimator 42, and may be configured to estimate the ear geometry based on these parameters. For example, the geometry estimator may be configured to perform a table lookup on a data structure (e.g., stored within the controller 20 of the output device 3) that associates one or more ear geometries with one or more (e.g., predefined) reflection parameters (or masks).

[0095] On the other hand, ear geometry can be estimated by other means. For example, a geometry estimator can be configured to receive N microphone signals and an input audio signal being used to drive a speaker, and can be configured to determine the geometry based on these signals. For example, the geometry can be determined using the signals to determine the (e.g., impulse) response at one or more microphones, and the geometry can be determined based on the response. On the other hand, a geometry estimator can determine the geometry based solely on microphone signals (e.g., no input audio signal).

[0096] On the other hand, the ear geometry can receive user input and / or sensor input, and can determine the ear geometry based on that input. As described herein, the output device can be a pair of earplugs, each having an ear tip that forms a seal within the ear canal when the earplugs are inserted into the user's ear. In one aspect, the output device can be fitted with one or more different types or sizes of ear tips. For example, a user can couple a particular ear tip (e.g., a smaller diameter relative to other ear tips) to the output device, which is sized to form a seal within the ear canal of the user's ear. For example, when a smaller ear tip is not large enough (e.g., does not have a sufficiently large diameter) to form a seal, the user can mount a larger ear tip to the output device, possibly due to the size of the user's ear canal. In one aspect, the ear geometry estimator can receive user input indicating what type (e.g., size, such as small, medium, or large) of ear tip should be coupled to the output device. Based on this type, the ear geometry can determine the geometry (e.g., size) of the ear canal. Returning to the previous example, when a user instructs that a large ear tip be coupled to the output device, the geometry estimator may determine that the ear canal is large (e.g., larger than average). Alternatively, the ear geometry estimator may determine the geometry of the user's ear based on sensor input from one or more sensors. For example, the ear geometry estimator may receive image data captured by one or more cameras communicatively coupled to a controller. Estimator 41 may determine the geometry of the user's ear based on the image data, which may include images of the user's ear (and / or ear canal). Alternatively, estimator 41 may use any method to determine the geometry of the user's ear. Alternatively, the ear geometry estimator may use a combination of one or more methods described herein to determine the geometry.

[0097] The severity predictor 43 is configured to receive (or filtered) a signal from the output of the reflection filter 45 and is configured to determine one or more characteristics of an auditory element (e.g., whether an obstruction exists in the ear canal) based on the output signal. In one aspect, the predictor may be an ML algorithm (e.g., a deep neural network (DNN)) configured to take the filtered signal as input and determine whether one or more auditory elements possess one or more characteristics. For example, the DNN may be configured to output an indication of the presence of an obstruction in the ear canal based on the filtered signal. In another aspect, the predictor may be configured to determine one or more physical characteristics of the obstruction. For example, the predictor may determine the amount of obstruction (e.g., size, shape, length, etc.) and / or the type of obstruction (e.g., whether the obstruction is earwax, whether the obstruction is fluid, etc.).

[0098] In one aspect, the predictor can be configured to receive an estimate of ear geometry and to determine characteristics based on that ear geometry. For example, the predictor can determine the presence of an obstruction in the ear canal based on a comparison between the ear geometry and a filtered signal (and / or a comparison between the estimated ear geometry and the determined physical characteristics of the predicted obstruction). Specifically, the predictor can determine whether the predicted obstruction (or at least a portion of it) is actually part of the natural shape, size, etc., of the user's ear canal. For example, the characteristics of the ear canal may vary from person to person; for instance, one person's ear canal may be narrower than another's. In this case, the predictor can determine the presence of an obstruction based on the difference between the user's estimated ear geometry and the physical characteristics of the predicted obstruction. For example, the ear geometry may indicate that at least a portion of the user's ear canal is tubular and has a specific diameter. However, the predicted obstruction may have a similar shape along the same portion but with a narrower diameter. The diameter of the obstruction may appear narrower because the object within the ear canal reduces the diameter of air (and sound) that propagates to the user's eardrum. Therefore, the predictor can determine the size of the blockage based on the difference between the smaller diameter of the blockage and the larger diameter of the user's (natural) ear geometry.

[0099] On the other hand, predictor 43 may be configured to receive calibration of in-ear microphone 21 from microphone calibrator 46 and may be configured to predict one or more characteristics based on the calibrated and filtered signal. In one aspect, calibrator may be configured to receive input audio signal and microphone signal from in-ear microphone and may be configured to perform microphone calibration (e.g., based on impulse response measurements of the output sound from a speaker at the microphone's location). In one aspect, the calibration operation may be performed before (or after) the user wears the output device. For example, predictor 43 may use calibration to address microphone characteristic problems (e.g., such as age) and compensate for microphone characteristics using the filtered signal from the calibration process. On the other hand, the calibration operation performed by calibrator may be the same as (or similar to) the impedance calibration performed by impedance calibrator 25. On the other hand, microphone calibrator may use any calibration method to compensate for microphone characteristics.

[0100] In one aspect, the operation of the ear geometry estimator 41 and the microphone calibrator 46 is optional (as indicated by the dashed boxes). For example, the characteristic severity predictor may predict the presence of a characteristic based on a filtered signal. In another aspect, the predictor may receive data from one or both boxes based on components communicatively coupled to the controller. As described herein, the output device may include N in-ear microphones 21, wherein the controller receives N microphone signals. In one aspect, when the output device includes only one microphone, the predictor may obtain calibration from the microphone calibrator. Or more specifically, when performing the operations described herein with only one microphone signal (e.g., using only one microphone signal to estimate reflection parameters and filtering the next microphone signal using those parameters), the predictor may use calibration. In another aspect, when the N microphone signals include two or more microphone signals obtained from the N in-ear microphones of the output device, the predictor 43 may not obtain calibration (e.g., the operation of the microphone calibrator may not be performed).

[0101] In one aspect, the characteristic severity predictor 43 is configured to determine the severity level of the predicted characteristic. Specifically, the severity level can be a numerical value (e.g., between 0 and 1), where a low value (e.g., 0.1) indicates low severity, while a high value (e.g., 1) indicates a severity higher than low severity. For example, regarding obstruction, the severity level can indicate one or more characteristics of the obstruction, such as size, shape, length, etc., where a low severity level indicates a smaller obstruction (e.g., relative to the size of the ear canal). However, a high severity level can indicate that the obstruction is more severe than a low severity level obstruction (e.g., a larger obstruction relative to the ear canal). In one aspect, the severity level can correspond to the amount of open space within the ear canal. For example, an obstruction with a low severity level may indicate an obstruction where the diameter of the open space within the ear canal is above a threshold. Conversely, an obstruction may have a high severity level when the diameter of the open space within the ear canal is below a threshold.

[0102] In one aspect, the predictor can use an estimate of ear geometry to determine the severity level, as described herein. For example, the predictor can determine the severity level of the blockage based on a comparison of ear geometry and a filtered signal (and / or ear canal characteristics determined by the predictor based on the filtered signal). As an example, ear geometry may indicate that the ear canal has a specific diameter, and the filtered signal may indicate a blockage with a reduced diameter. Therefore, the predictor can determine the severity based on the reduction (or difference) in diameter caused by the blockage.

[0103] On the other hand, the predictor can determine severity based on a comparison of a severity value with one or more thresholds. For example, the predictor can determine the severity value based on a filtered signal. As described herein, the severity value can be a value between 0 and 1. Using the severity value, the predictor can determine the severity level of the congestion based on whether the severity value is below one or more thresholds. As described herein, one or more thresholds can indicate the severity level, such as low, medium, and / or high. In this case, the severity level may be low when the severity level is below both the low and high thresholds (i.e., above the low threshold). The severity level may be medium if the severity value is above the low threshold but below the high threshold. And the severity level may be high when the severity value is above the high threshold.

[0104] Decision logic 44 is configured to receive the determined characteristics and / or the predicted severity level of the characteristics, and is configured to determine whether the predictor's assessment is accurate. Specifically, the decision logic is configured to determine a confidence score for the prediction made by the predictor and to determine whether the confidence score exceeds a threshold. For example, the decision logic is configured to receive an input audio signal for driving speaker 22 and is configured to determine the confidence score based on the audio content contained within the input audio signal. In one aspect, the determined confidence score may be based on the type of audio content within the input audio signal. For example, when the audio signal includes (e.g., mainly includes) speech, the confidence score may be low. Conversely, when the audio signal includes musical works, the confidence score may be high (or above a threshold). In another aspect, the confidence score may be based on characteristics of the audio content, such as whether the audio content includes transitions or attenuations. For example, when the audio content includes one or more attenuations (e.g., a transition when the sound attenuates from one intensity to another intensity (e.g., silence), the decision logic may determine a high confidence score. For example, when the audio content does not include attenuation (or includes attenuation below a threshold), such as when the audio content is constant (e.g., has a constant signal level, or a signal level not below a threshold), the confidence score may be low. On the other hand, the confidence score may be based on the playback level of the input audio signal (e.g., when the playback level is above a threshold, the confidence score may be low). On yet another hand, the confidence score may be based on other characteristics of the input audio signal.

[0105] The decision logic is configured to determine whether the determined confidence score exceeds a threshold (e.g., a predefined one). If so, the decision logic can be configured to output a notification about the determined characteristic. For example, the logic may output (e.g., audible) a notification indicating the presence of an obstruction in the ear canal and / or indicating the severity of the obstruction. In this case, when the severity level is low, the notification may indicate to the user that there may be a small obstruction in the ear canal. Alternatively, the notification may provide the user with advice to seek medical advice regarding the determined characteristic.

[0106] As described herein, the ear characteristic tracking and decision-making system 27 can be configured to track one or more characteristics of one or more auditory elements of a user's ear. In one aspect, the system can track predicted blockages within the ear canal of the user's ear. For example, upon determining the presence of a blockage, the system can track the blockage over a period of time (e.g., one day, one week, one month). Specifically, the system can perform one or more of the operations described herein to predict the presence and severity of a blockage once or multiple times within a given time period. In one aspect, the system can output a notification in response to determining that the severity level of the blockage is increasing (e.g., the severity value is increasing at a predetermined rate).

[0107] In one aspect, controller 20 may perform one or more additional audio signal processing operations. For example, controller may perform echo cancellation to eliminate all (or most) of the speaker signals that travel directly from the speaker to the in-ear microphone. For example, controller may determine one or more linear filters based on the transmission path between the in-ear microphone and the speaker, and apply the filters to the input audio signal to generate an echo estimate that is subtracted from N microphone signals captured by the in-ear microphone. In another aspect, controller may use any echo cancellation method.

[0108] Figure 5 This is a flowchart of one aspect of a process 60 for determining one or more characteristics of a user's ear based on estimated reflection parameters of the user's ear canal. In one aspect, process 60 may be executed by an output device 3 (e.g., controller 20).

[0109] At the start of process 60, controller 20 drives speaker 22 to project a first sound into the ear canal of the user's ear while the output device is being worn by the user (at box 61). Specifically, the controller may use (e.g., input) audio signals to drive speaker 22, which is arranged to project sound into the ear canal of the user's ear. As described herein, the audio signal may be part of a segment of audio content desired by the user, such as a musical piece, podcast, or movie soundtrack. The controller obtains a first set of microphone signals from a set of in-ear microphones 21, which includes the first sound captured by the microphones and reflections of the first sound (at box 62). As described herein, the controller may obtain one or more (or N) microphone signals from one or more microphones of the output device. The microphone signals may include sound generated by the speaker (e.g., a direct sound path between the speaker and the microphone) and reflections of sound from within the ear canal of the user's ear (e.g., sound reflected from the inner wall of the ear canal, sound reflected from the eardrum, etc.). The controller generates at least one reflection parameter of the ear canal of the user's ear based on the first set of microphone signals (at box 63). Specifically, the reflection parameter estimator 42 can generate one or more reflection parameters, each indicating the number of reflections received by the microphone in one or more frequency bands. Additionally, the estimator can estimate the DOA parameters based on a directional beam pattern generated by one or more microphone signals from a first set of microphone signals directed towards the user's ear canal and away from the output device. As described herein, these parameters can be combined (e.g., to create a reflection mask).

[0110] Controller 20 drives a speaker to project a second sound into the ear canal of the user's ear (at box 64). For example, the second sound may be a portion of the input audio signal used to drive the speaker to project the first sound. However, the second sound may be associated with a portion of a signal following a portion of the signal associated with the first sound (e.g., playing back the second sound after the first sound). On the other hand, the second sound may be associated with an audio signal different from the input audio signal used to drive the speaker when projecting the first sound. The controller obtains a second set of microphone signals from the set of in-ear microphones, which includes the second sound captured by the microphones and reflections of the second sound (at box 65). In one aspect, the second set of microphone signals may be captured after the first set of microphone signals. For example, the controller may obtain the second set of microphone signals once the reflection parameters have been estimated. In some aspects, the second set of microphone signals may be obtained after a certain time period (e.g., one minute, one hour, one day, etc.) from the start of estimating the reflection parameters.

[0111] Controller 20 generates a set of output (e.g., filtered) signals based on the second set of microphone signals and at least one reflection parameter, wherein the set of output signals includes only reflections of the second sound captured by the microphones (at box 66). Specifically, reflection filter 45 uses one or more reflection parameters (or reflection masks) to filter the spectral content of the second set of microphone signals that is not associated with reflections from (or reflections from) the user's ear canal. For example, the reflection filter uses reflection parameters to filter the second sound from the second set of microphone signals by removing spectral content across one or more frequency bands that are not associated with at least one reflection parameter.

[0112] Controller 20 determines one or more characteristics of one or more auditory elements of the user's ear based on this set of output signals (in box 67). Specifically, characteristic severity predictor 43 is configured to receive the output signals and apply them as input to a DNN (or characteristic severity model) that determines one or more characteristics as its output. For example, the predictor may determine the presence of an obstruction in the user's ear canal based on the output signals. On the other hand, the predictor may determine the (e.g., physical) characteristics of the obstruction as described herein, such as the size, shape, etc.

[0113] In one aspect, upon identifying a characteristic, the controller can present a notification to the user, alerting them to the characteristic. Specifically, the decision logic can output a notification (e.g., an auditory notification via speaker 22, a visual (pop-up) notification via the display screen, etc.) to remind the user that a characteristic has been detected, and / or additional information related to that characteristic, such as a description of the characteristic and / or its severity. For example, when the characteristic is an obstruction in the ear canal, the notification can indicate a less severe obstruction. With such information, the user can seek medical diagnosis from a medical professional, such as an audiologist.

[0114] In one aspect, the notification may be based on whether a confidence score associated with the audio content of the first sound (e.g., a first audio signal used to drive a speaker to produce the first sound) exceeds a confidence threshold. As described herein, the confidence score may be based on characteristics of the audio content of the first sound (e.g., whether the sound includes a threshold number of transitions, attenuations, and / or pauses). In response to the confidence score of the first sound exceeding the confidence threshold, the output device may output (present) a notification regarding the determined characteristics of the auditory element.

[0115] Variations exist in the execution process 30 in some aspects. For example, at least some specific operations in these processes may not be performed in the exact order shown and described. The specific operation may not be performed in a consecutive series of operations, and different specific operations may be performed in different aspects. For example, a severity predictor may determine the severity of a characteristic, as described herein.

[0116] Figure 6 Another block diagram is shown of an output device configured to determine the characteristics of a user's ear based on acoustic input impedance measurements, according to one aspect. In one aspect, this diagram shows that controller 20 includes... Figure 2 and Figure 4 The controller may include several operation blocks that can perform ear feature tracking and decision-making operations as described herein. In one aspect, the controller may include more or fewer operation blocks, such as those having... Figure 2 and Figure 4 The additional operation block shown.

[0117] As described herein, this figure illustrates a diagram in which a controller is configured to determine characteristics based on acoustic input impedance measurements. Specifically, this figure illustrates a controller configured to determine characteristics and / or predict (determine) characteristic severity (e.g., the severity level of the characteristic) based on acoustic input impedance measurements using an in-ear microphone. For example, the controller may be configured to use... Figure 2 Impedance analysis, as described in [the document], determines whether a user's ear canal contains one or more blockages. Therefore, characteristics such as blockages can be predicted based on N microphone signals (which can be two or more microphone signals). Figure 4 Unlike other controllers, this diagram illustrates that such predictions can be made based on a microphone signal.

[0118] As described herein, the model parameter estimator 26 is configured to receive the estimated acoustic input impedance from the impedance estimator 24 and is configured to use the acoustic input impedance to determine an equivalent acoustic-mechanical model of the user's ear. In one aspect, one of these parameters may correspond to (or be equivalent to) the ear canal of the user's ear. Such a parameter may indicate characteristics of the ear canal, such as shape, size, length, uniformity, etc.

[0119] The characteristic severity predictor 43 is configured to receive estimated parameters (such as parameters equivalent to the user's ear canal) and determine whether a certain characteristic exists in the user's ear. Specifically, the characteristic severity predictor may be configured to determine whether there is a blockage in the user's ear canal based on parameters associated with the user's ear canal. For example, the parameter may indicate that the ear canal is shorter than that of a normal person, which may be the result of an object or blockage in the ear canal. Alternatively, the predictor may perform a table lookup in a data structure that associates ear canal parameters with blockages, and upon identifying the parameter in the data structure associated with the estimated parameters, the predictor may determine that a blockage exists.

[0120] On the other hand, the predictor can determine the presence of obstruction based on monitoring ear canal parameters. As described herein, these parameters can indicate characteristics of the ear canal. For example, the predictor can determine this characteristic based on the parameter. Specifically, the predictor can perform a table lookup in a data structure that associates one or more characteristics of the user's ear canal with the estimated parameter. On the other hand, the characteristic can be associated with (or be a part of) the parameter. In one aspect, the predictor can monitor these characteristics over time. Specifically, model parameter estimator 26 can estimate one or more parameters over the time period, which the predictor 43 can use to accumulate characteristics. Based on these characteristics, the predictor can determine whether there are changes in the ear canal, such as whether the length of the ear canal decreases over time. The predictor can use such changes to determine whether there is obstruction in the ear canal.

[0121] Additionally, the predictor can determine the severity of the blockage based on monitoring of characteristics. Specifically, as the predictor monitors the blockage, the characteristics of the blockage can be compared to one or more thresholds. For example, the size of the blockage (e.g., relative to the size of the ear canal) can be compared to a threshold, and when the size exceeds the threshold, the severity level of the blockage can be high (e.g., indicating that the size of the blockage adversely restricts airflow between the outer ear and the eardrum). As described herein, a severity level can be determined for the blockage, but it can also be determined for any specific characteristic of one or more auditory elements.

[0122] As described so far, model parameter estimator 26 can be configured to estimate parameters associated with a user's ear canal. On the other hand, the parameter estimator can be configured to estimate parameters of a specific characteristic of one or more auditory elements. For example, the parameter estimator can estimate parameters associated with an obstruction (or object) within a user's ear (e.g., ear canal). In one aspect, ear characteristic tracking and decision-making system 27 can use this parameter to determine one or more characteristics of the obstruction, as described herein.

[0123] In determining a feature and / or its severity level, the decision logic is configured to determine whether the confidence score associated with the input audio signal is higher than a threshold, as described herein. If so, the decision logic may present one or more notifications to alert the user to the feature and / or the severity level associated with it.

[0124] As described above, in order to perform ear characteristic tracking and decision-making operations, one aspect of this technology considers the collection and use of available data from legitimate and specific sources. It is conceivable that, in some cases, the data being collected may include information or data that may be unique to an individual and therefore usable to identify a particular person. For example, the data may include location-based data, home address, online identifiers, demographic data, email address, telephone number, date of birth, other types of information related to the user's fitness level or health status (e.g., measurements of vital signs, medication-related information, exercise-related information), or other personal data (e.g., biometric identifiers).

[0125] It is conceivable that personal data could be beneficial to users in this technology. For example, health information could be used to determine whether one or more auditory elements of a user's ear have characteristics based on the user's consent.

[0126] It is conceivable that entities that collect, analyze, disclose, transfer, store, or otherwise use personal data will be required to comply with robust privacy practices and / or privacy policies. For example, due to industry or governmental requirements for protecting user privacy, these entities are expected to consistently apply and implement privacy practices that are generally considered to meet or exceed these requirements.

[0127] Any information regarding the use of personal data should be prominently provided, and users should have easy access to such information. This information should be updated whenever the way data is collected or used changes. Collection of users' personal information should be limited to lawful purposes. Furthermore, unless there is another lawful basis provided by applicable law, user consent must be obtained before collection / sharing occurs. Steps are needed to protect and ensure access to personal data and to help ensure that those authorized to access personal data comply with their privacy policies and procedures. It is conceivable that third-party assessments could be conducted to demonstrate that entities comply with widely accepted privacy policies and practices. Additionally, the suitability of policies and practices based on the types of personal data collected and / or accessed, as well as applicable laws and standards (e.g., stricter jurisdiction-specific considerations), should be considered. For example, in the United States, federal and / or state laws (e.g., the Health Insurance Portability and Accountability Act (HIPAA)) regulate the collection or access to certain health data. In other countries, health data is governed by different regulations and policies and should be handled accordingly.

[0128] It is also conceivable that users can selectively block certain uses or access to their personal data. In these cases, hardware and / or software components can be employed to block or prevent access to such personal data. For example, in cases such as ear characteristic tracking and decision-making, this technology can be configured to allow a user to opt-in or opt-out to participate in the collection of personal information data when the output device detects and tracks ear characteristics or at any time thereafter. In another example, the user can choose not to use audio data to perform these operations. Specifically, the user can choose to limit the amount (and duration) of audio data (or any type of data) captured and used for tracking ear characteristics. In addition to providing opt-in and opt-out options, this disclosure envisions providing notifications related to access to or use of personal information. For example, when the output device receives a microphone signal, the user can be notified to determine whether the auditory elements of the user's ear have characteristics as described herein (such as blockage). Additionally, the user can be reminded again before the output device accesses personal information data.

[0129] It is conceivable that minimizing the risk of unintentional or unauthorized access or use will influence how personal data is managed and processed. For example, one way to minimize risk is to limit the scope of data collection and delete data when it is no longer needed. Furthermore, where possible (e.g., for specific health applications), this disclosure may de-identify data to protect user privacy. Where possible, various de-identification methods may be used, for example, by removing tags or names, controlling the amount of data to be stored, or controlling the granularity of the data being stored (e.g., collecting locations at the country level rather than the address level, or aggregating data across users), and / or other methods (e.g., differential privacy) may be used.

[0130] Finally, while it is conceivable that personal data can be used to implement various publicly disclosed embodiments, it is also conceivable that various aspects can be implemented without access to such personal data. In other words, no aspect of this disclosure will become inoperable due to the loss of some or all of the personal data. In one example, content may be provided to a user based on aggregated data that is not specific to each user or contains only a minimal amount of personal data. In the latter case, the user may only receive content being processed on the user's device or other information that is not personal to the user.

[0131] As previously described, one aspect of this disclosure may be a non-transitory machine-readable medium (such as microelectronic memory) on which instructions are stored, programming one or more data processing units (generally referred to herein as a "processor") to perform network operations and audio signal processing operations, as described herein. In other aspects, some of these operations may be performed by specific hardware components containing hard-wired logic. Alternatively, those operations may be performed by any combination of programmed data processing units and fixed hard-wired circuit components.

[0132] While certain aspects have been described and illustrated in the accompanying drawings, it should be understood that such aspects are merely illustrative of the broad disclosure and not limiting, and that this disclosure is not limited to the specific structures and arrangements shown and described, as various other modifications will be apparent to those skilled in the art. Therefore, the description is to be regarded as exemplary and not restrictive.

[0133] In some aspects, this disclosure may include the language "[element A] and [element B] at least one". This language may refer to one or more of these elements. For example, "at least one of A and B" may refer to "A", "B", or "A and B". Specifically, "at least one of A and B" may refer to "at least one of A and at least one of B" or "at least either A or B". In some aspects, this disclosure may include the language "[element A], [element B], and / or [element C]". This language may refer to any of these elements or any combination thereof. For example, "A, B, and / or C" may refer to "A", "B", "C", "A and B", "A and C", "B and C", or "A, B, and C".

Claims

1. A method performed by an earphone including a speaker and an in-ear microphone, the method comprising: The headphones are calibrated using a first microphone signal from an in-ear microphone, including sound generated by a speaker, to obtain baseline measurements. The calibration is performed while the headphones are in a case arranged to contain the headphones when not being worn by a user. The calibration involves using the first microphone signal to estimate a secondary path transfer function, wherein the estimated secondary path transfer function is associated with a predefined acoustic input impedance, which is associated with the volume of the case. While the headphones are being worn by the user, an audio signal is used to drive the speaker to project sound into the user's ear canal; The sound from the ear canal of the user's ear is captured from the in-ear microphone as a second microphone signal; Model parameters, as part of a model, are determined based at least on the captured second microphone signal and the baseline measurements, the model being equivalent to one or more auditory elements of the user's ear; and Based on the model parameters, a notification related to one or more characteristics of one or more auditory elements of the user's ear is transmitted.

2. The method of claim 1, further comprising using the second microphone signal to determine a second-order path transfer function representing the response between the speaker and the in-ear microphone when the headphones are worn by a user, wherein the determined model parameters are based on the second-order path transfer function.

3. The method of claim 2, wherein the second-level path transfer function is determined as part of an active noise reduction process performed by the headphones to generate an anti-noise signal for driving the speaker.

4. The method according to claim 2, wherein, The model includes an acoustic-mechanical model, and determining the model parameters includes: The acoustic input impedance of the user's ear canal relative to the baseline measurement is measured using the second-order path transfer function; and The acoustic input impedance is used to determine the model parameters as part of the acoustic-mechanical model that is at least equivalent to the user's middle and outer ears.

5. The method of claim 4, wherein performing the calibration includes determining one or more impedance calibration parameters of an impedance model configured to estimate the acoustic input impedance of the user's ear canal, wherein measuring the acoustic input impedance includes estimating the acoustic input impedance by applying the determined one or more impedance calibration parameters and a second-order path transfer function to the impedance model.

6. The method of claim 4, wherein the acoustic input impedance of the user's ear canal is measured over a time period, wherein the method further comprises determining the one or more characteristics based on the change between the model parameters and corresponding model parameters of another acoustic-mechanical model over the time period.

7. The method according to claim 1, further comprising: The model parameters are compared with the corresponding model parameters of another acoustic-mechanical model; and The one or more features are identified based on the comparison between the model parameters and the corresponding model parameters.

8. An earphone, comprising: speaker; microphone; processor; as well as The memory contains instructions that, when executed by the processor, cause the earphones to... Calibration is performed on the headphones using a first microphone signal, including sound generated by a speaker, from a microphone to obtain baseline measurements. This calibration is performed while the headphones are in a charging case arranged to house them when not worn by a user. The calibration involves estimating a secondary path transfer function using the first microphone signal, wherein the estimated secondary path transfer function is associated with a predefined acoustic input impedance, which is associated with the volume of the charging case. While the headphones are being worn by the user, audio signals are used to drive the speaker to project sound into the user's ear canal. The sound from within the ear canal of the user's ear is captured from the microphone and used as a second microphone signal. Model parameters, as part of a model, are determined based at least on the second microphone signal and the baseline measurements, the model being equivalent to one or more auditory elements of the user's ear. Based on the model parameters, a notification related to one or more characteristics of one or more auditory elements of the user's ear is transmitted.

9. The earphone of claim 8, wherein the memory has further instructions for using the second microphone signal to determine a second-level path transfer function representing the response between the speaker and the microphone when the earphone is worn by a user, wherein the model parameters are based on the second-level path transfer function.

10. The headphones of claim 9, wherein the second-level path transfer function is determined to be part of an active noise reduction process performed by the headphones to generate an anti-noise signal for driving the speaker.

11. The headphones according to claim 9, wherein, The model includes an acoustic-mechanical model, and determining the model parameters includes: The acoustic input impedance of the user's ear canal relative to the baseline measurement is measured using the second-order path transfer function; and The acoustic input impedance is used to determine the model parameters as part of an acoustic-mechanical model that is at least equivalent to the user's middle and outer ear.

12. The headphones of claim 11, wherein performing the calibration includes determining one or more impedance calibration parameters of an impedance model configured to estimate the acoustic input impedance of the ear canal of the user's ear, wherein measuring the acoustic input impedance includes estimating the acoustic input impedance by applying the determined one or more impedance calibration parameters and a second-order path transfer function to the impedance model.

13. The headphones of claim 11, wherein the acoustic input impedance of the ear canal of the user's ear is measured over a time period, wherein the memory has further instructions for determining the one or more characteristics based on the change between the model parameters and corresponding model parameters of another acoustic-mechanical model over the time period.

14. The earphone of claim 8, wherein the memory has further instructions for performing the following operations: The model parameters are compared with the corresponding model parameters of a predefined acoustic-mechanical model; and The one or more features are identified based on the comparison between the model parameters and the corresponding model parameters.

15. A method performed by an earphone including a speaker and a set of in-ear microphones, the method comprising: While the headphones are being worn by the user, the speaker is driven to project a first sound into the ear canal of the user's ear; Receive a first set of microphone signals from the set of in-ear microphones, the first set of microphone signals including the first sound and the reflection of the first sound; At least one reflection parameter of the ear canal of the user's ear is generated based on the first set of microphone signals; Drive the speaker to project a second sound into the ear canal of the user's ear; Receive a second set of microphone signals from the first set of in-ear microphones, the second set of microphone signals including the second sound and the reflection of the second sound; as well as A set of output signals is generated based on the second set of microphone signals and the at least one reflection parameter; Determine whether the audio content of the first sound includes at least one of decay and pause; as well as In response to determining that the audio content of the first sound includes at least one of attenuation and pause, a notification relating to the characteristics of the auditory elements of the user's ear is transmitted based on the set of output signals.

16. The method of claim 15, wherein generating the at least one reflection parameter comprises determining, based on a comparison of a first set of microphone signals, whether the spectral content of a frequency band across a plurality of frequency bands of the first set of microphone signals primarily comprises a first sound as direct sound from the speaker or primarily comprises a reflection of the first sound, wherein the at least one reflection parameter is generated in response to the spectral content primarily comprising the reflection of the first sound.

17. The method of claim 16, wherein determining whether the spectral content primarily comprises a first sound as direct sound from the speaker or primarily comprises reflections of the first sound comprises determining, over a time period, whether the spectral content across the frequency band of the first set of microphone signals comprises a threshold number of reflections of the first sound.

18. The method of claim 15, further comprising: The severity value of the characteristic is determined based on the set of output signals; and The severity level of the feature is determined based on a comparison between the severity value and one or more thresholds.

19. The method of claim 15, further comprising estimating the ear geometry of the user's ear based on at least one of user input, sensor input, and the first set of microphone signals, wherein the characteristic of the auditory element is based on the estimated ear geometry of the user's ear.

20. The method of claim 19, further comprising determining the severity level of the characteristic based on a comparison of the ear geometry with a generated set of output signals.

Images