Internal noise source filtering for active acoustic sensing

Abstract

Techniques and devices for performing internal noise source filtering for active acoustic sensing are described. With internal noise source filtering (202), interference caused by the hearing-worn device (102) performing other operations (e.g., rendering audio content (204), performing active noise cancellation (206), or operating according to a pass-through mode (214)) may be attenuated within the received ultrasound signal to improve the sensitivity and accuracy of audio volumetric tracing. This performance improvement enables audio plethysmography to be performed when the audible wearable device (102) performs these other operations. Further, it improves the ability of audio plethysmography for speech processing, which may include speech activity detection, speech recognition, and / or dialog detection.

Description

Background Art

[0001] Wireless technology has become prevalent in everyday life, allowing users to easily access communications and data. One type of wireless technology is wireless hearables, examples of which include wireless earbuds and wireless headphones. Wireless hearables allow users to move freely while listening to audio content such as music, audiobooks, podcasts, and videos. With the popularity of wireless hearables, there is a market for adding additional features to existing hearables without introducing hardware changes. Summary of the Invention

[0002] Techniques and devices for performing internal noise source filtering for active acoustic sensing are described. Using internal noise source filtering, interference caused by a hearable device performing other operations (e.g., rendering audio content, performing active noise cancellation, or operating in a transparency mode) can be attenuated within the received ultrasonic signal to improve the sensitivity and accuracy of audio plethysmography. This performance improvement enables audio plethysmography to be performed while the hearable device is performing these other operations. Furthermore, it improves the ability of audio plethysmography to be used for voice processing, including voice activity detection, speech recognition, and / or dialog detection.

[0003] Aspects described below include a method for performing internal noise source filtering for active acoustic sensing. The method includes transmitting an audible signal that propagates within at least a portion of a user's ear canal during a first time period. The method also includes transmitting an ultrasonic transmit signal that propagates within at least a portion of the user's ear canal during the first time period. The method further includes receiving an ultrasonic receive signal during the first time period. The ultrasonic receive signal represents a version of the ultrasonic transmit signal having one or more characteristics modified based on propagation within the ear canal and based on vocalizations made by the user during the first time period. The received ultrasonic receive signal includes an internal noise component caused by interference generated by rendering of the audible signal. The method further includes generating a de-noised signal by filtering the internal noise component within the received ultrasonic receive signal based on the version of the audible signal. The method also includes detecting vocalizations based on the de-noised signal. In examples, the version of the audible signal may represent an electrical version of the audible signal and / or a digital version of the audible signal.

[0004] Aspects described below include a computer-readable storage medium comprising instructions that, in response to being executed by a processor, cause a hearable device to perform any of the methods described herein.

[0005] Aspects described below include an apparatus having at least one transducer and at least one processor. The apparatus is configured to use the at least one transducer and the at least one processor to perform any of the methods described herein.

[0006] Aspects described below include a system having components for performing internal noise source filtering for active acoustic sensing. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Devices and techniques for performing internal noise source filtering for active acoustic sensing are described with reference to the following drawings. Throughout the drawings, like numbers are used to reference similar features and components: Figure 1-1 An example environment in which active acoustic sensing may be implemented is shown; Figure 1-2 shows example geometric changes in the ear canal that can be detected using active acoustic sensing; Figure 2 An example environment is shown in which internal noise source filtering for active acoustic sensing may be implemented; Figure 3 Example internal noise sources that can affect audio plethysmography are shown; Figure 4 shows example signals that can be detected using a hearable device; Figure 5 An example operation of a microphone of a hearable device is shown; Figure 6 Example components of a computing device are shown; Figure 7 Example components of a hearable device are shown; Figure 8 shows example operations of two hearable devices; Figure 9 An example embodiment of a hearable device capable of performing internal noise source filtering for active acoustic sensing is shown; Figure 10 An example flow chart for operating a hearable device is shown; Figure 11 An example scheme implemented by a calibration module of a hearable device is shown; Figure 12 An example implementation of a pre-processing module for performing aspects of audio plethysmography is shown; Figure 13 An example implementation of a measurement module for performing internal noise source filtering is shown; Figure 14 An example implementation of an internal noise source filter stage for performing internal noise source filtering is shown; Figure 15 An example implementation of an external noise source filter stage is shown; Figure 16 An example method for performing internal noise source filtering is shown; Figure 17 Another example method for performing internal noise source filtering is shown; and Figure 18 An example computing system embodying or in which techniques enabling the use of internal noise source filtering for active acoustic sensing may be implemented is shown. DETAILED DESCRIPTION

[0008] As electronic devices become more common, users are incorporating them into their daily lives. For example, users can use electronic devices to obtain daily weather and traffic information, control the temperature of their homes, answer doorbells, turn lights on and off, and / or play background music. However, interacting with some electronic devices can be cumbersome and inefficient. For example, an electronic device may have a physical user interface that requires the user to physically touch the electronic device to navigate one or more prompts. In such situations, the user must divert their attention from other primary tasks to interact with the electronic device, which can be inconvenient and disruptive.

[0009] To address this issue, some electronic devices support voice control, which enables users to interact with electronic devices in a non-physical and less cognitively demanding manner compared to other interfaces that require physical contact and / or the user's visual attention. With voice control, electronic devices seamlessly exist in the surrounding environment and provide users with access to information and services while they perform primary tasks such as cooking, cleaning, driving, talking to people, or reading a book.

[0010] While voice control can provide a convenient means of interacting with electronic devices, there are several challenges associated with voice control. For example, in noisy environments, other external noises can make a user's voice difficult to perceive. Consequently, it can be challenging for voice control to detect and / or recognize voice commands spoken by a user. Furthermore, noisy environments can sometimes cause voice control to incorrectly respond to the voice of another person who is not authorized to use the electronic device.

[0011] Some devices address these challenges by integrating voice accelerometers (VAs) into earbud headphones. Voice accelerometers can detect when a user is speaking based on sound traveling through bone conduction. However, voice accelerometers can be bulky and expensive. To improve aesthetics and reduce bulk, it is desirable to design hearables with a smaller size. As space becomes limited, integrating additional components (such as voice accelerometers) within hearables can be challenging. As hearables become more popular, there is a market for adding additional features to existing hearables without introducing hardware changes.

[0012] According to one or more preferred embodiments, a hearable device, such as an earphone, is provided that is capable of performing a novel physiological monitoring process referred to herein as audio plethysmography. Audio plethysmography is an active acoustic method capable of sensing subtle, physiologically relevant changes observable in a user's outer and middle ears. Instead of relying on auxiliary sensors such as optical or electrical sensors, audio plethysmography involves sending and receiving acoustic signals that propagate at least partially within the user's ear canal. To perform audio plethysmography, the hearable device forms at least a partial seal in or around the user's outer ear. This seal enables the formation of an acoustic circuit comprising the seal, the hearable device, the ear canal, and the eardrum. By sending and receiving acoustic signals, the hearable device can identify changes in the acoustic circuit to detect and / or identify vocalizations made by the user. Vocalizations can include any sound produced using the user's lungs, vocal cords, and / or mouth. Example types of vocalizations can involve the user speaking, whispering, shouting, humming, whistling, singing, or uttering other utterances. Hearable devices can utilize audio plethysmography to support a variety of different features, including voice user interfaces (VUIs) and / or multi-factor voice authentication.

[0013] In addition to audio plethysmography, some hearable devices may also provide other features. Example features may include rendering (e.g., playing) audible content, providing active noise cancellation, and / or operating according to a transparency mode. Transparency mode enables sounds from the external environment to be rendered to the user. In this way, transparency mode enables the hearable device to be used as a hearing aid. Active noise cancellation can significantly attenuate sounds from the external environment, while transparency mode allows sounds from the external environment to pass to the user's ears. In some cases, transparency mode can further amplify these sounds to help the hearing impaired.

[0014] However, providing these other features while also performing audio plethysmography can be challenging. More specifically, signals generated for rendering audio content, providing active noise cancellation, and / or for operating in a transparency mode can interfere with ultrasound signals received for audio plethysmography. This interference can make it challenging for hearable devices to use audio plethysmography for speech processing.

[0015] To address this challenge, techniques and devices are described for performing internal noise source filtering for active acoustic sensing. Using internal noise source filtering, interference caused by other operations performed by the hearable device (e.g., rendering audio content, performing active noise cancellation, or operating in a transparency mode) can be attenuated within the received ultrasonic signal to improve the sensitivity and accuracy of audio plethysmography. This performance improvement enables audio plethysmography to be performed while the hearable device is performing these other operations. Furthermore, it improves the ability of audio plethysmography to be used for speech processing, including voice activity detection, speech recognition, and / or dialogue detection. Operating Environment

[0016] Figure 1-1 is an illustration of an example environment 100 in which active acoustic sensing can be implemented. In the example environment 100, a hearable device 102 is connected to a computing device 104 using a physical interface or a wireless interface. The hearable device 102 is a device that can play audible content provided by the computing device 104 and direct the audible content to the ear 108 of a user 106. In this example, the hearable device 102 operates in conjunction with the computing device 104. In other examples, the hearable device 102 can operate or be implemented as a standalone device. Although depicted as a smartphone, the computing device 104 can include other types of devices, including those related to Figure 6 The devices described.

[0017] The hearable device 102 is capable of performing audio plethysmography 110, an active acoustic sensing method that occurs at the ear 108. The hearable device 102 can perform this sensing without the use of other auxiliary sensors, such as optical or electrical sensors. Through audio plethysmography 110, the hearable device 102 can utilize speech processing to detect and / or recognize vocalizations made by the user 106. Example types of speech processing may include voice activity detection 112, speech recognition 114, and / or dialogue detection 116.

[0018] For voice activity detection 112, the hearable device 102 detects vocalizations made by the user 106. To perform voice activity detection 112, the hearable device 102 uses audio plethysmography 110 to detect subtle pressure waves that propagate to the ear canal 118 of the user 106. The pressure waves may occur due to movement associated with the jaw of the user 106, vibrations associated with vocalizations, or some combination thereof. Generally speaking, these pressure waves modify the characteristics of ultrasonic signals transmitted and received by the hearable device 102 and propagated through the ear canal 118. Using voice activity detection 112, the hearable device 102 can enhance the performance of a voice user interface or provide multi-factor authentication.

[0019] For speech recognition 114, the hearable device 102 recognizes words spoken by the user 106. The hearable device 102 can perform speech recognition 114 using audio plethysmography 110 or using audio plethysmography 110 to enhance information provided by another signal, such as an audible signal captured by the hearable device 102. Speech recognition 114 can also be used to enhance the performance of a voice user interface or to enhance multi-factor authentication.

[0020] For conversation detection 116, the hearable device 102 determines whether the user 106 is having a conversation with another person. To perform conversation detection 116, the hearable device 102 uses the audio plethysmogram 110 to filter the audible signal captured by the hearable device 102. When the signal associated with the audio plethysmogram 110 is not correlated with speech from another person, the audio plethysmogram 110 can be used to filter and / or attenuate signal components common between the audible signal and the signal used for the audio plethysmogram 110. This filtering improves the signal-to-noise ratio for detecting speech by a person other than the user 106.

[0021] In response to detecting the occurrence of a conversation, conversation detection 116 can be used to control the operation of the hearable device 102 and / or computing device 104. In one example, the audio content can be paused or resumed based on whether a conversation is detected. In another example, the volume of the audio content can be reduced or increased based on whether a conversation is detected. In yet another example, active noise cancellation can be temporarily suspended, and the hearable device 102 can operate according to a transparency mode based on the conversation detection 116 detecting the conversation. Alternatively, based on the conversation detection 116 detecting the absence of a conversation, active noise cancellation can be resumed and the hearable device 102 can exit the transparency mode.

[0022] To use audio plethysmography 110 , the user 106 positions the hearable device 102 in a manner that creates at least a partial seal 120 around or in the ear 108 . Figure 1-11 , portions of an ear 108 are shown, including an ear canal 118 and an eardrum 122 (or tympanic membrane). Due to seal 120, the hearable device 102, ear canal 118, and eardrum 122 are coupled together to form an acoustic loop. Audio plethysmography 110 involves, at least in part, measuring properties associated with this acoustic loop. The properties of the acoustic loop can change due to a variety of different situations or actions.

[0023] For example, consider Figure 1-2 , wherein a change occurs in the physical structure of the ear 108. Example changes in the physical structure include a change in the geometry of the ear canal 118 and / or a change in the volume of the ear canal 118. Such changes may be caused, at least in part, by bone conduction and / or pressure waves associated with vocalizations made by the user 106.

[0024] For example, at 124, the tissue surrounding the ear canal 118 and the ear drum 122 itself are slightly "squeezed" due to bone conduction and / or pressure waves. At 124, this squeezing causes the volume of the ear canal 118 to decrease slightly. However, at 126, the squeezing subsides and the volume of the ear canal 118 increases slightly relative to 124. The physical changes within the ear 108 can modulate the amplitude and / or phase of the ultrasonic signal propagating through the ear canal 118, as described with respect to FIG. Figure 8 As further described, techniques for audio plethysmography 110 may be performed while the hearable device 102 is playing audible content to the user 106, as described with respect to Figure 2 、 Figure 3 -1 and Figure 3-2 Further described.

[0025] Figure 2 Example environments 200-1, 200-2, and 200-3 are shown in which internal noise source filtering 202 can be implemented for active acoustic sensing. Environments 200-1, 200-2, and 200-3 depict example features of a hearable device 102 that can be active during audio plethysmography 110. In environment 200-1, the hearable device 102 renders audio content 204 for the user 106. The audio content 204 represents any type of sound generated by the hearable device 102, such as music, a ringtone, an alarm, a caller's voice, and the like.

[0026] In environment 200-2, the hearable device 102 provides active noise cancellation 206. Utilizing active noise cancellation 206, the hearable device 102 significantly attenuates other sounds present in the external environment, such as speech 208, music 210, and / or external noise 212. Active noise cancellation 206 can create a quiet environment for the user 106 to focus. Additionally or alternatively, active noise cancellation 206 can make it easier for the user 106 to hear audio content 204, which can also be played for the user 106 in environment 200-2.

[0027] In environment 200-3, the hearable device 102 operates in a transparency mode 214. The transparency mode enables sounds from the external environment to be rendered to the user 106. The rendering of these sounds may also include amplifying the sounds to assist the hearing-impaired. In environment 200-3, the transparency mode 214 enables the user 106 to hear another person's speech 216 (e.g., hear the other person speak).

[0028] When the hearable device 102 renders audio content 204, provides active noise cancellation 206, and / or operates according to a transparency mode 214, internal signals generated by the hearable device 102 during any of these operations may interfere with the received ultrasound signals. This interference may make the audio plethysmography 110 challenging to use for speech processing. To address this issue, the hearable device 102 performs internal noise source filtering 202. Figure 3 Example types of internal noise sources that may be filtered using internal noise source filtering 202 are further described.

[0029] Figure 3 Example components of the hearable device 102 are shown. In the depicted configuration, the hearable device 102 includes at least one microphone 302 and circuitry 304. The microphone 302 may be used to receive ultrasonic signals for audio plethysmography 110 and / or audible signals for other operations of the hearable device 102. The circuitry 304 may provide information about Figure 2 For example, the circuit system 304 can render the audio content 204 , perform active noise cancellation 206 , and / or support the transparency mode 214 .

[0030] During operation, microphone 302 receives a signal represented by received signal 306. While microphone 302 is generating received signal 306 and / or while hearable device 102 is processing received signal 306, circuit system 304 may generate one or more signals. Hearable device 102 renders (e.g., plays, transmits, or sends) the signal generated by circuit system 304. The rendered signal propagates within ear canal 118 of user 106.

[0031] From the perspective of any technique for processing received signal 306, the signal generated by circuit system 304 represents noise signal 308 (or self-generated noise signal 308), which can interfere with received signal 306. For example, noise signal 308 can affect the waveform characteristics (e.g., amplitude, frequency, and / or phase) of received signal 306. This modification of the waveform characteristics of received signal 306 is represented by internal noise component 310. In some cases, internal noise component 310 is caused by noise signal 308 being modulated onto or mixed with received signal 306. This interference between noise signal 308 and received signal 306 can be due to nonlinearities in microphone 302, mixing operations performed by hearable device 102, or some other component and / or operation of hearable device 102. This internal noise component 310 can make the use of audio plethysmography 110 for speech processing challenging.

[0032] Generally speaking, circuitry 304 represents internal noise sources 312. Example internal noise sources 312 may include any type of circuitry implemented within the hearable device 102 (e.g., within the housing of the hearable device 102) and whose operation may interfere with signals received for audio plethysmography 110. Figure 3 In FIG. 3 , internal noise sources 312 may include speakers 314 , active noise cancellation (ANC) circuitry 316 (ANC circuitry 316 ), transparency mode circuitry 318 , or some combination thereof.

[0033] Speaker 314 can be used to render audio content 204 in environment 200-1. To render audio content 204, speaker 314 generates an audible signal 320 that can represent noise signal 308. Audible signal 320 can be associated with music 322, human speech, or some other type of sound. Audible signal 320 can also be referred to as an audible spectrum signal and generally includes frequencies between 20 hertz (Hz) and 20 kilohertz (kHz).

[0034] Active noise cancellation circuitry 316 performs active noise cancellation 206, as shown in environment 200-2. To perform active noise cancellation 206, active noise cancellation circuitry 316 generates anti-noise signal 324. Anti-noise signal 324 represents another type of noise signal 308 that may interfere with received signal 306 and contribute to internal noise component 310.

[0035] Transparency mode circuitry 318 enables hearable device 102 to operate in transparency mode, as shown in environment 200-3. For transparency mode, transparency mode circuitry 318 generates a transparency mode signal 326 that includes sounds from the external environment. Transparency mode signal 326 represents another type of noise signal 308 that may interfere with received signal 306 and contribute to internal noise component 310.

[0036] Designing the hearable device 102 in a manner that mitigates interference caused by the noise signal 308 can be challenging. Cost and / or size constraints of the hearable device 102 can make it challenging to incorporate some interference shielding techniques and / or utilize more expensive components that are less likely to generate such interference. To address this challenge, techniques for internal noise source filtering 202 filter (e.g., attenuate) the internal noise component 310 within the received signal 306 to enhance the audio plethysmography 110 and / or speech processing. Figure 4 and Figure 5 Example techniques for speech processing utilizing audio plethysmography 110 are further described.

[0037] Figure 4 Example signals that may be detected by the hearable device 102 are shown. In an environment 400, a user 106 is wearing the hearable device 102. During operation, the hearable device 102 may receive a variety of different signals. In one aspect, the hearable device 102 receives at least one over-the-air (OTA) signal 402 (OTA signal 402). The over-the-air signal 402 may include a speech component 404 and / or a noise component 406. The speech component 404 may include an utterance made by the user 106, such as a voiceprint phrase used to activate a voice user interface. The noise component 406 may represent any undesirable audible sound that may mask or interfere with the detection of the utterance. The noise component 406 may represent Figure 2 The ambient noise 212, music 210 and / or speech 208. Generally speaking, the airborne signal 402 includes audible frequencies.

[0038] In another aspect, the hearable device 102 receives at least one bone conduction signal 408. The bone conduction signal 408 represents sound traveling via bone conduction to the ear 108 of the user 106. The bone conduction signal 408 also includes a speech component 404. In most cases, the bone conduction signal 408 does not include a noise component 406.

[0039] To perform active acoustic sensing, the hearable device 102 transmits and receives at least one ultrasonic signal 410. The ultrasonic signal 410 propagates within the ear canal 118. The vocalization of the user 106 may cause changes in the physical structure of the ear 108. Thus, the ultrasonic signal 410 may also include a speech component 404.

[0040] The ultrasonic signal 410 may not directly include the noise component 406. However, some designs of the hearable device 102 may cause the noise component 406 associated with the air signal 402 to interfere with the detection of the speech component 404 within the ultrasonic signal 410, as explained further below.

[0041] In some example embodiments, the hearable device 102 may be designed to minimize interference between the air signal 402 and the ultrasonic signal 410. For example, the hearable device 102 may utilize different microphones to receive these signals. In this case, the hearable device 102 may directly process the ultrasonic signal 410 to detect the vocalization. In other example embodiments, the hearable device 102 utilizes the same microphone to receive the air signal 402 and the ultrasonic signal 410. While this may be beneficial for meeting size constraints of the hearable device 102, it may result in a version of the noise component 406 being present within the ultrasonic signal 410. When the hearable device 102 receives the air signal 402, the bone conduction signal 408, and the ultrasonic signal 410, these signals may interact and make it challenging to detect the speech component 404 using the audio plethysmography 110. In particular, the air signal 402 may be modulated onto or mixed with the ultrasonic signal 410, as described with respect to Figure 5 Further described.

[0042] Figure 5 1 shows an example operation of the microphone 302 of the hearable device 102. During operation, the microphone 302 receives an airborne signal 402, a bone conduction signal 408, and an ultrasonic signal 410. The microphone 302 may include a filter module that may generate separate signals associated with different frequency ranges. In this example, the microphone 302 generates a received audible signal 504 and a received ultrasonic signal 506, which are received at Figure 5 These signals may be down-converted to baseband frequencies. In general, the received audible signal 504 and the received ultrasonic signal 506 represent electrical signals that may be processed by other components of the hearable device 102.

[0043] In graph 508, received audible signal 504 and received ultrasonic signal 506 are shown as including different frequencies. Received audible signal 504 may include frequencies associated with the audible spectrum (e.g., frequencies between approximately 20 Hz and 20 kHz). Thus, received audible signal 504 may also be referred to as a received audible spectrum signal. In contrast, received ultrasonic signal 506 may include frequencies associated with the ultrasonic spectrum (e.g., frequencies between approximately 20 kHz and 2 megahertz (MHz)).

[0044] The received audible signal 504 represents the convolution of the airborne signal 402 and the bone conduction signal 408. Thus, the received audible signal 504 includes a speech component 404 and a noise component 406 associated with the utterance. The speech component 404 is provided by both the airborne signal 402 and the bone conduction signal 408. The received audible signal 504 can be represented by Equation 1: where Y RAS represents the received audible signal 504, h BC represents bone conduction channel, S represents phonation, h OTA represents an air channel, and N represents noise (eg, external noise 212, music 210, and / or speech 208). The term represents the bone conduction component 512. The term represents the speech component 404 of the received audible signal 504 . The term represents the noise component 406 of the received audible signal 504 .

[0045] The received ultrasonic signal 506 represents the convolution of the ultrasonic signal 410 and a modulated version of the received audible signal 504 (represented by the modulation component 510). The modulation component 510 may be caused by the interaction of the air signal 402 and the bone conduction signal 408 with the ultrasonic signal 410 when the microphone 302 receives these signals. More specifically, the modulation component 510 may be caused by harmonics, interference, and / or intermodulation distortion generated based on the design of the hearable device 102, nonlinearities within one or more components of the hearable device 102, and / or the operation of the hearable device 102 (e.g., mixing operation). Generally speaking, the modulation component 510 represents a version of the received audible signal 504 shifted to ultrasonic frequencies. The modulation component 510 is linearly modulated onto the ultrasonic spectrum. The received ultrasonic signal 506 may be represented by Equation 2: where Y RUS represents the received ultrasonic signal 506, h US represents the ultrasonic channel, S represents the sound, hMC Indicates the modulation channel, and Y RAS The received audible signal 504 is represented. The term represents the speech component 404 of the received ultrasonic signal 506 . The term represents the modulation component 510. Due to the modulation channel, the received ultrasonic signal 506 includes a linearly modulated version of the received audible signal 504. Thus, the noise component 406 within the modulated version of the received audible signal 504 can make directly detecting the speech component 404 within the received ultrasonic signal 506 challenging.

[0046] Generally speaking, the speech component 404 is superimposed on the ultrasonic signal 410 and is associated with a vocalization. The speech component 404 is uncorrelated with the noise component 406. Due to variations in bone conduction and physical structure within the ear 108, the speech component 404 modulates the received ultrasonic signal 506 differently than the modulation component 510.

[0047] The speech component 404 of the received ultrasonic signal 506 is frequency dependent. In other words, different ultrasonic frequencies modulate the utterance differently. However, the bone conduction component 512 is not frequency selective. In other words, the bone conduction component 512 is associated with a fixed channel. Techniques for voice activity detection 112, speech recognition 114, and / or conversation detection 116 utilize the received audible signal 504 to extract the speech component 404 from the received ultrasonic signal 506, as described with respect to Figure 13 and Figure 14 Further described.

[0048] Figure 6 An example implementation of a computing device 104 is shown. The computing device 104 is shown with various non-limiting example devices, including a desktop computer 104-1, a tablet computer 104-2, a laptop computer 104-3, a television 104-4, a computing watch 104-5, computing glasses 104-6, a gaming system 104-7, a microwave oven 104-8, and a vehicle 104-9. Other devices may also be used, such as augmented and / or virtual reality headsets, home service devices, smart speakers, smart thermostats, baby monitors, Wi-Fi TM A router, a drone, a touchpad, a drawing tablet, a netbook, an e-reader, a home automation and control system, a wall display, and another home appliance. Note that computing device 104 can be wearable, non-wearable but mobile, or relatively non-mobile (e.g., a desktop computer and an appliance).

[0049] The computing device 104 includes one or more computer processors 602 and at least one computer-readable medium 604, including memory media and storage media. Applications and / or an operating system (not shown), embodied as computer-readable instructions on the computer-readable medium 604, can be executed by the computer processor 602 to provide some of the functionality described herein. The computer-readable medium 604 can optionally include an application 606, a voice user interface 608, and / or a voice authenticator 610. The application 606 can use information provided by the hearable device 102 to perform actions. Example actions can include displaying data associated with the audio plethysmography 110 to the user 106. For voice activity detection 112, the application 606 can indicate whether a vocalization was detected. The voice user interface 608 can enable the user 106 to control the computing device 104 via voice commands. The voice authenticator 610 can authenticate the user 106 and, upon successful authentication, enable use of the voice user interface 608. Application 606 , voice user interface 608 , and / or voice authenticator 610 may utilize aspects of voice activity detection 112 and / or speech recognition 114 to improve the performance and / or enhance the security of computing device 104 .

[0050] The computing device 104 may also include a network interface 612 for transmitting data through a wired, wireless, or optical network. For example, the network interface 612 may be connected through a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, a mesh network, a Bluetooth ® etc. to transmit data. The computing device 104 may also include a display 614. Although not explicitly shown, the hearable device 102 may be integrated into the computing device 104 or may be physically or wirelessly connected to the computing device 104. Figure 7 The hearable device 102 is further described.

[0051] Figure 7An example hearable device 102 is shown. The hearable device 102 is shown with various non-limiting example devices, including wireless earbuds 702-1, wired earbuds 702-2, and headphones 702-3. Earbuds 702-1 and 702-2 are types of in-ear devices that are placed into the ear canal 118. Each earbud 702-1 or 702-2 can represent the hearable device 102. Headphones 702-3 can rest on the ear 108 or cover the ear. Headphones 702-3 can represent closed-back headphones, open-back headphones, on-ear headphones, or over-ear headphones. Each headphone 702-3 includes two hearable devices 102 physically packaged together. Generally, there is one hearable device 102 for each ear 108 .

[0052] The hearable device 102 includes a communication interface 704 for communicating with the computing device 104, although this interface need not be used when the hearable device 102 is integrated within the computing device 104. The communication interface 704 can be a wired or wireless interface over which audio content is transferred from the computing device 104 to the hearable device 102. The hearable device 102 can also use the communication interface 704 to transfer data associated with the audio plethysmography 110 to the computing device 104. Generally speaking, the data provided by the communication interface 704 is in a format that can be used by the application 606, the voice user interface 608, and / or the voice authenticator 610.

[0053] The communication interface 704 also enables the hearable device 102 to communicate with another hearable device 102. During bistatic sensing, for example, the hearable device 102 can use the communication interface 704 to coordinate with another hearable device 102 to support binaural audio plethysmography 110, as described with respect to Figure 8 Specifically, the transmitting hearable device 102 may transmit timing and waveform information to the receiving hearable device 102 so that the receiving hearable device 102 can properly demodulate the received ultrasonic signal 506 .

[0054] The hearable device 102 includes at least one transducer 706 that can convert electrical signals into sound waves. The transducer 706 can also detect and convert sound waves into electrical signals. These sound waves can include ultrasonic frequencies and / or audible frequencies, either of which can be used for audio plethysmography 110. In particular, the spectrum (e.g., frequency range) used by the transducer 706 to generate the acoustic signal can include frequencies from the low end of the audible range to the high end of the ultrasonic range (e.g., between 20 Hertz (Hz) and 2 Megahertz (MHz)). Other example spectra for audio plethysmography 110 can include frequencies between 20 Hz and 20 kilohertz (kHz), between 20 kHz and 2 MHz, between 20 and 96 kHz, between 20 and 60 kHz, or between 30 and 40 kHz.

[0055] In an exemplary embodiment, the transducer 706 has a monostatic topology. Using this topology, the transducer 706 can convert electrical signals into acoustic waves and convert acoustic waves into electrical signals (e.g., can transmit or receive acoustic signals). Example monostatic transducers can include piezoelectric transducers, capacitive transducers, and micromachined ultrasonic transducers (MUTs) using microelectromechanical systems (MEMS) technology.

[0056] Alternatively, the transducer 706 can be implemented using a dual-base topology, which includes multiple transducers that are physically separated. In this case, a first transducer converts an electrical signal into an acoustic wave (e.g., transmits an acoustic signal), and a second transducer converts an acoustic wave into an electrical signal (e.g., receives an acoustic signal). The example dual-base topology can be implemented using at least one speaker 708 and at least one microphone 710. The speaker 708 and microphone 710 can be dedicated to audio plethysmography 110 or can be used for both audio plethysmography 110 and other functions of the computing device 104 (e.g., presenting audible content to the user 106, capturing the user's 106 voice for a phone call, or for voice control). The speaker 708 can represent Figure 3 The speaker 314. The microphone 710 may represent Figure 3 and Figure 5 Microphone 302.

[0057] Generally speaking, the speaker 708 and the microphone 710 are directed toward the ear canal 118 (e.g., oriented toward the ear canal 118). Thus, the speaker 708 can direct ultrasonic signals toward the ear canal 118, and the microphone 710 is responsive to receiving ultrasonic signals from a direction associated with the ear canal 118. In some cases, the hearable device 102 includes another microphone 710 that is directed away from the ear canal 118 toward the external environment (e.g., oriented away from the ear canal 118). This other microphone can be used to receive the over-the-air signal 402.

[0058] The hearable device 102 includes at least one analog circuit 712, which includes circuitry and logic for conditioning electrical signals in the analog domain. The analog circuit 712 can include analog-to-digital converters, digital-to-analog converters, amplifiers, filters, mixers, and switches for generating and modifying electrical signals. In some embodiments, the analog circuit 712 includes other hardware circuitry associated with the speaker 708 or microphone 710.

[0059] The hearable device 102 also includes at least one system processor 714 and at least one system media 716 (e.g., one or more computer-readable storage media). In the depicted configuration, the system media 716 includes a pre-processing module 718 and a measurement module 720. The system media 716 also optionally includes a calibration module 722. The pre-processing module 718, the measurement module 720, and the calibration module 722 can be implemented using hardware, software, firmware, or a combination thereof. In this example, the system processor 714 implements the pre-processing module 718, the measurement module 720, and the calibration module 722. In an alternative example, the computer processor 602 of the computing device 104 can implement at least a portion of the pre-processing module 718, the measurement module 720, and / or the calibration module 722. In this case, the hearable device 102 can transmit the digital samples of the acoustic signal to the computing device 104 using the communication interface 704.

[0060] about Figures 11 to 12 The operations of the pre-processing module 718, the measurement module 720, and the calibration module 722 are further described. Aspects of the internal noise source filtering 202 may be performed at least in part by the measurement module 720, as described with respect to Figure 13 and Figure 14 As further described, the measurement module 720 may also use active acoustic sensing to perform aspects of voice activity detection 112, speech recognition 114, and / or dialogue detection 116, as described with respect to Figure 13 Further described.

[0061] Some hearable devices 102 include active noise cancellation circuitry 316, which enables the hearable device 102 to reduce background or ambient noise. In such cases, the microphone 710 used for audio plethysmography 110 may be implemented using a feedback microphone of the active noise cancellation circuitry 316. During active noise cancellation, the feedback microphone provides feedback information regarding the performance of the active noise cancellation 206. During audio plethysmography 110, the feedback microphone receives an ultrasonic signal 410, which is provided to a pre-processing module 718. In some cases, active noise cancellation 206 and audio plethysmography 110 are performed simultaneously using the feedback microphone. In such cases, the ultrasonic signal 410 received by the feedback microphone may be provided to the pre-processing module 718, and the feedback signal for active noise cancellation 206 may be provided to the active noise cancellation circuitry 316. Other embodiments are possible in which the microphone 710 is implemented using a feedforward microphone of the active noise cancellation circuitry 316.

[0062] Furthermore, some hearable devices 102 include a transparency mode circuitry 318 that enables the hearable device 102 to amplify background or ambient sounds. In some embodiments, the transparency mode circuitry 318 can include at least one amplifier. The microphone 710 used for audio plethysmography 110 can also be used in the transparency mode 214. More specifically, the ultrasonic signal 410 received by the microphone 710 can be provided to the pre-processing module 718, and the sensed signal for the transparency mode 214 can be provided to the transparency mode circuitry 318. The active noise cancellation circuitry 316, the transparency mode circuitry 318, and / or the speaker 708 represent example types of circuitry 304 that can inadvertently act as an internal noise source 312 during audio plethysmography 110.

[0063] Despite Figure 7 6 and / or a voice authenticator 610. In this case, the voice user interface 608 enables the user 106 to control the operation of the hearable device 102 using voice control. The voice authenticator 610 can authenticate the user 106 and enable the voice user interface 608 for the hearable device 102. Figure 8 Different types of audio plethysmography 110 are further described. Active acoustic sensing

[0064] Figure 81 and 102-2 perform monaural audio plethysmography 110. This means that the hearable devices 102-1 and 102-2 independently perform audio plethysmography 110 on different ears 108 of a user 106. In this scenario, the first hearable device 102-1 is positioned near the right ear 108 of the user 106, and the second hearable device 102-2 is positioned near the left ear 108 of the user 106. Each hearable device 102-1 and 102-2 includes a speaker 708 and a microphone 710. The hearable devices 102-1 and 102-2 can operate in a monostatic manner during the same time period or during different time periods. In other words, each hearable device 102-1 and 102-2 can independently transmit and receive ultrasonic signals.

[0065] For example, the first hearable device 102-1 uses the speaker 708 to transmit a first ultrasonic transmit signal 802-1, which propagates within at least a portion of the right ear canal 118 of the user 106. The first hearable device 102-1 uses the microphone 710 to receive a first ultrasonic receive signal 804-1. The first ultrasonic receive signal 804-1 represents a version of the first ultrasonic transmit signal 802-1 that has been modified at least in part by an acoustic circuit associated with the right ear canal 118. This modification may cause the first ultrasonic receive signal 804-1 to change in amplitude, phase, and / or frequency relative to the first ultrasonic transmit signal 802-1.

[0066] Similarly, the second hearable device 102-2 uses the speaker 708 to transmit a second ultrasonic transmit signal 802-2, which propagates within at least a portion of the left ear canal 118 of the user 106. The second hearable device 102-2 receives a second ultrasonic receive signal 804-2 using the microphone 710. The second ultrasonic receive signal 804-2 represents a version of the second ultrasonic transmit signal 802-2 that has been modified by the acoustic circuit associated with the left ear canal 118. This modification may cause the second ultrasonic receive signal 804-2 to change in amplitude, phase, and / or frequency relative to the second ultrasonic transmit signal 802-2.

[0067] The technique of monaural audio plethysmography 110 can be particularly beneficial because it enables the computing device 104 to compile information from both the hearable devices 102-1 and 102-2, which can further improve measurement confidence. For some aspects of audio plethysmography 110, analyzing the acoustic path between the two ears 108 can be beneficial, as described further below.

[0068] In a second example operation, two hearable devices 102-1 and 102-2 perform binaural audio plethysmography 110. This means that the hearable devices 102-1 and 102-2 jointly perform audio plethysmography 110 across both ears 108 of the user 106. In this scenario, at least one of the hearable devices 102 (e.g., the first hearable device 102-1) includes a speaker 708, and at least one of the other hearable devices 102 (e.g., the second hearable device 102-2) includes a microphone 710. The hearable devices 102-1 and 102-2 operate together in a bistatic manner during the same time period.

[0069] During operation, the first hearable device 102-1 transmits a third ultrasonic transmit signal 802-3 using the speaker 708. The third ultrasonic transmit signal 802-3 propagates through the right ear canal 118 of the user 106. The third ultrasonic transmit signal 802-3 also propagates through the acoustic channel existing between the right and left ears 108. In the left ear 108, the third ultrasonic transmit signal 802-3 propagates through the left ear canal 118 of the user 106 and is represented as a third ultrasonic receive signal 804-3. The second hearable device 102-2 receives the third ultrasonic receive signal 804-3 using the microphone 710. The third ultrasonic receive signal 804-3 represents a version of the third ultrasonic transmit signal 802-3 modified by the acoustic circuit associated with the right ear canal 118, the acoustic channel associated with the face of the user 106, and the acoustic circuit associated with the left ear canal 118. Such modification can change the amplitude, phase, and / or frequency of the third ultrasonic receive signal 804-3 relative to the third ultrasonic transmit signal 802-3. In some cases, the hearable device 102-2 measures the time of flight (ToF) associated with the propagation from the first hearable device 102-1 to the second hearable device 102-2. Sometimes, a combination of monaural audio plethysmography and binaural audio plethysmography 110 is applied to further improve measurement confidence.

[0070] Figure 8 The ultrasonic transmission signal 802 can represent a variety of different types of signals, such as the above Figure 7 In an example embodiment, the ultrasonic transmission signal 802 may be Figure 4 and Figure 5The ultrasonic transmit signal 802 may be a continuous wave signal (e.g., a sinusoidal signal) or a pulsed signal. Some ultrasonic transmit signals 802 may have a specific pitch (or frequency). Other ultrasonic transmit signals 802 may have multiple pitches (or multiple frequencies). Various modulations may be applied to generate the ultrasonic transmit signal 802. Example modulations include linear frequency modulation, triangular frequency modulation, stepped frequency modulation, phase modulation, or amplitude modulation. The ultrasonic transmit signal 802 may be transmitted as part of a calibration procedure or a measurement procedure, such as as Figure 9 as further described in part.

[0071] Figure 9 An example embodiment of a hearable device 102 for performing voice activity detection 112 is shown. In the depicted configuration, the hearable device 102 includes a speaker 708, a microphone 710, an analog circuit 712, a pre-processing module 718, a measurement module 720, a calibration module 722, and the circuit system 304. However, other embodiments of the hearable device 102 are possible in which the hearable device 102 does not include the calibration module 722 to reduce processing power requirements. In this case, the pre-processing module 718 may perform the following operations: Figure 12 Various aspects of frequency selection are further described to improve the signal-to-noise ratio of the audio plethysmography 110 .

[0072] The outputs of speaker 708 and microphone 710 are coupled to an input of analog circuit 712. Preprocessing module 718 has an input coupled to the output of analog circuit 712. Preprocessing module 718 also has an output coupled to the inputs of measurement module 720 and calibration module 722. Measurement module 720 has other inputs coupled to microphone 710 and circuit system 304, respectively. Calibration module 722 has an output coupled to speaker 708.

[0073] Consider an example operation of the hearable device 102 according to the monaural audio plethysmography 110. In the event that the hearable device 102 includes the calibration module 722, the hearable device 102 may perform a calibration process before performing a measurement process. Figure 10 The calibration process and the measurement process are further described.

[0074] During both the calibration process and the measurement process, the speaker 708 transmits an ultrasonic transmit signal 802, and the microphone 710 receives an ultrasonic receive signal 804. During the calibration process, the ultrasonic transmit signal 802 and the ultrasonic receive signal 804 may have tones 902-1 through 902-M, where M represents a positive integer. During the measurement process, the ultrasonic transmit signal 802 and the ultrasonic receive signal 804 may have selected tones 904-1 through 904-N, where N represents a positive integer less than or equal to M. The selected tones 904-1 through 904-N may represent a subset (sometimes a true subset) of the tones 902-1 through 902-M. The microphone 710 may also receive the air signal 402 and the bone conduction signal 408 during the measurement process.

[0075] The analog circuit 712 performs analog-to-digital conversion to generate a digital transmit signal 906 and a digital receive signal 908 based on the ultrasonic transmit signal 802 and the received ultrasonic signal 506, respectively. The pre-processing module 718 performs frequency down-conversion and demodulation to generate at least one pre-processed signal 910 based on the digital transmit signal 906 and the digital receive signal 908. The pre-processing module 718 may also apply filtering to generate the pre-processed signal 910.

[0076] As part of the calibration procedure, the calibration module 722 processes the pre-processed signal 910 to determine the selected tones 904-1 through 904-N. The selected tones 904-1 through 904-N can improve the performance of the audio plethysmography 110 during the measurement procedure. The calibration module 722 uses a control signal to transmit the selected tones 904-1 through 904-N to the speaker 708. The speaker 708 receives the control signal identifying the selected tones 904-1 through 904-N and can transmit subsequent ultrasound transmission signals 802 for the measurement procedure using the selected tones 904-1 through 904-N.

[0077] As part of the measurement procedure, the measurement module 720 can use the preprocessed signal 910 and the noise signal 308 to perform various aspects of the internal noise source filtering 202. The measurement module 720 can also perform various aspects of voice activity detection 112, speech recognition 114, and / or conversation detection 116 to generate speech data 912. The speech data 912 can also be referred to as speech data or audio plethysmography data. In noisy environments, the measurement module 720 can also further process the preprocessed signal 910 using the received audible signal 504 provided by the microphone 710 to perform voice activity detection 112, speech recognition 114, and / or conversation detection 116. The speech data 912 can be transmitted to the application 606, the voice user interface 608, and / or the voice authenticator 610. The speech data 912 can include an indication of whether the user 106's voice was detected, phrases recognized based on the speech component 404 of the preprocessed signal 910, signals including the speech component 404, and the like. Additionally or alternatively, the voice data 912 may include control signals for controlling the operation of the hearable device 102 and / or the computing device 104. Figure 10 The calibration procedure and measurement procedure are further described.

[0078] Figure 10 An example flow chart 1000 for operating a hearable device 102 is shown. Figure 10 At 1002, the hearable device 102 can optionally perform a calibration procedure using the calibration module 722. The calibration procedure can determine appropriate characteristics (e.g., waveform or signal characteristics) of the ultrasonic transmit signal 802 to improve the audio plethysmography 110 (e.g., to enhance the performance of the voice activity detection 112). The calibration procedure enables the audio plethysmography 110 to take into account the wear of the hearable device 102 (e.g., the position of the hearable device 102 relative to the ear canal 118) and the physical structure of the ear canal 118 to determine a transmission frequency that can increase sensitivity. Using the calibration procedure, the hearable device 102 can dynamically adjust the transmission frequency (e.g., one or more carrier frequencies) each time a seal 120 is formed (e.g., based on the wear of the hearable device 102), based on the unique physical structure of the ear 108. Through this calibration procedure, the hearable device 102 on different ears 108 can operate at one or more different ultrasonic frequencies. The steps of the calibration procedure are further described below.

[0079] In some cases, the hearable device 102 may perform on-head detection (or in-ear detection) by detecting the presence of the seal 120 and initiating a calibration procedure based on a determination that the on-head detection is “true.” In other cases, the hearable device 102 may initiate a calibration procedure based on a specified schedule or timer, which may be controlled by the user 106 via the computing device 104.

[0080] At 1004, the hearable device 102 performs a calibration procedure by transmitting and receiving a first ultrasonic signal. The first ultrasonic signal propagates within at least a portion of the ear canal 118 of the user 106 and has a plurality of tones 902-1 through 902-M (or a plurality of carrier frequencies). The plurality of tones 902-1 through 902-M are transmitted in parallel or serially within a given time interval. The first ultrasonic transmit signal 802 can have a specific bandwidth on the order of a few kilohertz. For example, the ultrasonic transmit signal 802 can have a bandwidth of approximately 4 kHz, 5 kHz, 6 kHz, 8 kHz, 10 kHz, 16 kHz, or 20 kHz. In an example embodiment, the first ultrasonic transmit signal 802 is transmitted over a plurality of seconds, such as 2, 3, 4, 6, or more seconds. The duration of each tone 902 can be evenly divided over the total duration of the first ultrasonic transmit signal 802.

[0081] In an example embodiment, the ultrasonic transmit signal 902 has seven tones 902 (e.g., M equals 7). In some cases, the tones 902 are evenly spaced across the interval. For example, the tones 902 may be in 1 kHz increments between 32 kHz and 38 kHz (e.g., at approximately 32 kHz, 33 kHz, 34 kHz, 35 kHz, 36 kHz, 37 kHz, and 38 kHz). The term "approximately" means that the tones 902 may be within 5% or less of a given value (e.g., within 3%, 2%, or 1% of a given value).

[0082] The amplitude of the ultrasonic transmit signal 802 can be approximately the same across tones 902-1 to 902-M. In this way, power is evenly distributed across each tone 902. The number of tones 902 (e.g., M) can be determined based on the output power of the speaker 708. Increasing the number of tones 902 can increase the likelihood that the hearable device 102 can support voice activity detection 112 across various conditions, including the physical structure of the user's wear and the ear canal 118 of the user 106. However, based on the output power of the speaker 708, the amplitude of the ultrasonic transmit signal 802 can be limited across these tones 902. Therefore, the number of tones 902 can be optimized based on the amount of output power available for audio plethysmography 110.

[0083] At 1006, the calibration procedure selects one or more tones 904-1 through 904-N to be used for the measurement procedure based on the one or more modified characteristics of the ultrasonic receive signal 804. Figure 11 The process for selecting a tone 904 is further described. Generally speaking, the calibration procedure determines that the selected tone 904 improves the signal-to-noise ratio of the audio plethysmography 110 (or more specifically, the voice activity detection 112).

[0084] At 1008, the hearable device 102 performs a measurement procedure using the measurement module 720. According to the measurement procedure, the hearable device 102 transmits a second ultrasonic transmit signal 802 that propagates within at least a portion of the ear canal 118 of the user 106. If a calibration procedure was performed, the second ultrasonic transmit signal 802 may include selected tones 904-1 through 904-M as determined by the calibration procedure. The selected tones 904 may be transmitted in parallel or serially within a given time interval.

[0085] The amplitude of the second ultrasonic transmit signal 802 can be substantially the same across the selected tones 904-1 through 904-N. In this manner, power is evenly distributed across each selected tone. The amplitude of the second ultrasonic transmit signal 802 can be higher than the amplitude of the first ultrasonic transmit signal 802 because the available output power is distributed across fewer tones. Additionally or alternatively, the duration of each of the selected tones 904 of the second ultrasonic transmit signal 802 can be longer than the duration of the tones 902 of the first ultrasonic transmit signal 802. Higher amplitudes and / or longer durations can further improve the signal-to-noise performance of the hearable device 102 for audio plethysmography 110. By using several selected tones 904 determined to improve signal-to-noise performance, the measurement procedure can achieve higher accuracy for voice activity detection 112.

[0086] At 1012, the hearable device 102 performs audio plethysmography (eg, speech processing) using the second ultrasonic signal (eg, the second ultrasonic receive signal 804). One aspect of performing audio plethysmography 110 may include performing internal noise source filtering 202, such as with respect to Figure 12 and Figure 13 Further described. Figure 11 The calibration module 722 is further described.

[0087] Figure 11 An example scheme implemented by the calibration module 722 is shown. In the depicted configuration, the calibration module 722 implements a frequency selector that selects one or more tones 904 for the measurement procedure. In an example embodiment, the calibration module 722 includes at least one amplitude detector 1102, at least one phase detector 1104, at least one quality detector 1106, and at least one comparator 1108. The operation of these components will be described further below.

[0088] During the calibration procedure, the calibration module 722 receives the pre-processed signal 910 from the pre-processing module 718, as previously described with respect to FIG. Figure 9The pre-processed signal 910 may include amplitude and / or phase information associated with the plurality of tones 902-1 through 902-M, which has been used to transmit Figure 10 The first ultrasonic signal described at 1002 in .

[0089] In this example, the calibration module 722 uses the amplitude detector 1102 to extract the amplitude 1110 of the preprocessed signal 910 and uses the phase detector 1104 to extract the phase 1112 of the preprocessed signal 910. Alternatively, if the in-phase and quadrature components of the preprocessed signal 910 are received separately, the amplitude detector 1102 and the phase detector 1104 can measure the amplitude 1110 and the phase 1112 based on the in-phase and quadrature components, respectively.

[0090] The quality detector 1106 measures quality metrics 1114-1 to 1114-2M for each of the tones 902-1 to 902-M and each of the characteristics (e.g., amplitude 1110 and phase 1112). In general, the quality metrics 1114 can represent a variety of different metrics, including peak-to-average ratio and / or signal-to-noise ratio. The peak-to-average ratio represents the peak intensity within a frequency range of interest divided by the average intensity within that frequency range. A higher quality metric 1114 indicates a higher quality signal, or more generally, better performance of the audio plethysmography 110.

[0091] In one aspect, the comparator 1108 can evaluate the quality metrics 1114-1 through 1114-2M relative to a threshold 1116. The threshold 1116 can be set, for example, to a specific value. In other cases, the calibration module 722 can dynamically determine the threshold 1116 and update the threshold over time based on the observed quality metrics 1114-1 through 1114-2M. In an example embodiment, the comparator 1108 determines the selected tones 904-1 through 904-N for subsequent measurement procedures based on the frequencies associated with the quality metrics 1114-1 through 1114-M that are greater than or equal to the threshold 1116.

[0092] Additionally or alternatively, the comparator 1108 can evaluate the quality metrics 1114-1 through 1114-2M relative to each other. In an example embodiment, the comparator 1108 determines one of the selected tones 904 based on the frequency having the highest quality metric 1114 across the amplitude 1110. Furthermore, the comparator 1108 can determine one of the selected tones 904 based on the frequency having the highest quality metric 1114 across the phase 1112. In other embodiments, the comparator 1108 can determine a single selected tone 904 based on the frequency having the highest quality metric 1114 associated with the amplitude 1110 or the phase 1112.

[0093] In general, the calibration module 722 enables the selected tones 904-1 to 904-N to be dynamically adjusted prior to the measurement procedure based on the current environment, which can take into account the wear of the hearable device 102 (e.g., current insertion depth and / or rotation), the physical structure of the ear canal 118 of the user 106, and the response characteristics of the hearable device 102 (e.g., speaker, microphone, and / or housing). In this way, the calibration module 722 can improve the signal-to-noise performance of the hearable device 102 for the measurement procedure. The calibration module 722 can also determine which tones 904 generate ultrasonic receive signals 804 having desirable characteristics for voice activity detection 112. In general, the calibration procedure can be performed regardless of whether the user 106 is speaking.

[0094] exist Figures 9 to 11 , the calibration procedure and the measurement procedure are described as individual procedures that occur at different time intervals. In particular, the calibration procedure occurs before the measurement procedure. This enables the ultrasonic transmit signal 802 used for the measurement procedure to be transmitted with fewer tones than the ultrasonic transmit signal 802 used for the calibration procedure, which can increase the signal-to-noise performance of the audio plethysmography 110. However, in some embodiments, the hearable device 102 can have sufficient output power to perform the measurement procedure using a single ultrasonic transmit signal 802 using multiple tones 902-1 to 902-M. In this case, aspects of the calibration module can be integrated within the pre-processing module 718 as a frequency selector, which will be relevant to Figure 12 The frequency selector can effectively pass the selected tones 904-1 to 904-N for further processing. Figure 12 Aspects of the measurement procedure are further described. Internal noise source filtering

[0095] Figure 12 An example implementation of a pre-processing module 718 is shown. In the depicted configuration, the hearable device 102 includes a pre-processing module 718 coupled to a measurement module 720 and a calibration module 722. The measurement module 720 is also coupled to the microphone 710 (not shown) and the circuitry 304 (not shown).

[0096] The pre-processing module 718 includes at least one in-phase and quadrature mixer 1202 (I / Q mixer 1202) and at least one filter 1204. The in-phase and quadrature mixer 1202 performs frequency down-conversion. In an exemplary embodiment, the in-phase and quadrature mixer 1202 includes at least two mixers, at least one phase shifter, and at least one combiner (e.g., a summing circuit). The filter 1204 attenuates intermodulation products generated by the in-phase and quadrature mixer 1202. In an exemplary embodiment, the filter 1204 is implemented using a low-pass filter.

[0097] The pre-processing module 718 may optionally include at least one frequency selector 1206. The frequency selector 1206 may identify and select one or more tones 904 (or carrier frequencies) that provide a high quality signal for subsequent processing. The frequency selector 1206 may further pass the selected tones 904 to other processing modules and filter (or attenuate) other tones that were not selected. The frequency selector 1206 may be used in conjunction with Figure 11 For example, the frequency selector 1206 may include an amplitude detector 1102 , a phase detector 1104 , a quality detector 1106 , and a comparator 1108 .

[0098] During operation, in-phase and quadrature mixer 1202 uses a phase shifter and two mixers to generate in-phase and quadrature components associated with digital receive signal 908. Specifically, in-phase and quadrature mixer 1202 mixes digital receive signal 908 with a first version of digital transmit signal 906 having a zero-degree phase shift to generate an in-phase component. Furthermore, in-phase and quadrature mixer 1202 mixes digital receive signal 908 with a second version of digital transmit signal 906 having a 180-degree phase shift to generate a quadrature signal. This mixing operation downconverts digital receive signal 908 from acoustic frequencies to baseband frequencies. Using a combiner, in-phase and quadrature mixer 1202 combines the in-phase and quadrature components of digital receive signal 908 to generate downconverted signal 1208. The use of in-phase and quadrature mixer 1202 can further improve the signal-to-noise ratio of downconverted signal 1208 compared to other mixing techniques.

[0099] In this example, downconverted signal 1208 represents a combination of the in-phase and quadrature components of downmixed digital received signal 908. In an alternative embodiment, in-phase and quadrature mixer 1202 does not include a combiner and passes the in-phase and quadrature components separately to filter 1204. In this manner, the in-phase and quadrature components propagate through filter 1204 individually.

[0100] Filter 1204 generates a filtered signal 1210 based on downconverted signal 1208. In particular, filter 1204 filters downconverted signal 1208 to attenuate spurious or undesired frequencies (e.g., intermodulation products), some of which may be associated with the operation of in-phase and quadrature mixer 1202. In this example, filtered signal 1210 represents a combination of in-phase and quadrature components of downconverted signal 1208. Alternatively, filtered signal 1210 may represent separate or distinct in-phase and quadrature components that are individually passed to frequency selector 1206, calibration module 722, or measurement module 720.

[0101] During the measurement procedure, the pre-processing module 718 may optionally apply a frequency selector 1206. The frequency selector 1206 passes tones that meet a quality threshold level for the performance of the audio plethysmography 110. For example, the frequency selector 1206 passes tones 904 having an amplitude 1110 and / or phase 1112 with a quality metric 1114 that is greater than or equal to a threshold 1116. The resulting signal output by the frequency selector 1206 is represented by signal 1212. In some embodiments, this signal 1212 is passed to the measurement module 720 as the pre-processed signal 910. In other embodiments in which the frequency selector 1206 is not implemented, the filtered signal 1210 may be passed to the measurement module 720 and / or calibration module 722 as the pre-processed signal 910.

[0102] In general, the measurement module 720 can generate speech data 912 based on the pre-processed signal 910. In this case, the measurement module 720 can analyze changes in the amplitude 1110 and / or phase 1112 of the speech component 404 of the pre-processed signal 910 for speech processing. This processing technique can be utilized in embodiments of the hearable device 102 with minimal (if any) interference between the air signal 402 and the ultrasonic signal 410, or in situations where the audio plethysmography 110 is performed in a relatively quiet environment.

[0103] However, to handle noisy environments, the measurement module 720 can generate speech data 912 based on the preprocessed signal 910 and the received audible signal 504. More specifically, the measurement module 720 can utilize the received audible signal 504 as a reference to attenuate the modulation component 510 within the preprocessed signal 910 and improve the signal-to-noise ratio associated with the speech component 404.

[0104] To handle situations where internal noise sources 312 operate during audio plethysmography 110, measurement module 720 may also apply internal noise source filtering 202. Utilizing internal noise source filtering 202, measurement module 720 utilizes noise signal 308 as a reference to attenuate internal noise components 310 within pre-processed signal 910 and / or within received audible signal 504. Figure 13 An example implementation of a measurement module 720 that may perform internal noise source filtering 202 is further described.

[0105] Figure 13An example embodiment of a measurement module 720 for performing internal noise source filtering 202 and speech processing is shown. In the depicted configuration, the measurement module 720 includes at least one multistage filter 1302 and at least one speech processor 1304 (or speech processor). The multistage filter 1302 is coupled between the speech processor 1304 and the pre-processing module 718. Although not explicitly shown, the speech processor 1304 can be coupled to other components of the hearable device 102, such as the communication interface 704.

[0106] The speech processor 1304 can perform aspects of voice activity detection 112, speech recognition 114, and / or dialogue detection 116. In an example embodiment, the speech processor 1304 can be implemented using a machine-learned model or another module that performs signal and / or data processing. Generally speaking, the speech processor 1304 can analyze changes in the amplitude 1110 and / or phase 1112 of the speech component 404 of the pre-processed signal 910 for speech processing.

[0107] To enable the speech processor 1304 to detect and / or utilize the speech component 404, the multi-stage filter 1302 filters the pre-processed signal 910 to improve the signal-to-noise ratio associated with the speech component 404. In this example, the multi-stage filter 1302 includes at least one internal noise source filter stage 1306 and at least one external noise source filter stage 1308. Other embodiments are possible in which the multi-stage filter 1302 includes the internal noise source filter stage 1306 and does not include the external noise source filter stage 1308. Such embodiments are possible in situations in which audio plethysmography is performed in a relatively quiet environment and / or for embodiments in which the design of the hearable device 102 minimizes the effects of the modulation component 510.

[0108] Internal noise source filter stage 1306 performs internal noise source filtering 202 to attenuate internal noise components 310 within pre-processed signal 910 and / or received audible signal 504. Internal noise source filter stage 1306 may be implemented using at least two adaptive filters 1310. Figure 14 An example implementation of the internal noise source filter stage 1306 is further described.Other implementations are possible in which the internal noise source filter stage 1306 is implemented using a machine-learned model.

[0109] The external noise source filter stage 1306 uses the received audible signal 504 to attenuate the modulation component 510 within the pre-processed signal 910. The external noise source filter stage 1308 can be implemented using at least one filter module 1312. Figure 15An example implementation of the external noise source filter stage 1308 is further described.Other implementations are possible in which the external noise source filter stage 1308 is implemented using a machine-learned model.

[0110] During operation, the measurement module 720 receives a preprocessed signal 910 from the preprocessing module 718. The preprocessed signal 910 can include the internal noise component 310 and the speech component 404. In some embodiments, the preprocessed signal 910 also includes the modulation component 510. The internal noise component 310 and the modulation component 510 can make it challenging to use the preprocessed signal 910 directly for speech processing. In some cases, because the noise signal 308 is amplified within the ear canal 118, the internal noise component 310 can have a greater impact on the noise level of the preprocessed signal 910 than the modulation component 510.

[0111] The measurement module 720 may also accept the received audible signal 504 from the microphone 710. The received audible signal 504 may include the internal noise component 310, the speech component 404, and the bone conduction component 512. In a noisy environment, the received audible signal 504 may also include the noise component 406.

[0112] The internal noise source filter stage 1306 operates on the preprocessed signal 910 and, optionally, the received audible signal 504 to attenuate the internal noise component 310. The external noise source filter stage 1308 operates on the output of the internal noise source filter stage 1308 to attenuate the modulation component 510 within the preprocessed signal 910. The output of the external noise source filter stage 1308 is represented by a denoised signal 1314. The denoised signal 1314 represents a filtered version of the preprocessed signal 910 that includes the speech component 404.

[0113] The speech processor 1304 analyzes the denoised signal 1314 to generate speech data 912. In particular, the speech processor 1304 can generate speech data 912 associated with voice activity detection 112, speech recognition 114, and / or conversation detection 116. For voice activity detection 112, the speech data 912 can indicate whether a speech component 404 is detected. In some examples, the speech processor 1304 can perform a signal-to-noise ratio detection process that determines whether the amplitude of the input signal exceeds a detection threshold. If the amplitude exceeds the detection threshold, the speech processor 1304 generates speech data 912 to indicate that a vocalization was detected. Otherwise, if the amplitude does not exceed the detection threshold, the speech processor 1304 generates speech data 912 to indicate that a vocalization was not detected. Voice activity detection 112, speech recognition 114, and / or conversation detection 116 can be used in a variety of different ways to control the operation of the hearable device 102 and / or computing device 104. About Figure 14 and Figure 15 An example implementation of the multi-stage filter 1302 is further described.

[0114] Figure 14 An example implementation of an internal noise source filter stage 1306 is shown. In the depicted configuration, the internal noise source filter stage 1306 includes two adaptive filters 1310-1 and 1310-2. Adaptive filters 1310-1 and 1310-2 can apply various adaptive filtering techniques, including those based on least mean squares (LMS) or recursive least squares (RLS). Adaptive filter 1310-1 uses noise signal 308 as a noise reference to perform adaptive filtering to attenuate internal noise component 310 within preprocessed signal 910. Similarly, adaptive filter 1310-2 uses noise signal 308 as a noise reference to perform adaptive filtering to attenuate internal noise component 310 within received audible signal 504. In this example, noise signal 308 is uncorrelated with desired speech component 404 within preprocessed signal 910 and received audible signal 504. Thus, the noise signal 308 may be used as a reference signal to attenuate the internal noise component 310 within the pre-processed signal 910 and the received audible signal 504 using adaptive filtering techniques.

[0115] Adaptive filters 1310-1 and 1310-2 generate a denoised preprocessed signal 1402 and a denoised audible signal 1404, respectively. Denoised preprocessed signal 1402 includes speech component 404 and modulation component 510. Denoised audible signal 1404 includes speech component 404, noise component 406, and bone conduction component 512. Denoised audible signal 1404 may also be referred to as a denoised signal. The term "denoised audible signal 1404" is used to refer to a denoised version of received audible signal 504 and does not necessarily mean that denoised audible signal 1404 can be heard.

[0116] Generally speaking, the denoised preprocessed signal 1402 and the denoised audible signal 1404 do not include the internal noise component 310, or the internal noise component 310 is substantially attenuated relative to the corresponding input signal. To attenuate the modulation component 510 within the denoised preprocessed signal 1402, the filtering process continues with the external noise source filter stage 1306, which is about Figure 15 Further description.

[0117] Figure 15 An example implementation of an external noise source filter stage 1306 is shown. In the depicted configuration, the external noise source filter stage 1306 includes at least one filter module 1502 and at least one voicing enhancer 1504. The filter module 1502 can be implemented using at least one adaptive filter 1506 or at least one blind-source separator 1508.

[0118] Adaptive filter 1506 performs adaptive filtering using received audible signal 504 as a noise reference to separate speech component 404 from modulation component 510 (or modulated noise component 406) of preprocessed signal 910. Example adaptive filtering techniques utilized by adaptive filter 1506 may include techniques based on least mean squares (LMS) or recursive least squares (RLS). Blind source separator 1508 performs blind source separation (BSS) using received audible signal 504 to separate speech component 404 from modulation component 510 of preprocessed signal 910. To perform adaptive filtering or blind source separation (BSS), preprocessed signal 910 represents a primary reference and received audible signal 504 represents a secondary reference or noise reference. Generally speaking, adaptive filter 1506 and blind source separator 1508 can utilize received audible signal 504 to significantly attenuate modulation component 510 (or modulated noise component 406) within preprocessed signal 910.

[0119] The voicing enhancer 1504 enhances (e.g., amplifies relative to the noise level) the speech component 404 within the output signal provided by the filter module 1502. In this manner, the voicing enhancer 1504 can increase the sensitivity for speech processing. In an example embodiment, the voicing enhancer 1504 is implemented using a Wiener filter 1510. Example Method

[0120] Figure 16 and Figure 17 Depicted are example methods 1600 and 1700 for implementing various aspects of voice activity detection 112 using active acoustic sensing. Methods 1600 and 1700 are shown as a set of operations (or actions) performed, but are not necessarily limited to the order or combination of operations shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, or chained to provide a wide range of additional and / or alternative methods. In portions of the following discussion, reference may be made to FIG. 1 and FIG. Figure 2 Environments 100, 200-1, 200-2, 200-3, and Figure 6 and Figure 7 The entities detailed in the foregoing are referenced by way of example only. The techniques are not limited to being performed by one entity or multiple entities operating on one device.

[0121] At 1602, an audible signal is transmitted during a first time period. The audible signal is intended to propagate within at least a portion of an ear canal of a user. For example, the circuit system 304 transmits (or renders) the audible signal during the first time period. The audible signal is intended to propagate within at least a portion of an ear canal 118 of the user 106. The circuit system 304 may include a speaker 314 or 708, active noise cancellation circuit system 316, or a transparency mode circuit system 318. The audible signal may be used to play audio content 204 for the user 106 (as shown in environment 200-1), provide active noise cancellation 206 (as shown in environment 200-2), or enable the hearable device 102 to operate according to the transparency mode 214 (as shown in environment 200-3). Figure 2 as shown in environment 200-3).

[0122] At 1604, an ultrasonic transmit signal is transmitted during a first time period. The ultrasonic transmit signal is intended to propagate within at least a portion of the user's ear canal. For example, the transducer 706 (or speaker 708) of the hearable device 102 transmits the ultrasonic transmit signal 802. The ultrasonic transmit signal 802 propagates within at least a portion of the ear canal 118 of the user 106, as described with respect to FIG. Figure 4 and Figure 8The ultrasonic transmission signal 802 may have at least two tones 904 (or frequencies). In some embodiments, the plurality of tones 904 may be determined based on a calibration procedure, such as Figure 10 In other embodiments, the plurality of tones 904 provides frequency diversity and can improve the performance of the audio plethysmography 110, as described with respect to Figure 12 As stated.

[0123] At 1606, an ultrasonic receive signal is received. The ultrasonic receive signal represents a version of the ultrasonic transmit signal having one or more waveform characteristics modified based on propagation within the ear canal and based on vocalizations by the user during a first time period. The ultrasonic receive signal includes an internal noise component caused by interference generated by the rendering of the audible signal.

[0124] For example, the transducer 706 (or microphone 710) of the hearable device 102 receives an ultrasonic receive signal 804. The ultrasonic receive signal 804 represents a version of the ultrasonic transmit signal 802 having one or more waveform characteristics modified based on propagation within the ear canal 118 and based on vocalizations made by the user 106 during the first time period. Example waveform characteristics include amplitude, phase, and / or frequency.

[0125] The ultrasonic receive signal 804 includes an internal noise component 310, which is caused by interference generated by the rendering of the audible signal. The internal noise component 310 can represent a portion of the audible signal that is modulated onto or mixed with the ultrasonic receive signal 804. Generally speaking, the internal noise component 310 represents a portion of the ultrasonic receive signal 804 that is modified (e.g., in amplitude, phase, and / or frequency) by interference associated with the audible signal.

[0126] The hearable device 102 that receives the ultrasonic receive signal 804 may be the same hearable device 102 that sent the ultrasonic transmit signal 802 (e.g., Figure 8 ), or another hearable device 102 that has not sent the ultrasonic transmission signal 802 (e.g., Figure 8 In some embodiments, the feedback microphone of the active noise cancellation circuit 724 can receive the ultrasonic receive signal 804.

[0127] At 1608, a denoised signal is generated by filtering the internal noise component within the received ultrasonic receive signal based on a version of the audible signal. For example, the measurement module 720 (e.g., the multi-stage filter 1302) generates a denoised signal by filtering the internal noise component within the received ultrasonic receive signal based on a version of the audible signal. Figure 13The internal noise component 310 within the ultrasonic receive signal 804 is filtered to generate a denoised signal 1314 using a version of the audible signal represented by the noise signal 308 shown in FIG.

[0128] The version of the audible signal can represent a digital version of the audible signal, a version of the audible signal that is down-converted or up-converted to a specific frequency range (e.g., baseband frequency) for denoising, a pre-processed version of the audible signal, or some combination thereof. Generally speaking, the term "version of the audible signal" means that there can be some difference between playing the audible signal for the user 106 and providing the audible signal as the noise signal 308 for the internal noise source filtering process. Explained another way, the version of the audible signal is related to the audible signal, but can be modified to support the operation of the hearable device 102. In this case, the operation involves internal noise source filtering.

[0129] At 1610, an utterance is detected based on the de-noised signal. For example, the hearable device 102 uses the measurement module 720 (e.g., the speech processor 1304) to detect the utterance based on the de-noised signal 1314. More specifically, the measurement module 720 analyzes the speech component 404 present within the de-noised signal 1314 to perform voice activity detection 112, speech recognition 114, and / or dialogue detection 116. The measurement module 720 can generate speech data 912, which can be used to control the hearable device 102 and / or the computing device 104.

[0130] exist Figure 17 At 1702 in the embodiment, active acoustic sensing is performed to detect pressure waves propagating within the ear canal of the user and associated with the user's vocalization. For example, the hearable device 102 performs active acoustic sensing to detect pressure waves propagating within the ear canal 118 of the user 106 and associated with the user's vocalization. To perform active acoustic sensing, the hearable device 102 transmits and receives ultrasonic signals 410 (e.g., ultrasonic transmit signal 802 and ultrasonic receive signal 804). The received ultrasonic signal 410 includes a speech component 404, which enables the audio plethysmography 110 to be used for speech processing.

[0131] At 1704, an audible signal is rendered that interferes with active acoustic sensing. For example, circuit system 304 renders an audible signal that interferes with active acoustic sensing. Circuit system 304 may include speaker 314 or 708, active noise cancellation circuit system 316, and / or transparency mode circuit system 318. The audible signal may be audible signal 320 that includes audio content 204, anti-noise signal 324, and / or transparency mode signal 326.

[0132] At 1706, internal noise source filtering is performed to attenuate interference caused by the rendering of the audible signal. For example, the measurement module 720 performs internal noise source filtering 202 to attenuate interference caused by the rendering of the audible signal (e.g., internal noise component 310). Such interference can be attenuated within the pre-processed signal 910 and optionally the received audible signal 504, as shown in FIG. Figure 14 As shown in .

[0133] At 1708, speech processing is performed based on the active acoustic sensing and internal noise source filtering. For example, the speech processor 1304 performs speech processing (e.g., voice activity detection 112, speech recognition 114, and / or dialogue detection 116) based on the active acoustic sensing and internal noise source filtering (e.g., based on the de-noised pre-processed signal 1402).

[0134] At 1710 , a signal is generated to control the operation of at least one of the hearable device or a computing device coupled to the hearable device. For example, the measurement module 720 generates voice data 912 that can be used to control the operation of the hearable device 102 and / or the computing device 104.

[0135] Throughout the disclosure, the term "version of a signal" is used to indicate that the second signal can be a modified version of the first signal. A "version of a signal" (or second signal) can represent a digital version of the first signal, an analog version of the first signal, an electrical version of the first signal having voltage and current, an acoustic version of the first signal having acoustic properties, a down-converted version of the first signal having a lower frequency range, an up-converted version of the signal having a higher frequency range, a pre-processed version of the signal (e.g., a version whose amplitude, phase, and / or frequency has been modified in some manner), a filtered version of the signal, etc. In general, a "version of a signal" refers to a signal that has been modified using techniques known in the art to facilitate operation of the hearable device 102. Example computing system

[0136] Figure 18 1800, which may be implemented as described with reference to the previous Figure 6 and Figure 7 Any type of client, server, and / or computing device described herein may be used to implement aspects of active acoustic sensing using hearable devices.

[0137] The computing system 1800 includes a communication device 1802 that enables wired and / or wireless transmission of device data 1804 (e.g., received data, data being received, data scheduled for broadcast, or data packets of data). The communication device 1802 or computing system 1800 may include one or more hearable devices 102. The device data 1804 or other device content may include configuration settings for the device, media content stored on the device, and / or information associated with a user of the device. The media content stored on the computing system 1800 may include any type of audio, video, and / or image data. The computing system 1800 includes one or more data inputs 1806 via which any type of data, media content, and / or input may be received, such as human speech, user-selectable input (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0138] Computing system 1800 also includes a communication interface 1808, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. Communication interface 1808 provides a connection and / or communication link between computing system 1800 and a communication network through which other electronic, computing, and communication devices communicate data with computing system 1800.

[0139] The computing system 1800 includes one or more processors 1810 (e.g., any of microprocessors, controllers, etc.) that process various computer-executable instructions to control the operation of the computing system 1800. Alternatively or in addition, the computing system 1800 can be implemented using any one or combination of hardware, firmware, or fixed logic circuitry in conjunction with processing and control circuitry, generally designated 1812. Although not shown, the computing system 1800 can include a system bus or data transfer system that couples the various components within the device. The system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures.

[0140] The computing system 1800 also includes computer-readable media 1814, such as one or more memory devices that implement persistent and / or non-transitory data storage (i.e., as opposed to mere signal transmission). Examples of computer-readable media include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage. The disk storage can be implemented as any type of magnetic or optical storage device, such as a hard drive, a recordable and / or rewritable compact disc (CD), any type of digital versatile disc (DVD), etc. The computing system 1800 can also include a mass storage medium device (storage medium) 1816.

[0141] The computer-readable medium 1814 provides a data storage mechanism to store device data 1804, as well as various device applications 1818 and any other type of information and / or data related to operational aspects of the computing system 1800. For example, an operating system 1820 may be maintained as a computer application with the computer-readable medium 1814 and executed on the processor 1810. The device applications 1818 may also include a device manager, such as any form of control application, software application, signal processing and control module, code native to a particular device, a hardware abstraction layer for a particular device, and the like.

[0142] The device applications 1818 also include any system components, engines, or managers for implementing the internal noise source filtering 202 for active acoustic sensing. In this example, the device applications 1818 include the pre-processing module 718, the measurement module 720, and optionally the calibration module 722. Although not explicitly shown, the device applications 1818 may also include the internal noise source filter stage 1306, the application 606, the voice user interface 608, and / or the voice authenticator 610.

[0143] Throughout this disclosure, examples are described in which a computing system 1800 (e.g., hearable device 102, computing device 104, client device, server device, computer, or another type of computing system) can analyze information (e.g., various audible and / or ultrasonic signals) associated with a user (e.g., vocalization). Furthermore, controls can be provided to the user 106 to allow the user 106 to select whether and when the systems, programs, and / or features described herein can enable the collection of information (e.g., information about the user's social network, social actions, social activities, occupation, user preferences, user's current location), as well as whether to send content or communications from the server to the user 106. The computing system 1800 can be configured to use the information only after the computing system 1800 receives explicit permission from the user 106 to use the data. For example, in the case where the hearable device 102 analyzes signals for voice activity detection 112, speech recognition 114, and / or conversation detection 116, the individual user 106 can be provided with an opportunity to provide input to control whether the program or feature of the computing system 1800 can collect and utilize the data. Further, individual users 106 may have ongoing control over what programs may or may not process information.

[0144] Furthermore, the collected information may be pre-processed in one or more ways before being transmitted, stored, or otherwise used so that personally identifiable information is removed. For example, before computing system 1800 shares data with another device, the identity of user 106 may be processed so that no personally identifiable information can be determined for user 106. Thus, user 106 may have control over whether information about user 106 and user 106's device is collected, and if so, how computing system 1800 and / or remote computing systems may use such information. in conclusion

[0145] Although techniques for and apparatuses including those for performing internal noise source filtering for active acoustic sensing have been described in language specific to features and / or methods, it should be understood that the subject matter of the accompanying examples is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of internal noise source filtering for active acoustic sensing.

[0146] Some examples are provided below.

[0147] Example 1: A method comprising: transmitting an audible signal during a first time period, the audible signal propagating within at least a portion of an ear canal of a user; transmitting an ultrasonic transmit signal during the first time period, the ultrasonic transmit signal propagating within at least a portion of the ear canal of the user; receiving an ultrasonic receive signal during the first time period, the ultrasonic receive signal representing a version of the ultrasonic transmit signal having one or more characteristics modified based on the propagation within the ear canal and based on vocalizations by the user during the first time period, the received ultrasonic receive signal including an internal noise component caused by interference generated by the rendering of the audible signal; generating a de-noised signal by filtering the internal noise component within the received ultrasonic receive signal based on a version of the audible signal; and The utterance is detected based on the de-noised signal.

[0148] Example 2: The method according to Example 1 further includes: Operation of the hearable device and / or operation of a computing device coupled to the hearable device is controlled based on the detected vocalization.

[0149] Example 3: The method of Example 1 or 2, wherein: detecting the utterance comprises performing speech processing using the de-noised signal; and The method further includes controlling operation of the hearable device and / or operation of a computing device coupled to the hearable device based on the speech processing.

[0150] Example 4: The method of Example 3, wherein controlling the hearable device comprises at least one of: Pause or resume rendering of audio content; adjusting the volume of the audio content being rendered; Enable or disable active noise cancellation; or Enable or disable Transparency mode.

[0151] Example 5: The method of Example 3 or 4, wherein performing the speech processing comprises at least one of: Perform voice activity detection; Perform speech recognition; or Perform conversation detection.

[0152] Example 6: The method of any preceding example, wherein the audible signal comprises audio content requested by the user.

[0153] Example 7: The method of Example 6, wherein the audio content comprises music.

[0154] Example 8: The method according to any of the preceding examples, further comprising: performing active noise cancellation using the audible signal, Wherein the audible signal comprises an anti-noise signal.

[0155] Example 9: The method according to any of the preceding examples, further comprising: Operate your hearable device according to the transparency mode. The audible signal includes a transparency mode signal that provides sound from an external environment to the ear canal of the user.

[0156] EXAMPLE 10 The method of any preceding example, wherein generating the de-noised signal comprises filtering the received version of the ultrasonic receive signal based on the version of the audible signal using an adaptive filter.

[0157] Example 11: The method of Example 10, further comprising: receiving an airborne signal comprising audible frequencies during the first time period, the audible frequencies comprising the vocalization made by the user during the first time period, wherein: The received ultrasonic reception signal includes a modulation component caused by interference generated by receiving the air signal; and Generating the de-noised signal further includes attenuating the modulation component within the received ultrasonic receive signal using the received over-the-air signal.

[0158] Example 12: The method of Example 11, wherein: The over-the-air signal includes the internal noise component; and Generating the denoised signal comprises: generating a de-noised version of the received ultrasonic receive signal by filtering the version of the received ultrasonic receive signal based on the version of the audible signal using the adaptive filter; generating a de-noised audible signal by filtering a version of the over-the-air signal based on the version of the audible signal using another adaptive filter; and The denoised signal is generated by filtering the denoised version of the received ultrasonic receive signal based on the denoised audible signal.

[0159] EXAMPLE 13 The method of any one of Examples 11 or 12, wherein receiving the ultrasonic receive signal and receiving the air-to-air signal comprises receiving the ultrasonic receive signal and the air-to-air signal using the same microphone.

[0160] Example 14: A method according to any preceding example, wherein vocalization comprises speech, humming, whistling or singing.

[0161] EXAMPLE 15 The method of any preceding example, wherein transmitting the ultrasonic transmit signal comprises transmitting the ultrasonic transmit signal having at least two tones.

[0162] Example 16: A computer-readable storage medium comprising instructions that, in response to being executed by a processor, cause a hearable device to perform any of the methods of Examples 1 to 15.

[0163] Example 17: An apparatus comprising: at least one transducer; and at least one processor, the apparatus being configured to perform any one of the methods according to Examples 1 to 15 using the at least one transducer and the at least one processor.

[0164] Example 18: The apparatus of Example 17, further comprising: Speakers; and an active noise cancellation circuit, the active noise cancellation circuit comprising a feedback microphone, The at least one transducer includes the speaker and the feedback microphone.

[0165] Example 19: The apparatus of Example 17, wherein: The at least one transducer includes a speaker and a microphone; The speaker is configured to be positioned proximate to a first ear of a user; and The microphone is configured to be positioned proximate to a second ear of the user.

[0166] Example 20: The apparatus of any one of Examples 17 to 19, wherein the apparatus comprises: At least one pair of earbuds; or Headphones.

Claims

1. A method comprising: transmitting an audible signal during a first time period, the audible signal propagating within at least a portion of an ear canal of a user; transmitting an ultrasonic transmit signal during the first time period, the ultrasonic transmit signal propagating within at least a portion of the ear canal of the user; receiving an ultrasonic receive signal during the first time period, the ultrasonic receive signal representing a version of the ultrasonic transmit signal having one or more characteristics modified based on the propagation within the ear canal and based on vocalizations by the user during the first time period, the received ultrasonic receive signal including an internal noise component caused by interference generated by the rendering of the audible signal; generating a de-noised signal by filtering the internal noise component within the received ultrasonic receive signal based on a version of the audible signal; as well as The utterance is detected based on the de-noised signal.

2. The method according to claim 1, further comprising: Operation of the hearable device and / or operation of a computing device coupled to the hearable device is controlled based on the detected vocalization.

3. The method according to claim 1 or 2, wherein: detecting the utterance comprises performing speech processing using the de-noised signal; and The method further includes controlling operation of the hearable device and / or operation of a computing device coupled to the hearable device based on the speech processing.

4. The method of claim 3, wherein controlling the hearable device comprises at least one of: Pause or resume rendering of audio content; adjusting the volume of the audio content being rendered; Enable or disable active noise cancellation; or Enable or disable Transparency mode.

5. The method of claim 3 or 4, wherein performing the speech processing comprises at least one of: Perform voice activity detection; Perform speech recognition; or Perform conversation detection.

6. A method according to any preceding claim, wherein the audible signal comprises audio content requested by the user. The method of claim 6 , wherein the audio content comprises music.

8. The method according to any preceding claim, further comprising: performing active noise cancellation using the audible signal, Wherein the audible signal comprises an anti-noise signal.

9. The method according to any preceding claim, further comprising: Operate your hearable device according to the transparency mode. The audible signal includes a transparency mode signal that provides sound from an external environment to the ear canal of the user.

10. A method according to any preceding claim, wherein generating the de-noised signal comprises filtering the received version of the ultrasonic receive signal based on a version of the audible signal using an adaptive filter.

11. The method according to claim 10, further comprising: receiving an airborne signal comprising audible frequencies during the first time period, the audible frequencies comprising the vocalization made by the user during the first time period, wherein: The received ultrasonic reception signal includes a modulation component caused by interference generated by receiving the air signal; and Generating the de-noised signal further includes attenuating the modulation component within the received ultrasonic receive signal using the received over-the-air signal.

12. The method according to claim 11, wherein: The airborne signal includes the internal noise component; and Generating the denoised signal comprises: generating a de-noised version of the received ultrasonic receive signal by filtering the version of the received ultrasonic receive signal based on the version of the audible signal using the adaptive filter; generating a de-noised audible signal by filtering a version of the over-the-air signal based on the version of the audible signal using another adaptive filter; and The denoised signal is generated by filtering the denoised version of the received ultrasonic receive signal based on the denoised audible signal. 13 . The method according to claim 11 , wherein receiving the ultrasonic reception signal and receiving the air signal comprise receiving the ultrasonic reception signal and the air signal using the same microphone.

14. A method according to any preceding claim, wherein the vocalisation comprises speech, humming, whistling or singing.

15. The method of any preceding claim, wherein transmitting the ultrasonic transmit signal comprises transmitting the ultrasonic transmit signal having at least two tones. 16 . A computer-readable storage medium comprising instructions, the instructions, in response to being executed by a processor, causing a hearable device to perform any one of the methods according to claim 1 .

17. An apparatus comprising: at least one transducer; and at least one processor, the apparatus being configured to perform any one of the methods according to claims 1 to 15 using the at least one transducer and the at least one processor.

18. The apparatus according to claim 17, further comprising: speaker; and an active noise cancellation circuit, the active noise cancellation circuit comprising a feedback microphone, The at least one transducer includes the speaker and the feedback microphone.

19. The apparatus of claim 17, wherein: The at least one transducer includes a speaker and a microphone; The speaker is configured to be positioned proximate to a first ear of a user; and The microphone is configured to be positioned proximate to a second ear of the user.

20. The device according to any one of claims 17 to 19, wherein the device comprises: At least one pair of earbuds; or Headphones.

Images