System and method for reducing the effects of water retention in microphone ports

Abstract

A system is provided that corrects audio received by an audio sensor that is distorted by the presence of water in a port associated with the audio sensor. More specifically, a wearable computing device can include an audio sensor that can detect an audio signal in the vicinity of the wearable computing device. To enable the audio sensor to more accurately capture audio signal data, the wearable computing device can include a port (e.g., a microphone or "mic" port) that connects the audio sensor to the outside of the wearable computing device. However, in some situations, the port may be partially or entirely filled with water. For example, when a user is swimming while wearing the wearable computing device, the wearable computing device may be submerged in water and the port may be filled with water.

Description

[Technical field]

[0001] Claiming priority This application claims priority to U.S. Provisional Application No. 63 / 295,222, filed December 30, 2021, the entire contents of which are hereby incorporated by reference into this patent application.

[0002] The present disclosure relates generally to removing distortion from sensors in wearable computing systems, and more specifically, to systems and methods for mitigating the effects that water standing in a microphone port has on audio data captured by an audio sensor. [Background technology]

[0003] Advances in wearable technology, such as fitness trackers and smart watches, have allowed these devices to be used in a variety of situations. For example, fitness trackers and smart watches can be used in situations where water may get into ports associated with microphones or other audio sensors without damaging the device. However, when water partially or completely blocks a port associated with an audio sensor, the audio data generated by that sensor may be distorted. This distortion may reduce the device's ability to accurately analyze and respond to detected audio. Summary of the Invention

[0004] Aspects and advantages of embodiments of the present disclosure are set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments. One exemplary aspect of the present disclosure is a method directed to compensating for distortions introduced into audio data caused by water in a port associated with an audio sensor. The method includes accessing port status data by a wearable computing device including one or more processors. The method further includes determining, by the wearable computing device, a water presence likelihood value representative of a likelihood that water is currently present in the port based on the port status data. The method further includes applying, by the wearable computing device, a digital correction filter to data generated by the audio sensor in accordance with a determination that the water presence likelihood value exceeds a pre-determined threshold, the digital correction filter correcting the data generated by the audio sensor to compensate for the presence of water in the port.

[0005] Another exemplary aspect of the present disclosure is directed to a wearable computing device, the wearable computing device including one or more processors, an audio sensor, a port connecting the audio sensor to an exterior of a surface of the wearable computing device, and a non-transitory computer-readable memory storing instructions that, when executed by the one or more processors, cause the wearable computing device to perform an operation, the operation including accessing port status data, the operation further including determining a water presence likelihood value representative of a likelihood that water is currently present in the port based on the port status data, the operation including applying a digital correction filter to data generated by the audio sensor in accordance with a determination that the water presence likelihood value exceeds a pre-determined threshold, the digital correction filter correcting the data generated by the audio sensor to compensate for the presence of water in the port.

[0006] Another example aspect of the present disclosure is directed to a non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations including generating a water presence signal indicative of the presence of water in the port, and further including, in response to the water presence signal, applying a digital correction filter to data generated by an audio sensor, the digital correction filter correcting the data generated by the audio sensor to compensate for the presence of water in the port.

[0007] Other example aspects of the present disclosure are directed to systems, apparatus, computer program products (such as tangible non-transitory computer readable media, and such as software downloadable over a communications network without necessarily being stored in a non-transitory form), user interfaces, memory devices, and electronic devices for implementing and utilizing user computing devices.

[0008] These and other features, aspects, and advantages of various embodiments will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0009] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates an exemplary wearable computing device according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a block diagram of an exemplary computing environment including a wearable computing device having a sensor according to an exemplary embodiment of the present disclosure. [Diagram 3]1 illustrates an exemplary wearable computing system, according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating an example system for reducing audio distortion in a wearable computing device, according to an example embodiment of the present disclosure. [Diagram 5] FIG. 1 illustrates a block diagram of an exemplary water likelihood model according to an exemplary embodiment of the present disclosure. [Figure 6] 4 is a flowchart illustrating an example process for mitigating distortion in a signal generated by an audio sensor, according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Reference will now be made in detail to the embodiments, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the embodiments and is not intended as a limitation of the disclosure. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments without departing from the scope or spirit of the disclosure. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield still a further embodiment. Accordingly, it is intended that aspects of the disclosure cover such modifications and variations.

[0012] In general, the present disclosure is directed to a system that corrects audio received by an audio sensor that is distorted by the presence of water in a port associated with the audio sensor. More specifically, a wearable computing device may include an audio sensor that can detect an audio signal in the vicinity of the wearable computing device. To enable the audio sensor to more accurately capture audio signal data, the wearable computing device may include a port (e.g., a microphone or "mic" port) that connects the audio sensor to the outside of the wearable computing device. However, in some situations, the port may be partially or entirely filled with water. For example, when a user is swimming while wearing the wearable computing device, the wearable computing device may be submerged in water and the port may be filled with water.

[0013] When water fills the port, the audio signal generated by the audio sensor may be distorted. To compensate for distortion from water in the port, the wearable computing device may use one or more digital correction filters to remove or reduce the distortion. However, to ensure that the digital correction filters are not used when there is no water in the port, the wearable computing device may determine whether water is currently present in the port, or at least whether there is a high probability that water is present in the port.

[0014] The wearable computing device can determine whether the port contains water based on one or more factors, which may include, but are not limited to, information collected by sensors included in the wearable computing device, activity information recorded by or automatically detected for the user, attempts by the user to remove water from the speaker, evaporation rates around the wearable computing device, and / or distortions detected in data generated by audio sensors.

[0015] These factors can be used to generate feature data. The feature data can include a number of features corresponding to the factors and can be used in determining whether water is present in the port. For example, the water determination system can use the feature data as an input to an algorithm or machine learning model. The algorithm or machine learning model can be configured to generate a water likelihood score. The water likelihood score can represent the likelihood of water being present in the port based on the features. When the water likelihood score exceeds a threshold, the wearable computing device can apply a digital correction filter to the data generated by the audio sensor to compensate for the presence of water in the port. In some examples, parameters of the filter can be updated over time as the likelihood of the continued presence of water in the port decreases or the amount of water in the port decreases. For example, the decrease in the likelihood of the continued presence of water in the port may automatically decay over time according to a decay schedule or parameters, or the likelihood of the continued presence of water in the port may decrease as the algorithm or machine learning model evaluates new input feature data over time.

[0016] In one particular example, a user may record a swimming activity and then select a "water removal" function from the wearable computing device. The wearable computing device may determine that there is a high likelihood of water in the port based on the recorded swimming activity and the user's selection of the water removal function (which typically functions to remove water from the speaker). As a result, the wearable computing device may apply a digital correction filter to the audio data generated by the audio sensor based on the amount of time that has elapsed since the swimming activity was recorded and the current evaporation rate associated with the area in which the wearable computing device is located.

[0017] More specifically, a wearable computing device may include any computing device integrated into an object intended to be worn by a user. For example, a wearable computing device may include, but is not limited to, a computing device integrated into a smart watch, a fitness band, a jewelry such as a smart ring or a smart necklace, a computing device integrated into an item of clothing such as a jacket, shoes, pants, or wearable glasses with embedded computing elements. In some examples, a wearable computing device may include one or more sensors intended to collect information with the permission of a user wearing the wearable computing device.

[0018] In some embodiments, the sensor may include an audio sensor, such as one or more microphones. In order to capture the audio information surrounding the wearable computing device as accurately as possible, the wearable computing device may have a port that connects the audio sensor (internal to the wearable computing device) to the outside of the wearable computing device. This port may also be called a microphone port. This port may allow the audio signal to reach the audio sensor more clearly. However, if the audio port is blocked, the quality of the received audio signal may be reduced or the audio signal itself may be changed.

[0019] In some examples, the wearable computing device can be made to safely come into contact with water (e.g., a waterproof or water-resistant device). In this case, the presence of water does not damage the device, but water may fill or at least partially block the port, thereby altering the quality of the audio data collected by the audio sensor. Thus, the wearable computing device can improve the quality of the audio data by mitigating the presence of water when it fills the port.

[0020] To effectively mitigate the presence of water, the wearable computing device can determine whether water is present in the port. Determining whether water is present in the port can be based on one or more signals accessed by the wearable computing device. For example, the wearable computing device can include an additional sensor operable to determine whether water is present in the port. One such sensor can be a humidity sensor (or water sensor or moisture sensor). The humidity sensor can measure humidity in an area surrounding the wearable computing device, and based in part on the humidity data, the wearable computing device can estimate whether water is present in the port. In some examples, the wearable computing device can include multiple ports, and the humidity sensor can have access to a different port than the audio sensor.

[0021] Another possible sensor could be one or more electrodes that can directly detect the presence of water in the port. For example, the electrodes may be connected to portions of the port and may respond to the presence of water in such a way that an electrical signal is generated (e.g., the presence of water in the port may promote increased transmission of an electrical signal between the electrodes). The electrical signal can be analyzed along with other data to determine the presence of water in the port.

[0022] In addition to sensor data, the wearable computing device may have access to data provided by the user. For example, the wearable computing device may have an application that allows the user to record activities that the user performs. In other examples, algorithms or machine learning models may be used to automatically detect activities that the user performs based on various device data, such as sensor data (e.g., data from an accelerometer, etc.). Based on the recorded or detected activity, the wearable computing device may determine whether water may be present in the port. For example, the wearable computing device may determine that there is a high probability that water is present in the port if the user recently recorded a swimming activity. This activity record may be combined with motion detection of the wearable computing device itself to determine whether the user was wearing the wearable computing device while performing the swimming activity.

[0023] In some examples, a wearable computing device may include a feature intended to help remove water from a speaker included in the wearable computing device. For example, a wearable computing device such as a smart watch or fitness band may include a "water removal" feature that provides a low signal to the speaker to cause it to vibrate or otherwise mechanically remove water from the speaker. If a user determines that water is present in a microphone speaker, the user may select this feature. For example, the user may notice that the speaker is producing poor quality audio that is muffled or otherwise altered by water in or near the speaker. If the user selects this feature, the wearable computing device may determine that there is a relatively high likelihood that water is also present in the microphone port.

[0024] In some examples, the wearable computing device can use weather data accessible over a network and user location data (e.g., based on the wearable computing device's GPS signal) to determine whether the user was in a location where rain may have been encountered. For example, if a forecast for a particular area suggests heavy rain and the user's location data indicates that the user was in that area, the wearable computing device can determine that there is a high probability that water is present in the port.

[0025] In some examples, the wearable computing device may also analyze the audio signal generated by the audio sensor to determine whether the audio signal appears to be distorted in a manner associated with water in the port. For example, the wearable computing device may include or have access to data representative of the types of distortions characteristically caused by the presence of water. This data, or other data representative of how an audio signal may be distorted by the presence of water, may be used to analyze the data generated by the audio sensor to determine whether the data includes evidence of distortion due to water in the port.

[0026] For example, the wearable computing device can capture audio data from background noise and analyze it for evidence of water-based distortion. Specific indicators of water-based distortion may include, but are not limited to, background noise being very quiet (e.g., compared to previously captured background noise), low volume at high frequencies, muffled quality of the background sound, etc. In some examples, to accurately analyze distortion of the captured background sound, the wearable computing device can collect long-term average sound data over time. In some examples, different long-term averages (or baselines) can be captured for different locations (which may have different acoustic characteristics) frequented by the user.

[0027] In some examples, the wearable computing device can use a speaker to play a known sound (e.g., a test sound) at a known volume. The wearable computing device can then analyze audio data generated by an audio sensor for that sound. This can allow the wearable computing device to more easily detect distortions in the audio data.

[0028] Similarly, when performing two-way communication (e.g., one user talking to another user), the wearable computing device may use an echo canceller to prevent the user's own voice from being played through the user's own speaker. A transfer function generated as part of the echo canceller may be compared to the data captured by the audio sensor to detect any distortions.

[0029] In some embodiments, the data can be device-specific, such that the particular size and dimensions of the port can affect the likelihood that distortion of the data is due to water in the port. For example, different port designs may result in different shapes of water droplets blocking the port. In some embodiments, different shapes of water droplets may result in different characteristic frequencies when distorting the audio data. Thus, if the shape of a particular port of a wearable computing device is known, the data can be analyzed to identify one or more characteristic frequencies.

[0030] In some examples, a machine learning classification model can be used to determine whether an audio signal generated by an audio sensor exhibits distortion (and potentially what type of distortion) caused by the presence of water in the microphone port. For example, the machine learning classification model can be trained with training data that includes audio samples that have been labeled with ground truth information indicating whether the audio samples correspond to or exhibit distortion (and potentially what type of distortion) caused by the presence of water in the port.

[0031] In some examples, the wearable computing device can generate multiple features from the above data, with each feature representing a particular type of data that may indicate the presence of water in the port. In some examples, the generated features can be used as input to an algorithm or machine learning model that generates a water likelihood score. The water likelihood score can represent the likelihood that water is currently in the port. In some examples, the water likelihood score can be compared to a threshold. If the water likelihood score exceeds the threshold, the wearable computing device can determine that the port contains water.

[0032] In response to determining that the port contains water, the wearable computing device can use a digital compensation filter. In some examples, the digital compensation filter can be associated with a particular model of wearable computing device, such that each wearable computing device has a filter that removes a particular type of audio distortion introduced by water in the port of that particular wearable computing device. Additionally or alternatively, the digital compensation filter selected for an application can depend on the type or amount of water contained in the port.

[0033] In some embodiments, the parameters of the digital compensation filter may be changed over time such that the compensation characteristics of the filter are updated as the likelihood of continued presence of water in the port decreases or as the amount of water in the port decreases (e.g., water evaporates or leaks out of the port). In this manner, the digital compensation filter may be variable over time. For example, the use of the digital compensation filter may be reduced linearly or nonlinearly.

[0034] The systems and methods of the present disclosure provide multiple technical effects and advantages. As an example, the proposed system can provide a modified or compensated audio signal that accounts for and counteracts distortion caused by the presence of water in the microphone port. Thus, the systems and methods of the present disclosure can provide improved audio performance (e.g., audio collection performance) by the wearable computing device, which represents an improvement in the functionality of the device itself.

[0035] Exemplary aspects of the present disclosure will now be described in more detail with reference to the figures. FIG. 1 illustrates a front view of an exemplary wearable computing device 100 according to an exemplary embodiment of the present disclosure. In one embodiment, the wearable computing device 100 may be a wristband, a bracelet, a watch, an armband, a ring placed around a user's finger, or other wearable product that may be equipped with sensors as described in this disclosure. In one exemplary embodiment, the wearable computing device 100 is comprised of a display 102, a device housing 104, a band 106, and one or more sensors. In one embodiment, the display 102 may be configured to present data to the user regarding the user's skin temperature, heart rate, sleep state, electroencephalogram, electrocardiogram, electromyogram, electrooculogram, and other physiological data of the user (e.g., blood oxygen level). The display 102 may also be configured to convey data from additional ambient sensors contained within the wearable computing device 100. Exemplary information conveyed to the display 102 from these additional ambient sensors may include the location, altitude, and weather of a location associated with the user. The display 102 may also convey data regarding the user's movement (eg, whether the user is stationary, walking and / or running).

[0036] In an exemplary embodiment, the display 102 can be configured to receive data input by a user. In one embodiment, the user can input on the display to request that the wearable computing device 100 generate additional data to display to the user. In response, the display 102 can present instructions to the user for acquiring the data. In some examples, the wearable computing device 100 can display instructions to the user (e.g., "Place your finger on the sensor for 10 seconds").

[0037] In an exemplary embodiment, the device housing 104 can be configured to include one or more sensors described in this disclosure. Exemplary sensors contained in the device housing 104 can include audio sensors, motion sensors (e.g., accelerometers), pulse oximeters, IR motion sensors, skin temperature sensors, internal device temperature sensors, location sensors (e.g., GPS), altitude sensors, heart rate sensors, pressure sensors, gyroscopes, environmental sensors (e.g., bedside ultrasound sensors), and other physiological sensors (e.g., blood oxygen level sensors). In an embodiment, the device housing 104 can also be configured to include one or more processors. The device housing 104 includes a port that connects the audio sensor to the outside of the device housing 104, allowing audio information to reach the audio sensor without having to pass through the device housing 104.

[0038] The band 106 can be configured to secure the wearable computing device 100 around a user's arm, for example, by connecting both ends of the band 106 with a buckle, clasp, or another similar fastening device, thereby allowing the wearable computing device 100 to be worn by the user.

[0039] 2 illustrates an exemplary computing environment including a wearable computing device 100 according to an exemplary embodiment of the present disclosure. In this example, the wearable computing device 100 can include one or more processors 202, memory 204, an audio sensor 210, a water determination system 212, and a distortion reduction system 214.

[0040] More specifically, the one or more processors 202 may be any suitable processing device that may be incorporated into the form factor of the wearable computing device 100. For example, such processors 202 may include one or more of one or more processor cores, microprocessors, application specific integrated circuits (ASICs), FPGAs, controllers, microcontrollers, etc. The one or more processors 202 may be one or more processors operatively connected together. The memory 204 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and combinations thereof.

[0041] In particular, in some devices, the memory 204 can store instructions for implementing the water determination system 212 and the distortion reduction system 214. Thus, the wearable computing device 100 can implement the water determination system 212 and the distortion reduction system 214 to perform aspects of the disclosure.

[0042] It is understood that the term "system" can refer to specialized hardware, computer logic executing on a more general processor, or some combination thereof. Thus, a system can be implemented in hardware, application specific circuits, firmware, and / or software controlling a general purpose processor. In one embodiment, a system can be implemented as a program code file stored in a storage device, loaded into memory and executed by a processor, or can be provided from a computer program product (e.g., computer executable instructions) stored in a tangible computer readable storage medium, such as a RAM, a hard disk, or an optical or magnetic medium.

[0043] Memory 204 may also include data 206 and instructions 208 that are acquired, manipulated, created, or stored by one or more processors 202. In some exemplary embodiments, such data may be accessed and used as input to water determination system 212 and / or distortion mitigation system 214. In some examples, memory 204 may include data used to perform one or more processes and instructions that describe how those processes may be performed.

[0044] In some examples, the wearable computing device 100 may include one or more sensors. For example, the sensors may include, but are not limited to, one or more of an audio sensor 210, a motion sensor (e.g., an accelerometer), a pulse oximeter, an IR motion sensor, a skin temperature sensor, an internal device temperature sensor, a location sensor (e.g., GPS), an altitude sensor, a heart rate sensor, an audio sensor, a pressure sensor, a humidity sensor, and other physiological sensors (e.g., a blood oxygen level sensor).

[0045] In some examples, audio sensor 210 can detect audio signals in an environment surrounding wearable computing device 100. For example, audio sensor 210 can be configured to capture audio signals associated with a language being spoken in the vicinity of wearable computing device 100. In this manner, a user can vocalize a command and audio sensor 210 can capture data representing the command. Audio sensor 210 can generate the captured data as audio data. Wearable computing device 100 can process the audio data to identify specific words or phrases and can execute one or more commands based on the identified words or phrases.

[0046] The audio sensor 210 may also be connected to a port (also called a microphone port or "mic port"), which is an open air channel connecting the audio sensor to the outside of the wearable computing device 100, allowing audio signals to be more easily detected by the audio sensor 210 located inside the housing of the wearable computing device 100. Without such a port, the audio signal would only be detected after passing through the housing of the wearable computing device 100.

[0047] The port may become partially or completely blocked by water. For example, when a user wears a fitness band while swimming, the microphone port of the fitness band may fill with water. Once the user exits the water, the water may not all immediately drain out of the port. The presence of water in the port may distort the audio signal, such that the audio signal recorded by the audio sensor 210 may not accurately reflect the original sounds in the wearable computing device 100's environment. This distortion may reduce the accuracy of the wearable computing device 100 in processing the audio signal data generated by the audio sensor to identify keywords or phrases that the user may vocalize to execute commands via the wearable computing device 100.

[0048] To compensate for possible distorting effects of water in the port, the wearable computing device 100 can use the water determination system 212 to determine whether the port is currently partially or completely blocked with water. Determining whether water is present in the port can be based on one or more signals accessed by the wearable computing device 100. For example, the wearable computing device 100 can include additional sensors that function to determine whether water is present in the port. One such sensor can be a humidity sensor. The humidity sensor can measure humidity in an area surrounding the wearable computing device 100 (or within the port), and based in part on the humidity data, the wearable computing device 100 can estimate whether water is present in the port.

[0049] Another sensor that may be included in the wearable computing device 100 is one or more electrodes that can directly detect the presence of water in the port. For example, the electrodes may be connected to portions of the port and may respond to the presence of water in a manner that generates an electrical signal (e.g., the presence of water in the port may encourage an increase in electrical conductivity between the electrodes). The electrical signal may be analyzed to determine that water is present in the port based on the measured electrical conductivity. In some examples, if the electrical conductivity between two electrodes exceeds a certain value, the wearable computing device 100 may determine that the port contains water. In some examples, the strength of the signal may be measured, and a signal strength above a particular threshold may determine that there is evidence of water in the port.

[0050] In addition to data generated by one or more sensors, the wearable computing device 100 may have access to data provided by the user. For example, the wearable computing device 100 may include or provide access to an activity tracking application that allows the user to record activities the user performs. In other examples, an algorithm or machine learning model may be used to automatically detect activities the user performs based on various device data, such as sensor data (e.g., data from an accelerometer, etc.). Based on the recorded or detected activity, the wearable computing device 100 may determine whether water is likely to be present in the port. For example, the wearable computing device 100 may determine that water is likely to be present in the port due to a user recently recording a water-based activity, such as swimming, canoeing, rafting, etc. This recorded activity may be used in combination with motion detection of the wearable computing device 100 itself to determine whether the user was wearing the wearable computing device 100 while performing a water-based activity.

[0051] In some examples, the wearable computing device 100 may include a feature intended to help remove water from one or more speakers included in the wearable computing device. For example, a wearable computing device 100, such as a smart watch or fitness band, may include a "water removal" feature that results in the wearable computing device initiating a process that attempts to remove water from the speaker. A user may select this feature when they wish to remove water from the speaker. In some examples, the water removal feature may be enabled by instructing the speaker to play a known signal that causes the speaker to vibrate or shake in an attempt to expel water from the speaker by physical agitation. The wearable computing device 100 may determine whether the feature has been activated by a user, and if so, may determine that there is a relatively high likelihood that water is also present in the port. Additionally, the user may place the wearable computing device in a waterproof mode (e.g., by locking the touch screen). A user actively setting the device to a mode associated with exposure to water may be a strong cue that water may be present in the port.

[0052] In some examples, the wearable computing device 100 can use weather data accessible over a network and user location data (e.g., based on the wearable computing device's GPS signal) to determine whether the user was in a location where rain may have been encountered. For example, if a forecast for a particular area suggests heavy rain and the user's location data indicates that the user was in that area, the wearable computing device can determine that there is a high probability that water is present in the port.

[0053] In some examples, the wearable computing device 100 may also analyze the audio signal generated by the audio sensor to determine whether the audio signal appears to be distorted in a manner related to water in the port. For example, the wearable computing device 100 may include or have access to data representative of the types of distortions commonly seen when water is present. This data, or other data representative of how an audio signal may be distorted by the presence of water, may be used to analyze the data generated by the audio sensor to determine whether the data includes evidence of distortion due to water in the port. For example, the wearable computing device may capture audio data from background noise and analyze it for evidence of water-based distortion. Specific indicators of water-based distortion may include, but are not limited to, the background noise being very quiet (e.g., compared to previously captured background noise), low volume at high frequencies, muffled quality of the background sound, etc. In some examples, the effect of water distortion on the captured audio data may be similar to a low pass filter.

[0054] In some examples, the data can be device specific, such that the specific size and dimensions of the port can affect the likelihood that distortions in the data are due to water in the port. In some examples, a machine learning classification model can be used to determine whether an audio signal generated by an audio sensor exhibits distortions (and potentially what type of distortions) caused by the presence of water in the microphone port. For example, the machine learning classification model can be trained with training data that includes audio samples that have been labeled with ground truth information indicating whether the audio samples correspond to or exhibit distortions (and potentially what type of distortions) caused by the presence of water in the microphone port.

[0055] In some examples, the wearable computing device 100 can have access to data representing the current evaporation rate of water in the port based on a number of possible factors including temperature, humidity, port geometry, etc. The evaporation rate can be used in conjunction with other data to determine if the water in the port has evaporated. The temperature can be determined based on a sensor or can be obtained via a computer network.

[0056] In some examples, the wearable computing device 100 can generate multiple features from the above data, with each feature representing a particular type of data that may indicate the presence of water in the port. In some examples, the generated features can be used as input to an algorithm or machine learning model that generates a water likelihood score. The water likelihood score can represent the likelihood that water is currently in the port. In some examples, the water likelihood score can be compared to a threshold. If the water likelihood score exceeds the threshold, the wearable computing device 100 can determine that the port contains water.

[0057] If the wearable computing device 100 determines that the port contains water, it can use the distortion mitigation system 214 to mitigate the effects of distortion from the water in the port. In some examples, the distortion mitigation system 214 can employ digital correction filters to modify the data output by the audio sensor 210. In some examples, the digital correction filters can be associated with a particular model of wearable computing device 100 such that each particular wearable computing device has a filter that removes a particular type of audio distortion introduced by water in the port of that particular wearable computing device 100. Additionally or alternatively, the digital correction filter selected can depend on the type or amount of water contained in the port.

[0058] In some embodiments, the parameters of the digital compensation filter may be changed over time such that the compensation characteristics of the filter are updated as the likelihood of continued presence of water in the port decreases or as the amount of water in the port decreases (e.g., as the water evaporates or leaks out of the port). In this manner, the digital compensation filter may be variable over time.

[0059] Once the digital correction filter is activated, the wearable computing device 100 can present current port status information to the user. For example, the wearable computing device 100 can display an icon representing the current presence of water in the port. Similarly, the wearable computing device 100 can display information describing possible further actions the user can take to mitigate the presence of water in the port.

[0060] 3 illustrates an exemplary wearable computing system according to an exemplary embodiment of the present disclosure. The wearable computing device 100 includes a water determination system 212, a distortion reduction system 214, an audio sensor 210, and a port 302, among other components (not shown).

[0061] The audio sensor 210 can be a microphone that can detect audio signals in the environment and generate audio data based on those signals. The port 302 connects the audio signal to the outside of the wearable computing device 100. As mentioned above, the port 302 is an open air channel that allows the audio signal to be directly detected by the audio sensor 210.

[0062] The water determination system 212 can use information from multiple signals (including analysis of the data generated by the audio sensor 210) to determine whether the port 302 is partially or completely blocked with water.

[0063] If the water determination system 212 determines that water may be present in the port 302, the distortion reduction system 214 can apply a digital correction filter to modify the data output by the audio sensor 210. By associating the digital correction filter with a particular model of wearable computing device 100, each particular wearable computing device can have a filter that removes a particular type of audio distortion introduced by water in the port of that particular wearable computing device 100. Additionally or alternatively, the digital correction filter selected can depend on the type or amount of water contained in the port. The digital correction filter can modify the signal generated by the audio sensor 210 to remove or minimize distortion introduced by water in the port.

[0064] In some examples, the distortion reduction system 214 can adjust parameters of the digital compensation filter over time. In some examples, the adjustments can be based on an evaporation rate associated with the current environment of the wearable computing device 100. In other examples, the water determination system 212 can provide updated information indicative of the amount of water still present in the port 302, and the digital compensation filter can be adjusted based on the updated information.

[0065] FIG. 4 is a block diagram illustrating an example system for reducing audio distortion in a wearable computing device according to an example embodiment of the present disclosure. A prioritization of sources of data can provide a signal that can be used to determine whether water is present in a port of a particular wearable computing device 100. For example, sensor output data 402 can be collected as a first signal. The sensor output data 402 can be the output of a humidity sensor, a water detection sensor (e.g., using electrodes), the audio sensor 210 itself, or any other sensor that may be useful in determining whether water is present in a port. For example, the sensor output data 402 can include humidity sensor data. The humidity sensor data can represent a degree of humidity in the surroundings of the wearable computing device 100. In some examples, the humidity sensor data can represent a specific location, such as the inside of the port.

[0066] The sensor output data 402 may include data generated by the electrodes. As discussed above, water may increase the electrical conductivity between two electrodes. Thus, the wearable computing device 100 may determine that water is present in the port based on the measured electrical conductivity. In some examples, if the electrical conductivity between the two electrodes exceeds a particular value, the wearable computing device 100 may determine that the port contains water. In some examples, the strength of the signal may be measured, and a signal strength above a particular threshold may determine that there is evidence of water in the port.

[0067] Another source of information can be activity data 404 based on activities logged by a user. The activities can include any activities that may be water-related. In some examples, a user can directly input a log of their activities using an activity logging application. Alternatively, the wearable computing device can directly track (e.g., using a motion sensor) and log the activities automatically. Activities where the likelihood of water being present (such as swimming, canoeing, kayaking, diving, rafting, scuba diving, snorkeling, or any water-based activity) can increase the likelihood that water is present in the port. In some examples, the length of time between the activity and the current time can be used to generate an estimate of the likelihood that water is still present in the port. For example, water may leak or evaporate from the port over time. As a result, the longer the time between the logged activity and the current time, the less likely water is still remaining in the port. The amount of the likelihood that the likelihood decreases can be affected by the rate of evaporation in the area of ​​the wearable computing device 100.

[0068] Another source of data can be interaction data 406 between the user and the wearable computing device 100. For example, the user can perform one or more actions to remove the water from the port. For example, the user can directly shake or otherwise push the device. In another example, the wearable computing device can have a water removal mechanism that can function to attempt to remove the water from the speaker. Such a mechanism can include vibrating or shaking the speaker in an attempt to expel the water from the speaker. Any attempt by the user to remove the water in the speaker can be recorded by the wearable computing device and used as a signal to determine that water may also be present in the port.

[0069] Another source of data can be detectable distortions in the audio data 408 generated by the audio sensor to determine whether the audio signal appears to be distorted in a manner associated with water in the port. For example, the wearable computing device 100 can analyze the audio data from the audio sensor 210. The wearable computing device 100 can include or have access to data that represents the types of distortions that are common when water is present. In some examples, the data can be device specific, such that the particular size and dimensions of the port can affect the likelihood that distortions in the data are due to water in the port.

[0070] In some examples, a machine learning classification model can be used to determine whether an audio signal generated by an audio sensor exhibits distortion (and potentially what type of distortion) caused by the presence of water in the microphone port. For example, the machine learning classification model can be trained with training data including audio samples labeled with ground truth information indicating whether the audio samples correspond to or exhibit distortion (and potentially what type of distortion) caused by the presence of water in the microphone port. Once trained, the machine learning classification model can take raw audio data as input and output a decision representing the likelihood that the raw audio signal is distorted by water in the port. Alternatively, or additionally, the output of the machine learning classification model can be parameters specific to a digital correction filter that can remove distortion from the audio data 408.

[0071] In some embodiments, the wearable computing device 100 can use the data to generate multiple features from the data, with each feature representing a particular type of data that may be indicative of the presence of water in the port. In some embodiments, the feature data can be normalized to a value between 0 and 1. In some embodiments, the feature data can be weighted based on a determination of which features are more or less important than other features.

[0072] In some examples, the feature data can be used as input to the water determination system 212 to generate a water likelihood score. The water likelihood score can represent the likelihood that water is currently in the port. In some examples, the water likelihood score can be compared to a threshold. If the water likelihood score exceeds the threshold, the wearable computing device can determine that the port contains water. In some examples, the water determination system 212 can include or have access to an algorithm or machine learning model that can be used to generate the water likelihood score.

[0073] Once a water likelihood score has been generated (and possibly compared to a threshold), the water determination system 212 can determine whether the port contains water and pass that determination to the distortion reduction system 214. In response to a determination that the port contains water, the distortion reduction system 214 can initiate the use of a digital compensation filter. The digital compensation filter can be a filter that takes the output of the audio sensor 210 and processes it based on the specific characteristics or parameters of the digital compensation filter. In some examples, the digital compensation filter can be associated with a particular model of wearable computing device 100, such that each particular wearable computing device has a filter that removes a particular type of audio distortion introduced by water in the port of that particular wearable computing device. Additionally or alternatively, the digital compensation filter selected for an application can depend on the type or amount of water contained in the port. Once the digital compensation filter is initiated, the distortion reduction system 214 can periodically re-examine whether the digital compensation filter is still needed. The water determination system 212 can periodically update a determination related to the presence of water in the port.

[0074] 5 illustrates a block diagram of an example water likelihood model 500 according to an example embodiment of the present disclosure. In this example, the water likelihood model can take as input feature data based on one or more sources of data associated with water in a port. In some examples, the input data 502 can be based on analysis of audio data collected by a sensor, submitted by a user, accessed over a network, and / or captured. The water likelihood model 500 can output data 506 including a water likelihood score.

[0075] In some examples, the water likelihood model 500 may generate or have access to feature data based on the input data 502. In some examples, the feature data may represent data collected by a sensor, data submitted by a user, data accessed over a network, and / or data generated based on analysis of captured audio data and / or correlations between different data types or sources.

[0076] Once the feature data is generated, the water likelihood model 500 can generate a water likelihood score that represents the likelihood that the port contains water. In some embodiments, the output data 506 can include a decision indicating whether the port contains water and a confidence value that represents the degree to which the water likelihood model 500 is confident that the port contains water. In some embodiments, the decision and confidence value can be generated based on the water likelihood value. For example, the output data can have a 70% confidence value associated with a decision that water is present in the port.

[0077] Output data 506 may be received from the water likelihood model 500 and may be communicated to the distortion mitigation system 214. In some embodiments, the output data may be communicated to the wearable computing device 100 for indication to a user (e.g., indicating the presence of water in the port).

[0078] FIG. 6 is a flow chart illustrating an example process for reducing distortion of a signal generated by an audio sensor according to an example embodiment of the present disclosure. One or more portions of the method may be implemented by one or more computing devices, such as, for example, the computing devices described herein. Additionally, one or more portions of the method may be implemented as an algorithm on hardware components of the device(s) described herein. FIG. 6 shows elements performed in a particular order for purposes of illustration and description. Those skilled in the art will appreciate that, using the disclosure provided herein, any elements of the methods described herein may be adapted, rearranged, extended, omitted, combined and / or modified in various manners without departing from the scope of the present disclosure. The method may be implemented by one or more computing devices, such as one or more computing devices illustrated in FIGS. 1-3.

[0079] A wearable computing device (e.g., wearable computing device 100 of FIG. 1) can be configured to reduce distortion in a signal generated by an audio sensor. The wearable computing device 100 can include one or more processors, an audio sensor, and a port connecting the audio sensor to an exterior surface of the wearable computing device 100. The wearable computing device can further include a non-transitory computer-readable memory that stores instructions that, when executed by the one or more processors, cause the wearable computing device to perform operations. In some examples, the audio sensor is a microphone.

[0080] At 602, the operations can include accessing port status data. In some examples, the port status data can include user-supplied application data. The user-supplied application data can include recent activity data recorded by the user. For example, the recent activity data recorded by the user can include one or more water-based activities, such as swimming, diving, scuba diving, boating, etc.

[0081] In some examples, the port status data can include data describing a user interaction with the wearable computing device. For example, the user interaction with the wearable computing device can include a user selecting a water removal function to remove water from a speaker included in the wearable computing device. The user interaction with the wearable computing device 100 can further include the user physically shaking or otherwise attempting to manually expel water from the port. Data describing the user shaking the wearable computing device 100 can be collected from a motion sensor (e.g., an accelerometer or gyroscope).

[0082] In some examples, the wearable computing device 100 includes a humidity sensor, and the port status data includes humidity data generated by the humidity sensor. In some examples, the wearable computing device includes an electrode in the port, and the port status data includes data generated by the electrode indicating whether water is sensed in the port. In some examples, the housing of the wearable computing device can be made of a non-metallic material (e.g., plastic, etc.) such that electrical conductivity between the two electrodes is not compromised.

[0083] In some examples, the port status data can be generated by analyzing audio data generated by an audio sensor to identify audio distortions associated with water in the port. The port status data can include a current water evaporation rate.

[0084] At 604, the wearable computing device 100 may determine a water presence likelihood value representative of the likelihood that water is currently present in the port based on the port status data. At 606, in accordance with a determination that the water presence likelihood value exceeds a threshold, the wearable computing device 100 may apply a digital compensation filter to the data generated by the audio sensor, where the digital compensation filter corrects the data generated by the audio sensor to compensate for the presence of water in the port. In some examples, the threshold may be pre-determined for a particular wearable computing device. In other examples, the threshold may be adaptive, such that the threshold may be determined based at least in part on current conditions and data collected by the wearable computing device based on past situations where the threshold was exceeded. The digital compensation filter may be a fixed compensation filter. In some examples, the digital compensation filter may be an adaptive compensation filter. The digital compensation filter may be updated over time based on a current rate of water evaporation.

[0085] The technology described herein refers to sensors and other computer-based systems, as well as actions performed and information transmitted to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions between components. For example, the server processes described herein may be implemented using a single server or multiple servers working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0086] Although the present subject matter has been described in detail with respect to specific exemplary embodiments thereof, it will be appreciated that those skilled in the art, upon having access to the foregoing, may readily generate modifications, variations, and equivalents to such embodiments. Accordingly, the scope of the present disclosure is by way of example rather than limitation, and the disclosure of the present subject matter is not intended to exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to those skilled in the art.

Claims

**Claim 1** A wearable computing device comprising: one or more processors; an audio sensor; a port for connecting the audio sensor outside the surface of the wearable computing device; a non-transitory computer-readable memory storing instructions that, when executed by the one or more processors, cause the wearable computing device to perform operations including: accessing port status data; determining, based on the port status data, a water presence likelihood value representing a likelihood that water is currently present in the port; applying a digital correction filter to data generated by the audio sensor in accordance with a determination that the water presence likelihood value exceeds a threshold, wherein the digital correction filter corrects the data generated by the audio sensor to compensate for the presence of water in the port. **Claim 2** The wearable computing device of claim 1, wherein the port status data includes application data supplied by a user. **Claim 3** The wearable computing device of claim 2, wherein the application data supplied by the user includes recent activity data recorded by the user. **Claim 4** The wearable computing device according to any one of claims 1 to 3, wherein the port status data includes data representing user interaction with the wearable computing device. **Claim 5** The wearable computing device of claim 4, wherein the user interaction with the wearable computing device includes the user selecting a water removal function. **Claim 6** The wearable computing device of claim 1, further comprising a humidity sensor, wherein the port status data includes humidity data generated by the humidity sensor. **Claim 7** The wearable computing device according to any one of claims 1 to 3 and 6, wherein the port status data includes a current evaporation rate of water. **Claim 8** The wearable computing device according to any one of claims 1 to 3 and 6 includes an electrode in the port, and the port status data includes data generated by the electrode indicating whether water is detected in the port.

9. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the port status data is generated by analyzing audio data generated by the audio sensor to identify an audio distortion related to water in the port.

10. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the port status data includes a current water evaporation rate.

11. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the digital correction filter is a fixed compensation filter.

12. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the digital correction filter is an adaptive compensation filter.

13. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the digital correction filter is updated over time based on a current water evaporation rate.

14. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the threshold value is predetermined.

15. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the threshold value is adaptively determined.

16. The wearable computing device according to any one of claims 1 to 3 and 6, wherein the audio sensor is a microphone.

17. A computer-implemented method for correcting distortion introduced into audio data caused by water in a port associated with an audio sensor, comprising: accessing port status data by a wearable computing device including one or more processors; determining, by the wearable computing device, a water presence likelihood value representing a likelihood that water is currently present in the port, based on the port status data; In accordance with a determination that the water presence likelihood value exceeds a threshold, applying, by the wearable computing device, a digital correction filter to data generated by the audio sensor, the digital correction filter correcting the data generated by the audio sensor to compensate for the presence of water in the port, a computer-implemented method.

18. The computer-implemented method according to claim 17, wherein the port status data includes application data supplied by a user.

19. A program that, when executed by one or more computing devices, causes the one or more computing devices to execute the computer-implemented method according to claim 17 or 18.

Images