Detecting and measuring snoring

Abstract

When a user of an electronic device is determined to be asleep, methods described herein can capture an audio signal using at least one microphone. At least one audio frame can be determined from the audio signal. The at least one audio frame represents a frequency spectrum detected by the at least one microphone over a certain time period. One or more sounds associated with the at least one audio frame can be determined. Sleep-related information can be generated. The information identifies the one or more sounds as potential sleep disruptors.

Description

Technical Field

[0001] The present technology relates to the field of wearable devices. More specifically, the present technology relates to techniques for collecting and processing biometric data. Background Art

[0002] Wearable electronic devices are becoming increasingly popular among consumers. Wearable electronic devices can track a user's activity using various sensors. The data captured from these sensors can be analyzed to provide the user with information that can help the user maintain a healthy lifestyle. To determine information about a user's activity, conventional wearable electronic devices collect activity data and perform calculations on the data, such as on the device, remotely (e.g., in a server environment), or a combination of both. Summary of the Invention

[0003] Various embodiments of the present technology may include systems, methods, and non-transitory computer-readable media configured to capture an audio signal using at least one microphone when a user of an electronic device is determined to be asleep. At least one audio frame may be determined from the audio signal, wherein the at least one audio frame represents a frequency spectrum detected by the at least one microphone over a period of time. One or more sounds associated with the at least one audio frame may be determined. Sleep-related information may be generated, wherein the information identifies the one or more sounds as potential sources of sleep disruption.

[0004] In some embodiments, determining one or more sounds associated with the at least one audio frame further comprises determining information describing the one or more sounds based at least in part on a machine learning model relating a frequency spectrum represented by the at least one audio frame to the one or more sounds.

[0005] In some embodiments, the systems, methods, and non-transitory computer-readable media may also be configured to: extract a feature set based on a frequency spectrum represented by at least one audio frame; provide the feature set to a machine learning model; and obtain information describing one or more sounds from the machine learning model.

[0006] In some embodiments, the feature set includes at least Mel-Frequency Cepstral Coefficients (MFCCs), which describe the frequency spectrum represented by the at least one audio frame.

[0007] In some embodiments, determining one or more sounds associated with the at least one audio frame further comprises determining one or more sources of the one or more sounds, and identifying the one or more sources of the one or more sounds in the generated sleep-related information.

[0008] In some embodiments, the one or more sounds associated with the at least one audio frame are determined while the audio signal is being captured.

[0009] In some embodiments, the systems, methods, and non-transitory computer-readable media may be further configured to discard the at least one audio frame after determining one or more sounds associated with the at least one audio frame.

[0010] In some embodiments, at least one microphone for capturing audio signals is included in one or both of the electronic device or a computing device separate from the electronic device.

[0011] In some embodiments, the systems, methods, and non-transitory computer-readable media may be further configured to provide the generated sleep-related information via an interface accessible to the electronic device or a computing device separate from the electronic device.

[0012] In some embodiments, the interface provides one or more options for audibly replaying at least one audio frame associated with one or more sounds.

[0013] In some embodiments, the sleep-related information provides one or more of the following: the total time the user was determined to be sleeping, one or more time periods when the user was determined to be awake, or one or more time periods when the user was determined to be snoring.

[0014] In some embodiments, the sleep-related information provides one or more of the following: the total time the user was snoring, the percentage of time the user was snoring relative to the total time the user was determined to be asleep, or a range of percentages of time the user was snoring relative to the total time the user was determined to be asleep.

[0015] In some embodiments, the sleep-related information provides a classification of one or more time periods during which the user was determined to be snoring based on snoring intensity.

[0016] In some embodiments, the classification labels the one or more time periods during which the user was determined to be snoring as first stage snoring, second stage snoring, or third stage snoring.

[0017] In some embodiments, the sleep-related information identifies a correlation between one or more time periods during which the user is determined to be awake and one or more sounds identified as potential sleep interrupters.

[0018] In some embodiments, the sleep-related information provides a list of sounds detected while the user was asleep.

[0019] In some embodiments, the list identifies sounds detected while the user was asleep and the corresponding number of instances of the detected sounds.

[0020] In some embodiments, the sleep-related information provides one or more of: a baseline noise level while the user was asleep, an average noise level while the user was asleep, a maximum noise level while the user was asleep, or a total number of sounds detected while the user was asleep.

[0021] It should be understood that many other features, applications, embodiments and / or variations of the disclosed technology will become apparent from the accompanying drawings and the following detailed description. Additional and / or alternative embodiments of the structures, systems, non-transitory computer-readable media, and methods described herein may be employed without departing from the principles of the disclosed technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figures 1A to 1C An example schematic diagram according to an embodiment of the present technology is illustrated.

[0023] Figure 2A is a block diagram illustrating an example environment including a wearable device in accordance with embodiments of the present technology.

[0024] Figure 2B is a block diagram illustrating example sensors that may communicate with a processor of a wearable device in accordance with embodiments of the present technology.

[0025] Figure 2C is a block diagram illustrating an example snore module in accordance with embodiments of the present technology.

[0026] Figure 3 Illustrated is an example snorer identification module in accordance with embodiments of the present technology.

[0027] Figure 4 Illustrated is an example snore metric module in accordance with embodiments of the present technology.

[0028] Figure 5 Illustrated are example diagrams of captured audio and sensor data in accordance with embodiments of the present technology.

[0029] Figures 6A to 6K An example schematic diagram of a user interface according to an embodiment of the present technology is illustrated.

[0030] Figure 7 、 Figures 8A to 8B and Figures 9 to 11 An example method according to an embodiment of the present technology is illustrated.

[0031] For illustrative purposes only, the accompanying drawings depict various embodiments of the disclosed technology, wherein the same reference numerals are used to identify the same elements in the drawings. Those skilled in the art will readily appreciate from the following discussion that alternative embodiments of the structures and methods shown in the drawings may be employed without departing from the principles of the disclosed technology described herein. DETAILED DESCRIPTION

[0032] Traditional methods for determining whether a user of a wearable device snores may rely on audio data captured by a microphone. For example, the audio data may reflect spectral characteristics that indicate a person snoring. However, these traditional methods are prone to error when there are multiple entities in the room (e.g., people, animals). In this case, although the audio data captured by the microphone in the room may still reflect spectral characteristics that indicate the entity snoring, the audio data alone cannot generally be used to isolate the source of the snoring. Furthermore, even if there is only one entity in the room, traditional methods generally cannot determine various sleep-related information, such as the intensity of the snoring produced by the entity. The importance of sleep to physical and mental health is irrefutable. In fact, a large number of scientific studies have shown that lack of sleep can affect a person's overall health and make a person susceptible to serious diseases such as obesity, heart disease, hypertension, and diabetes. Therefore, there is a need for innovative computer technology solutions that can, among other things, address the technical problems associated with the need to identify wearable device users who snore, and can also determine sleep-related information that helps these users improve their sleep habits and ultimately improve their overall health.

[0033] Improved methods arising from computer technology overcome the above-mentioned and other shortcomings associated with traditional methods, particularly shortcomings arising in the field of computer technology. In various embodiments, sensor data captured by sensors of a wearable device can be used in conjunction with audio data to determine whether a user of the wearable device is snoring. Generally speaking, "snoring" can refer to sounds caused by or associated with disturbed breathing while a person is sleeping. Snoring can be audible to a person (such as traditional snoring) and / or detected by electronic sensors (such as microphones in a wearable device or other device close to the sleeping person). If it is determined that the user of the wearable device is snoring, various information describing the snoring (e.g., snoring metrics) can be determined for the user. For example, the information can identify one or more durations of snoring activity detected for the user over a period of time. In some cases, determining that the user is not snoring can also provide useful information. For example, the user 102 of the wearable device 104 may be sleeping in the same room with a pet 106, such as Figure 1AAs shown in the example of . In this example, animal 106 is snoring, while user 102 is not. In various embodiments, based on various factors, such as a combination of audio data containing snoring (e.g., audio data captured by a microphone in wearable device 104) and sensor data captured by wearable device 104, it can be determined whether the snoring originates from user 102 of wearable device 104 or another entity present in the room. For example, based on audio data recording animal 106 snoring over a period of time, a first breathing phase pattern can be determined. Further, based on sensor data captured by one or more sensors in wearable device 104 over the same period of time, a second breathing phase pattern can be determined. For example, based on sensor data captured by one or more plethysmogram (PPG) sensors, motion detectors (e.g., accelerometers), respiratory monitors (e.g., sensors capable of detecting expansion and contraction of the chest or abdomen, sensors capable of detecting inhalation and / or exhalation, etc.), electrodermal activity (EDA) sensors, temperature sensors, or a combination thereof, the second breathing phase pattern can be determined. Such sensor data may provide physiological information describing the user 102 of the wearable device 104, such as when the user 102 inhales and exhales. In these embodiments, a magnitude of a correlation between a first breathing phase pattern determined based on the audio data and a second breathing phase pattern determined based on the sensor data may be determined. Based on the magnitude of the correlation, it may be determined whether the snoring originates from the user 102 of the wearable device 104 or from another entity in the room (e.g., an animal 106). Figure 1A In the example of FIG. 1 , based on the correlation between the first breathing phase pattern and the second breathing phase pattern failing to match or satisfy a threshold qualitative and / or quantitative value (e.g., signal phase matching, a derived “score,” or other derived quantity), it is determined that the snoring is not originating from the user 102 of the wearable device 104. This determination may be particularly useful so that the user 102 can receive accurate sleep-related information and recommendations without false positives.

[0034] If it is determined that the user 102 of the wearable device 104 is snoring, various information describing the snoring (e.g., snoring metrics) may be determined for the user 102. In some cases, the accuracy of such information, such as the intensity of the snoring, may be affected by factors such as the distance 108 between the wearable device 104 that is recording the user's snoring and the mouth 110 of the user 102 emitting the snoring. Figure 1BAs shown in the example of . For example, when distance is taken into account, a snoring sound that is determined to be very intense (e.g., 75 dB) due to the proximity of the wearable device 104 to the user's mouth 110 may actually be of moderate intensity (e.g., 50 dB). Therefore, in various embodiments, distance 108 may be determined (or estimated) to improve the accuracy of various snoring metrics for the user. For example, distance 108 may be determined based on the wrist posture and sleep posture of the user 102. In some embodiments, the posture may be determined based on the relationship between the orientation of the wearable device 104, the posture of the wearable device 104 with respect to magnetic north pole, and the musculoskeletal constraints imposed by a rigid human posture model. In some embodiments, the posture may be determined based on sensor data, such as a ballistocardiogram (GCG) pattern detected by the wearable device 104, location information associated with the wearable device 104, such as information provided by an altimeter, gyroscope, or other sensor, etc. In some embodiments, distance 108 may be adjusted (or estimated) based on the height of the wearable device user 102. Many variations are possible, including combinations of the various embodiments described above.

[0035] Figure 1C An example of a wearable device 150 is illustrated. The wearable device 150 includes both a computing device 152 and a wristband portion 154. The wearable device 150 may include a display screen 156 for viewing and accessing various information (e.g., snoring metrics). The display screen 156 may be a capacitive touch screen that may be configured to respond when contacted by a charge holding member or tool (such as a human finger). The wearable device 150 may include one or more buttons 158 that may be selected to provide various user inputs. The wearable device 150 may include one or more functional components (or modules) designed to determine one or more physiological metrics associated with a user (or wearer) of the wearable device 150, such as a heart rate sensor, a body temperature sensor, and an ambient temperature sensor, to name a few. These modules may be arranged or associated with the bottom / back surface of the wearable device 150 and may be in contact (or substantially in contact) with human skin when the wearable device 150 is worn. In some embodiments, the tactile sensor may be arranged on the interior (or skin side) of the wearable device 150. Referring below Figures 2A to 2C More details describing the wearable device 150 are provided.

[0036] Figure 2A is a block diagram illustrating an example wearable device 200 according to an embodiment of the present technology. The wearable device 200 may be a smartwatch, a watch, a wristband fitness device, a wristband health device, an activity monitoring device, or the like, although the concepts described herein may be implemented in any type of portable or wearable device that includes one or more sensors. For example, the wearable device 200 may be implemented as Figure 1CWearable device 150. Wearable device 200 may include one or more processors 202, memory 204, sensors 206, wireless transceiver 208, interface 210, and snore module 212. Processor 202 may interact with memory 204, sensors 206, wireless transceiver 208, interface 210, and snore module 212 to perform various operations described herein. Memory 204 may store instructions for causing processor 202 to perform certain operations (or actions). In some embodiments, sensor 206 may collect various types of sensor data that may be used by wearable device 200 to detect and respond to various scenarios. Wearable device 200 may be configured to wirelessly communicate with one or more client devices 214 and / or server 216, for example, directly via wireless transceiver 208 or when within range of a wireless access point (e.g., via a personal area network (PAN), such as Bluetooth pairing, a wireless local area network (WLAN), a wide area network (WAN), etc.). Client device 214 can be a smartphone, tablet computer, or another mobile device executing software (e.g., a mobile application) configured to perform one or more operations described herein. Server 216 can be implemented using one or more computing systems that execute software configured to perform one or more operations described herein. Depending on the implementation, the operations described herein can be performed solely by wearable device 200 or in a distributed manner by wearable device 200, client device 214, and server 216. For example, in some embodiments, snore module 212 (or some operations performed by snore module 212) can be implemented by client device 214 and / or server 216. Wearable device 200 can collect one or more types of data from sensor 206 and / or external devices. The collected data can be transmitted to other devices (e.g., client device 214 and / or server 216), allowing the collected data to be processed by client device 214 and / or server 216 using any and all methods described herein. The collected data can also be viewed, for example, using a web browser or software application. For example, when worn by a user, wearable device 200 can perform biometric monitoring by counting and storing the user's steps based on sensor data collected by sensor 206. Wearable device 200 can transmit data representing the user's steps to a user account on a web service (e.g., fitbit.com), a computer, a mobile phone, and / or a health website, where the data can be stored, processed, and / or visualized by the user. Wearable device 200 can measure or calculate other metrics besides the user's steps.Such metrics may include, but are not limited to, energy expenditure (e.g., calories burned), floors climbed and / or descended; heart rate, heart beat waveform, heart rate variability, heart rate recovery, respiration, oxygen saturation (SpO2), blood volume, blood glucose, skin moisture and skin pigment levels, location and / or heading (e.g., via GPS, Global Navigation Satellite System (GLONASS) or similar systems), altitude, walking speed and / or running distance, blood pressure, blood glucose, skin conductance, skin and / or body temperature, muscle state measured via electromyography, brain activity measured by electroencephalography, weight, body fat, caloric intake, nutrient intake from food, medication intake, sleep periods (e.g., clock time, sleep stages, sleep quality and / or duration), pH level, hydration level, and respiratory rate, to name a few. The wearable device 200 can also measure or calculate metrics related to the user's environment, such as, for example, barometric pressure, weather conditions (e.g., temperature, humidity, pollen count, air quality, rain / snow conditions, wind speed), light exposure (e.g., ambient light, ultraviolet (UV) exposure, time and / or duration spent in darkness), noise exposure, and / or magnetic fields. In addition, the wearable device 200 can obtain data from the sensors 206 and can calculate metrics obtained from such data (or, in some embodiments, transmit some or all of such sensor data (or a subset thereof) to another computing device (such as a server environment or a personal computing device) in a synchronous or asynchronous manner, which is then transmitted back to the wearable device and / or used to generate commands that, when transmitted to the wearable device, cause the wearable device to perform one or more actions (e.g., display a message, generate an alarm, etc.) in order to calculate one or more of the metrics). For example, the wearable device 200 can calculate the user's stress or relaxation level based on a combination of heart rate variability, skin conduction, noise pollution, and / or sleep quality. In another example, the wearable device 200 can determine the effectiveness of a medical intervention (e.g., medication) based on a combination of data related to medication intake, sleep, and / or activity. In another example, the wearable device 200 can determine the effectiveness of an allergy medication based on a combination of data related to pollen levels, medication intake, sleep, and / or activity. These examples are provided for illustration only and are not intended to be limiting or exclusive. See below. Figure 2B More details are provided describing the sensor 206. Figure 2C More details are provided describing the snore module 212 .

[0037] Figure 2Bis a block diagram illustrating several example sensors 220 that can be included in a wearable device 200 according to embodiments of the present technology. For example, the wearable device 200 can include at least one accelerometer 222 (e.g., a multi-axis accelerometer), a gyroscope 224, a magnetometer 226, an altimeter 228, a GPS receiver 230, a green plethysmogram (PPG) sensor 232, a red PPG sensor 234, an infrared (IR) PPG sensor 236, one or more microphones 238, and one or more other sensors 240, all of which can communicate with the processor 202. For example, the one or more other sensors 240 can include a temperature sensor, an ambient light sensor, a galvanic skin response (GSR) sensor, a capacitive sensor, a humidity sensor, a force sensor, a gravity sensor, a piezoelectric film sensor, and a rotation vector sensor, all of which can communicate with the processor 202. The processor 202 can use input received from any sensor or combination of sensors to detect the start of an activity (or exercise) and / or track metrics of the activity. In some embodiments, some sensors 220 may be placed on the user's chest, the mattress on which the user sleeps, or the user's nightstand while the wearable device 200 is worn by the user. Figure 2B The examples illustrate various sensors, but in other embodiments, the wearable device 200 may include a smaller number of sensors and / or any other subset and combination of sensors. Additionally, in some embodiments, the GPS receiver 230 may be implemented by the client device 214 rather than by the wearable device 200. In these embodiments, the wearable device 200 may wirelessly communicate with the client device 214 to access geolocation data. In related aspects, the processor 202 and other components of the wearable device 200 may be implemented using any of a variety of suitable circuitry, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are partially implemented in software, the device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. In further related aspects, the processor 202 and other components of the wearable device 200 can be implemented as a system on a chip (SoC), which can include one or more central processing unit (CPU) cores that use one or more reduced instruction set computing (RISC) instruction sets, a GPS receiver 230, wireless wide area network (WWAN) radio circuitry, WLAN radio circuitry, and / or other software and hardware to support the wearable device 200. Furthermore, many variations are possible.

[0038] Figure 2CAn example snore module 252 is illustrated according to an embodiment of the present disclosure. Figure 2C As shown in the example of , the snore module 252 may include a sensor data module 254, a snorer identification module 256, a snore metric module 258, and a task cycle module 260. In some cases, the snore module 252 may interact with at least one data store 280 to access data stored by a wearable device (such as a wearable device) implementing the snore module 252. Figure 2A The components (e.g., modules) shown in this figure and throughout the figures herein are provided as examples, and other embodiments may include additional, fewer, integrated, or different components. In some embodiments, the snore module 252 may be implemented in part or in whole as software, hardware, or any combination thereof. In general, the modules discussed herein may be associated with software, hardware, or any combination thereof. In some embodiments, one or more functions, tasks, and / or operations of a module may be performed or performed by a software routine, a software process, hardware, and / or any combination thereof. In some cases, the snore module 252, or at least a portion thereof, may be implemented in part or in whole as software running on a wearable device. In some embodiments, the snore module 252 may be implemented in part or in whole as software running on one or more computing devices. For example, the snore module 252, or at least a portion thereof, may be implemented as software running on a user computing device such as Figure 2A In some cases, the snore module 252 may be partially or entirely hosted on a server such as a client device 214. Figure 2A The server 216 of FIG. 214 is implemented in or configured to operate in conjunction with the server. It should be understood that many variations or other possibilities are possible.

[0039] The sensor data module 254 can be configured to access sensor data captured by various sensors of the wearable device 200, as described above with reference to Figures 2A to 2BAs described. For example, the sensor data may include data captured by one or more microphones, a plethysmogram (PPG) sensor, an accelerometer, a gyroscope, a magnetometer, and an altimeter, to name a few examples. For example, the sensor data module 254 may obtain such sensor data in real time directly from a sensor associated with the wearable device 200 or obtain such sensor data from a data repository 280 that may store such sensor data. In some cases, the sensor data module 254 may obtain sensor data captured by one or more external sensors separate from the wearable device 200. For example, a separate microphone separate from the wearable device 200 may be used to record (or capture) audio data of the user snoring. Many variations are possible. In some embodiments, the sensor data may be captured and / or supplemented by one or more non-wearable devices (e.g., a bed sheet or other non-wearable device). For example, in some embodiments, the non-wearable device may be configured to detect human motion based on ultrasound. In addition, many variations are possible.

[0040] The snorer identification module 256 can be configured to determine whether the user of the wearable device is snoring. For example, the snorer identification module 256 can determine whether the user is snoring based on audio data captured by one or more microphones associated with the wearable device and sensor data captured by one or more sensors associated with the wearable device. Figure 3 More details describing the snorer identification module 256 are provided.

[0041] The snoring metric module 258 can be configured to determine snoring information (e.g., a metric) of a user of the wearable device. For example, the snoring metric module 258 can determine one or more durations of snoring activity of the user over a period of time. In another example, the snoring metric module 258 can determine changes in the level or intensity of the user's snoring over a period of time. In some embodiments, such a metric can be determined based on an estimated distance (or proximity) between the wearable device and the user's mouth. Figure 4 More details describing the snore metric module 258 are provided.

[0042] The task cycle module 260 can be configured to selectively activate at least one microphone in the wearable device based on a breathing (or respiratory) phase pattern determined for the user. For example, the breathing phase pattern can be determined based on any of the techniques described herein. In various embodiments, the task cycle module 260 can time the activation of the microphone to record snoring sounds emitted by the user. That is, based on the breathing phase pattern, the task cycle module 260 can activate the microphone when the user inhales (e.g., when snoring sounds are emitted) and disable the microphone after inhalation (e.g., after the snoring sounds are recorded). For example, if the user snores once every four seconds, the task cycle module 260 can activate the microphone before snoring is expected to occur, while disabling (or turning off) the microphone between snoring sounds. This selective activation helps save power and storage space, which would otherwise drain the wearable device if the microphone remained active throughout the user's sleep cycle (e.g., overnight). In some embodiments, the task cycle module 260 can determine when the user's sleep activity is disturbed and can activate the microphone to record sounds that may cause the disturbance. In these embodiments, the task cycle module 260 can apply well-known classifiers to detect when the user is asleep and when they are not. In some embodiments, the user can interact with the wearable device to deactivate the task cycle, allowing the microphone to fully record the user's sleep activity, for example, to perform a complete sleep diagnosis on the user. Many variations are possible.

[0043] The sound analysis module 262 can be configured to identify and classify sounds based on audio signals captured by one or more microphones associated with the wearable device. For example, the sound analysis module 262 can enable the microphone associated with the wearable device to capture an audio signal (or recording) when the user of the wearable device is determined to be asleep. In addition to ambient noise occurring in the user's sleeping environment, the audio signal can capture sounds made by the user while sleeping (e.g., snoring). In this example, the sound analysis module 262 can analyze the audio signal to identify sounds (e.g., engine noise, dog barking, etc.) and the corresponding sources of the sounds (e.g., vehicles, animals, etc.). In various embodiments, the classified sounds can be used to identify potential sources of sleep interruption that the user may not be aware of. In some embodiments, information describing the classified sounds can be provided through an interface. For example, in some embodiments, the interface can provide an option to replay the classified sounds, such as Figure 6B In some embodiments, the interface may identify the classified sounds in a list of sounds that may disturb the user's sleep, such as Figure 6C In some embodiments, information describing the classified sound can be provided in the snore report, such as Figure 6DAs shown in the example of . The snore report can provide various sleep-related information, including, for example, the total time the user was determined to be sleeping, any time periods when the user was determined to be awake, and any time periods when the user was determined to be snoring. In some embodiments, the sleep-related information can identify a correlation between one or more time periods when the user was determined to be awake and any classified sounds identified as potential sleep interrupters during those time periods. Many variations are possible.

[0044] The sound analysis module 262 can analyze and classify audio based on various methods for processing audio signals. In some embodiments, the sound analysis module 262 can analyze and classify audio based on machine learning. For example, the sound analysis module 262 can obtain an audio signal to be analyzed. The sound analysis module 262 can segment the audio signal into a set of audio frames. Each audio frame can represent the frequency spectrum of a sound within a certain time period. According to one embodiment, the audio frames may or may not overlap. In these embodiments, the sound analysis module 262 can analyze the audio frames by extracting a set of features from the frequency spectrum represented by the audio frames. For example, in some embodiments, the feature set can include Mel-Frequency Cepstral Coefficients (MFCCs), which describe the frequency spectrum represented by the audio frames. The sound analysis module 262 can provide the feature set to a machine learning model that has been trained to classify the feature set as a specific sound. Depending on the embodiment, the machine learning model can be trained to classify the feature set based on different specificities. For example, the feature set can be classified as a specific sound source (e.g., a vehicle, an animal, a stove, etc.) or different sounds (e.g., a vehicle horn, a dog barking, etc.). In some embodiments, the machine learning model can be trained based on the feature set extracted from a set of labeled audio frames. For example, audio frames can be labeled based on the sounds represented in the audio frames. Many variations are possible. For example, in some embodiments, audio frames can be clustered based on their extracted features. For example, each audio frame cluster can be labeled as corresponding to a specific sound. The audio frame clusters can then be used to train a machine learning model.

[0045] In some embodiments, the sound analysis module 262 can analyze and classify audio based on a privacy-by-design approach. In these embodiments, the sound analysis module 262 can analyze and classify the audio in real time (or near real time) as the audio signal is captured. The sound analysis module 262 can automatically discard the audio data after classifying it using the methods described herein. In this approach, the sound analysis module 262 can still provide information describing the sounds identified in the sleep environment without saving the audio data.

[0046] Figure 31 illustrates an example of a snorer identification module 302 according to an embodiment of the present disclosure. In some embodiments, Figure 2C The snorer identification module 256 may be implemented using the snorer identification module 302. Figure 3 As shown in the example of , the snorer identification module 302 may include an audio-based respiration determination module 304 , a sensor-based respiration determination module 306 , a similarity module 308 , and a clustering module 310 .

[0047] The audio-based respiration determination module 304 can be configured to determine (or estimate) the respiration phase of an entity (e.g., a person, an animal) based on audio data of the entity's snoring. Generally speaking, snoring can refer to a unique sound made when air flows through relaxed tissue in the entity's throat, causing the relaxed tissue to vibrate when the entity inhales and exhales. For example, in various embodiments, the audio-based respiration determination module 304 can obtain audio data that records the entity's snoring over a period of time. The audio data can provide an audio spectrum that includes spectral characteristics indicative of the entity's snoring at different time intervals, such as Figure 5, as shown in the example of audio data 502. The audio spectrum may reflect the corresponding energy level (or sound pressure level) associated with each snore produced by the entity. Depending on the embodiment, the audio data may be captured by one or more microphones in the wearable device, one or more independent microphones separate from the wearable device (e.g., microphones in a mobile device), or a combination thereof. In some embodiments, the audio data may be preprocessed. For example, in some embodiments, the audio data may be preprocessed by removing (or reducing) ambient or surrounding noise captured in the audio data using well-known noise reduction techniques. Such preprocessing can help improve snore-related estimates, such as snore level (or volume) estimates. In some embodiments, the audio data may be preprocessed by removing (or reducing) motion noise captured in the audio data. For example, the audio data may be captured by a microphone in the wearable device. In this example, when the entity moves during sleep, the microphone may capture motion noise. In some cases, this motion noise may distort the entity's snoring activity captured by the microphone. To improve the detection and processing of the entity's snoring activity, the audio-based respiration determination module 304 may exclude portions of the audio data that include instances of motion noise. For example, the audio-based breathing determination module 304 can detect motion based on sensor data and can classify the overlay noise for sounds emitted when the entity breathes (i.e., inhales and exhales). In some embodiments, motion noise instances can be identified by correlating motion detected by one or more accelerometers in the wearable device. In some embodiments, motion noise instances can be identified by correlating motion detected by one or more PPG sensors in the wearable device. Many variations are possible. In various embodiments, the audio-based breathing determination module 304 can determine (or estimate) the entity's breathing (or respiration) phase based on audio data according to well-known techniques. The entity's breathing phase based on audio data can be used to determine whether the entity's snoring corresponds to the user of the wearable device based on sensor data captured by the wearable device, as described below.

[0048] The sensor-based respiration determination module 306 can be configured to determine (or estimate) the respiration phase of the user of the wearable device based on the sensor data captured by the wearable device. For example, in some embodiments, the sensor-based respiration determination module 306 can obtain sensor data from one or more plethysmogram (PPG) sensors in the wearable device. For example, the PPG sensor can include a green PPG sensor, a red PPG sensor, an infrared (IR) PPG sensor, or a combination thereof. Each PPG sensor can apply well-known light-based technology to independently measure the blood flow rate controlled by the user's heart when pumping blood, i.e., a heart rate signal. These measurements can indicate that the user is snoring at various time intervals, such as Figure 5, as shown in the PPG data 504 in the example. In various embodiments, the measurements taken by each PPG sensor can be used to determine (or estimate) the corresponding respiratory (or respiration) phase for the user based on well-known techniques. In these embodiments, a correlation can be determined between the respiratory phase of the user based on the measurements captured by the PPG sensor and the respiratory phase of the entity based on the audio data (as determined by the audio-based respiration determination module 304). If a threshold correlation exists, the user of the wearable device can be determined to be the source of the snoring sound captured in the audio data, as described below with reference to the similarity module 308. Other types of sensor data can be used to determine such correlations. For example, in some embodiments, respiration rate and phase information can be determined using lower-frequency variations in the DC level of the PPG signal captured by the PPG sensor. In some embodiments, respiration rate and phase information can be determined from variations in the amplitude of AC variations in the PPG signal corresponding to heartbeats. Such variations in the PPG signal may be caused by underlying physiological changes in the mechanical properties and chemical composition of blood and arteries in the tissue where the PPG signal is sensed by the PPG sensor. In some embodiments, the sensor-based respiration determination module 306 can obtain sensor data from one or more accelerometers in the wearable device. Generally speaking, an accelerometer can measure the rate of change of the wearable device's velocity. In some cases, these measurements can indicate that the user is snoring at various time intervals, such as Figure 5 , as shown in the accelerometer data 506 in the example. In various embodiments, the measurements made by the accelerometer can be used to determine (or estimate) a corresponding breathing phase for the user based on well-known techniques. For example, a user of a wearable device can place the wearable device on their chest. In this example, the measurements made by the accelerometer in the wearable device can be used to determine the breathing phase as the user inhales and exhales. In these embodiments, the magnitude of the correlation between the breathing phase of the user based on the measurements captured by the accelerometer and the breathing phase of the entity based on the audio data can be determined, as described below. Many variations are possible. For example, in various embodiments, other types of sensor data can be used to determine (or estimate) the breathing phase, such as a ballistocardiogram (BCG) pattern detected by the wearable device, an electrocardiogram (ECG) signal determined by the wearable device, or an impedance cardiogram (ICG) signal determined by the wearable device. For example, these signals can be determined based on dry electrodes in the wearable device that inject and sense current.

[0049] The similarity module 308 can be configured to determine the magnitude of the similarity (or correlation) between the breathing phase determined based on the audio data and the breathing phase determined based on the sensor data (e.g., a PPG sensor, an accelerometer, etc.). In various embodiments, the similarity module 308 can apply well-known techniques to determine the magnitude of the correlation between the breathing phase determined based on the audio data and the breathing phase determined based on the sensor data. For example, the similarity module 308 can determine the magnitude of the correlation based on autocorrelation or cross-correlation. In some embodiments, the similarity module 308 can determine that there is a threshold magnitude correlation between the breathing phase determined based on the audio data and the breathing phase determined based on the sensor data. If a threshold magnitude correlation exists, it can be determined that the snoring captured by the audio data originates from the user of the wearable device. In this case, the audio data can be associated with the user of the wearable device. In some embodiments, once this association is established, the snoring activity can be processed and analyzed to provide various insights to the user of the wearable device, such as Figures 6A to 6J As shown in the example of . Many variations are possible. For example, in some embodiments, the similarity module 308 can determine the magnitude of the correlation between breathing stages based on the respiratory (or respiration) rate value. For example, the respiratory rate value can indicate the frequency at which breathing is performed (e.g., one breath every 3 seconds). In some embodiments, the similarity module 308 can determine a first respiratory rate value for a respiratory stage determined based on the audio data. The first respiratory rate value can be determined based on frequency and stage information describing the respiratory stage determined based on the audio data. Similarly, the similarity module 308 can determine a second respiratory rate value for a respiratory stage determined based on the sensor data. The second respiratory rate value can be determined based on frequency and stage information describing the respiratory stage determined based on the sensor data. In these embodiments, it can be determined that when the first respiratory rate value matches the second respiratory rate value, the snoring captured by the audio data corresponds to the user of the wearable device. In some embodiments, the similarity module 308 can express the magnitude of the correlation as a probability, which measures the likelihood that the snoring activity captured by the audio data originates from the user of the wearable device. Many variations are possible.

[0050] The clustering module 310 can be configured to generate clusters of snoring activity based on the audio data. For example, in some cases, the audio data may capture multiple entities snoring over a period of time. In such cases, it may be difficult to distinguish snoring from one entity from snoring from another. To help distinguish between snoring entities, in various embodiments, the clustering module 310 may extract various spectral features from the audio data based on well-known techniques, such as Mel-Frequency Cepstrum (MFC), to generate clusters. For example, the extracted spectral features may be divided into different frequency bands. Furthermore, the energy magnitude of each band may be determined using a logarithmic scale. The bands may then be clustered based on their respective energy magnitudes relative to time. In various embodiments, bands representing snoring activity from different entities may be included in different corresponding clusters. For example, a band representing snoring activity from a first entity may be included in a first cluster, while a band representing snoring activity from a second entity may be included in a second cluster. In various embodiments, such clustering may be used to determine which snoring activity corresponds to the user wearing the wearable device. In the preceding example, it may be determined whether portions of the data included in a first cluster or a second cluster (e.g., a set of frequency bands within a time period) have a threshold magnitude correlation with respect to time with a respiratory (or respiration) rate determined (or estimated) based on a sensor in the wearable device. If a threshold magnitude correlation is determined between a portion of the data from the first cluster and the respiratory rate determined by the sensor of the wearable device, it may be determined that the snoring activity represented by the first cluster is generated by the user of the wearable device. Alternatively, if a threshold magnitude correlation is determined between a portion of the data from the second cluster and the respiratory rate determined by the sensor of the wearable device, it may be determined that the snoring activity represented by the second cluster is generated by the user of the wearable device. In various embodiments, once a cluster is determined to be associated with a user, the audio data represented by the cluster may be evaluated to determine various sleep-related and snoring-related information of the user, as described herein.

[0051] Figure 4 FIG. 4 illustrates an example of a snore metric module 402 according to an embodiment of the present disclosure. In some embodiments, Figure 2C The snore measurement module 258 may be implemented using the snore measurement module 402. Figure 4 As shown in the example of , the snore metric module 402 may include a distance determination module 404 and a snore data module 406 .

[0052] The distance determination module 404 can determine (or estimate) the distance (or proximity) between a user's wrist, on which a wearable device (e.g., wearable device 200 of FIG. 2 ) is worn, and the user's mouth. In various embodiments, the determined distance can be used to improve the accuracy of various snore metrics for the user. For example, in some embodiments, the determined distance can be used to improve the accuracy of a snore intensity determined for the user. Generally speaking, snore intensity can be determined based on the sound pressure level (SPL) measured by the wearable device. The accuracy of the SPL measured by the wearable device can be affected by the distance between the wearable device and the mouth of the user snoring. In various implementations, to help improve its accuracy, the SPL measured by the wearable device can be corrected based on the distance between the user's wrist, on which the wearable device is worn, and the user's mouth. For example, a snore that is determined to be very intense (e.g., 70 dB) due to the wearable device being positioned close to the user's mouth can be reclassified as moderate intensity (e.g., 50 dB) by taking into account the close distance between the wearable device and the user's mouth. In various embodiments, the distance determination module 404 may measure the distance based on a specified height of the user of the wearable device. For example, the user's arm length may be estimated based on the user's height. For example, the arm length may be estimated as a certain segment of the height. The estimated arm length may be used to correct the estimated distance. In various embodiments, the distance determination module 404 may determine (or estimate) the distance between the wearable device and the user's mouth based on the posture of the wearable device user. For example, in various embodiments, the distance determination module 404 may determine (or estimate) the user's wrist posture. In some embodiments, the wrist posture may be determined based on the orientation of the wearable device and the orientation of the wearable device relative to magnetic north. For example, the orientation of the wearable device may be determined based on the angle of a face of the wearable device (e.g., a display screen) and a gravity vector determined by one or more accelerometers in the wearable device. Further, the orientation of the wearable device may be determined based on one or more magnetometers in the wearable device. In some embodiments, location information accessible to the wearable device and / or a computing device communicating with the wearable device may be used to account for changes in magnetic north. For example, in some embodiments, WiFi signals may be used to determine whether the user is at home or elsewhere. In these embodiments, the determined position can be used to adjust the direction of the magnetic north pole, thereby improving the accuracy of the wearable device's posture. Once the wrist posture is determined, the distance determination module 404 can determine (or estimate) one or more sleep (or body) postures of the user. For example, the distance determination module 404 can determine the sleep posture by narrowing the space of potential sleep postures based on a rigid human posture model. The rigid human posture model can narrow the space of potential sleep postures based on the determined wrist posture and human musculoskeletal constraints.In various embodiments, the distance determination module 404 may also narrow the space of potential sleep postures based on the assumption that snoring entities typically sleep in a supine (or face-up) position. Generally speaking, sleeping in a non-supine position makes snoring more difficult because such a position is less likely to induce vibrations of relaxed tissue in the entity's throat as the entity inhales and exhales. In various embodiments, the distance determination module 404 may implement functionality that maps a set of inputs (such as the wearable device's orientation and posture) to one or more potential wrist postures and sleep postures. In some embodiments, the user's wrist and sleep posture may be determined (or estimated) based on a ballistocardiogram (BCG) pattern detected by the wearable device. For example, the BCG pattern may be determined by one or more accelerometers in the wearable device. In various embodiments, the BCG pattern may represent the propagation of heartbeats from the user's heart to the user's wrist where the wearable device is worn over a certain time period. In these embodiments, certain propagation patterns may indicate certain postures. Therefore, the BCG pattern may be correlated with the user's wrist posture and / or sleep posture. In various embodiments, the distance determination module 404 may implement functionality that maps the BCG pattern to one or more potential wrist postures and / or sleep postures.

[0053] The snore data module 406 can generate various snore information for the user based on the user's snoring activities. For example, the snore data module 406 can obtain audio data that records (or captures) the user's snoring activities during a certain period of time (e.g., at night). The snore data module 406 can evaluate the obtained audio data to determine the user's snoring information. For example, in some embodiments, the snore data module 406 can determine a snore level measurement based on the audio data of the user's snoring activities. For example, the snore level measurement can be determined based on the user's snoring activities recorded during a certain period of time. In some embodiments, each snore level measurement can provide a measurement of the sound detected and recorded by the audio data. For example, in some embodiments, the snore level measurement can be measured using decibels (dB). In various embodiments, the snore level measurement can be represented using one or more visual graphs, such as Figures 6A to 6J As shown in the example of . In some embodiments, the snore data module 406 can determine a sound level measurement based on the audio data of the user's snoring activity recorded during a certain time period. The sound level measurement can measure any sound detected during the certain time period, including ambient (or environmental) noise. In some embodiments, the snore data module 406 can determine the duration of the snoring activity. For example, once the snore data module 406 determines the snoring in the audio data, the snore data module 406 can determine (or estimate) the length of time during which the snoring occurred. In various embodiments, the duration of the snoring activity can be represented by one or more visual graphs, such as Figures 6A to 6JIn some embodiments, the snore data module 406 may determine a snore intensity measurement based on audio data of the user's snoring activity recorded over a certain period of time. For example, a snore intensity measurement of the snore may be determined by correcting the snore level measurement of the snore based on the distance from the microphone recording the snore to the mouth of the user generating the snoring sound. In some embodiments, multiple microphones may be used to record the user's snoring activity to improve distance estimation. In various embodiments, the snore intensity measurement may be represented by one or more visual graphs, such as Figures 6A to 6J as shown in the example.

[0054] In some embodiments, the snore data module 406 can determine various types of sound events based on the audio data of the user's snoring activity. For example, the sound events can correspond to ambient noises or sounds produced by the user. Further, the sound events can refer to snoring, sleep apnea events, motion noises, heavy breathing, or ambient noises, to name a few examples. In various embodiments, the sound events can be represented by one or more visual graphs, such as Figures 6A to 6J As shown in the example of . In some embodiments, such a visual map may provide an option (or visual indicator) for replaying sound events detected while the user was sleeping. In these embodiments, the option to replay the sound events may be selected. In some embodiments, the snore data module 406 may identify sound events that occurred before a change was detected in the user's sleep state. For example, if the user suddenly transitions from a deep sleep state to an awake state, the snore data module 406 may determine and extract any sound events that occurred before the user's sleep state changed. In this example, the snore data module 406 may provide an option to replay the sound events so that the user can more easily identify the source of the sounds that disturbed the user during sleep. Many variations are possible.

[0055] Figure 5An example diagram 500 illustrates audio and sensor data captured over a period of time. For example, diagram 500 illustrates audio data 502 captured by a microphone, plethysmogram (PPG) sensor data 504 captured by a PPG sensor (e.g., a green PPG sensor, a red PPG sensor, and an infrared PPG sensor), and sensor data 506 captured by an accelerometer over a period of time. In example diagram 500, audio data 502 represents spectral properties of audio detected and recorded while a user of a wearable device (e.g., wearable device 200 of FIG. 2 ) is asleep. Similarly, PPG sensor data 504 captured by the PPG sensor represents physiological measurements recorded by the wearable device while the user is asleep. Further, sensor data 506 captured by the accelerometer represents motion or vibration detected by the wearable device while the user is asleep. Diagram 500 helps visualize the correlation between audio and sensor data that can be used to determine various information about the user, as described herein. For example, audio data 502, PPG data 504, and accelerometer data 506 can each be independently used to determine a breathing (or respiration) rate for a user. In another example, a correlation between audio data 502 and sensor data (e.g., PPG data 504, accelerometer data 506) can be used to determine whether the user of the wearable device is snoring, as described above. Many variations are possible.

[0056] Figure 6AAn example 600 of an interface 604 according to an embodiment of the present disclosure is illustrated. Interface 604 is presented on a display screen of a computing device (such as computing device 602) or a wearable device (e.g., wearable device 200 of FIG. 2 ). Interface 604 can be provided by an application (e.g., a web browser, a fitness application, etc.) running on computing device 602. In this example, interface 604 provides a snore map 606 that plots various information determined for a user of the wearable device. For example, snore map 606 includes a first visual graph 608 representing the user's sleep state detected by the wearable device. For example, the first visual graph can visually represent changes in the user's sleep state over a certain time period while the user sleeps. In some embodiments, the user's sleep state can be categorized as one of an awake state, a rapid eye movement (REM) state, a light sleep state, and a deep sleep state. Many variations are possible. Snore map 606 also includes a second visual graph 610 representing sound level measurements determined while the user sleeps over a certain time period. For example, each sound level measurement can represent the amount of sound detected at a certain point in time from audio data recording the user's sleep activity. Furthermore, the snore graph 606 also includes a third visual graph 612 representing snore intensity measurements determined while the user was sleeping during a certain time period. The snore graph 606 provides a detailed assessment of the user's sleep activity while allowing comparison of the user's sleep state, sound level measurements, and snore intensity at any given point in time during the user's sleep activity. In various embodiments, the snore graph 606 may provide an option for replaying sound events detected during the user's sleep activity. For example, Figure 6B The interface 604 in displays a message 614 indicating a sound event of interest detected during the user's sleep activity. The interface 604 also provides an option (or visual indicator) 616 that can be selected to replay a recording of the sound event of interest. Many variations are possible.

[0057] Figure 6CAn example 620 of an interface 624 according to an embodiment of the present disclosure is illustrated. Interface 624 is presented on a display screen of a computing device 602 or a wearable device. Interface 624 can be provided by an application (e.g., a web browser, a fitness application, etc.) running on computing device 602. Interface 624 can provide various information determined for the user of the wearable device. In this example, interface 624 provides a first visual graph 626 that plots changes in the user's sleep state over a certain time period. In some embodiments, the user's sleep state can be categorized as one of an awake state, a rapid eye movement (REM) state, a light sleep state, and a deep sleep state. Interface 624 also includes a second visual graph 628 that plots sound levels (e.g., decibel levels) measured over a certain time period while the user was sleeping. In various embodiments, interface 624 can provide information 630 describing the types of sound events detected while the user was sleeping. In some embodiments, information 630 can provide a total number of sound events that may have contributed to the user transitioning from a sleep state (e.g., REM, light sleep, deep sleep) to an awake state, as well as a corresponding number of specific types of sound events (e.g., snoring, heavy breathing, motion noise, etc.) that may have awakened the user. In some embodiments, information 630 may provide the total number of sound events detected during the entire user's sleep activity and a corresponding number of specific types of sound levels (e.g., snoring, heavy breathing, movement noise, etc.). In some embodiments, information 630 may provide suggestions (or insights) for improving sleep habits. For example, information 630 may inform the user that the user woke up several times last night after detecting noisy sound events. In another example, information 630 may inform the user that the snoring of another entity (e.g., the user's partner) may be affecting the user's sleep. In yet another example, information 630 may provide the user with access to one or more audio recordings of sound events (such as snoring and apnea events), which can be shared with the user's medical professional for further analysis. In yet another example, information 630 may inform the user of the overall sound level associated with their sleep environment compared to other sleep environments (e.g., "Your sleep environment is quieter than 90% of other users"). Many variations are possible.

[0058] Figure 6DAn example snore report 632 is illustrated, which can be provided for presentation via an interface of a computing device, such as computing device 602 or a wearable device (e.g., wearable device 200 of FIG. 2 ). The interface can be provided via an application (e.g., a web browser, a fitness application, etc.) running on computing device 602. Snore report 632 can provide various information determined for a user of the wearable device. Snore report 632 can provide a summary of the user's sleep activity for a given day. For example, snore report 632 can provide the total time the user was determined to be asleep during the day. Snore report 632 can also describe the user's snoring activity for the day, including the total duration of the snoring activity (e.g., "40 minutes") and / or a percentage representing the user's snoring activity relative to the time the user was determined to be asleep (e.g., "15%"). In some embodiments, snore report 632 can categorize and provide the user's snoring activity based on snoring intensity 634 (e.g., "moderate snoring detected"). For example, based on snoring intensity, snoring activity can be categorized as "quiet," "moderate," or "loud." In some embodiments, the snore report 632 may provide a visual graph 636 that plots the user's sleep activity for a day. For example, the visual graph 636 may identify portions 638 of sleep activity corresponding to snoring episodes. The visual graph 636 may also identify portions 640 of sleep activity during which ambient noise was detected, and may also indicate the corresponding noise level (e.g., in dBA). Further, the visual graph 636 may identify portions 642 of sleep activity during which the user was determined to be awake. In some embodiments, the snore report 632 may provide a noise level analysis 644 that may indicate one or more of: a baseline noise level during the user's sleep activity, an average noise level during the user's sleep activity, a maximum noise level during the user's sleep activity, and a number of sound events detected during the user's sleep activity. Many variations are possible.

[0059] Figure 6EAnother example snore report 648 is illustrated that can be provided for presentation via an interface of a computing device, such as computing device 602 or a wearable device (e.g., wearable device 200 of FIG. 2 ). The snore report 648 can provide various information determined for a user of the wearable device. In various embodiments, the snore report 648 can provide a summary of the user's sleep activity for a given day, as determined using any of the methods described herein. For example, the snore report 648 can provide the total time the user was determined to be in bed (e.g., “in bed for 6 hours and 7 minutes”), the total time the user was determined to be asleep (e.g., “asleep for 5 hours and 24 minutes”), and the total time the user was determined to be snoring (e.g., “snoring for 1 hour and 0 minutes”). In some embodiments, the snore report 648 can categorize and provide the user's snoring activity based on snoring intensity 650 (e.g., “rare snoring detected”). In some embodiments, the snore report 648 may indicate a percentage range representing the user's snoring activity relative to the amount of time the user was determined to be asleep (e.g., "You snored between 10% and 40% of your total sleep time."). In some embodiments, the snore report 648 may provide a visual graph 652 that plots the user's sleep activity for a day. For example, the visual graph 652 may identify portions of sleep activity during which ambient noise was detected and may also indicate the corresponding noise level (e.g., in dBA). Further, the visual graph 652 may identify portions of sleep activity during which the user was determined to be awake. In some embodiments, the snore report 648 may indicate portions 654 of sleep activity during which the user was determined to be asleep. Many variations are possible.

[0060] Figure 6F Another example snore report 658 is illustrated, which can be provided for presentation via an interface of a computing device, such as computing device 602 or a wearable device (e.g., wearable device 200 of FIG. 2 ). Snore report 658 can provide various information determined for a user of the wearable device. In various embodiments, snore report 658 can provide a summary of sleep for various time periods (e.g., weekly, monthly, etc.). For example, snore report 658 can provide an average amount of sleep for a user over a certain time period (e.g., “6 hours and 6 minutes on average”). Further, for a given day 660, snore report 658 can indicate the total time the user was determined to be asleep and the total time the user was determined to be snoring. In some embodiments, snore report 658 can provide a snore level (or classification) 662 for the day. Snore level 662 can be determined based on a category of the user's snoring intensity for the day. For example, depending on the snoring intensity, the noise level 662 can be classified as “none to light,” “moderate,” or “loud.” Naturally, many variations are possible.

[0061] Figure 6GSound level information 664 is illustrated, which can be provided for presentation via an interface of a computing device, such as computing device 602 or a wearable device (e.g., wearable device 200 of FIG. 2 ). Sound level information 664 can provide various information determined for a user of the wearable device. For example, in some embodiments, sound level information 664 can be provided in a snoring report. Sound level information 664 can indicate a typical sound level for a user's sleeping location (e.g., a bedroom). For example, the sound level can be determined based on the amount of snoring and background noise detected at the user's sleeping location when the user was determined to be asleep.

[0062] Figures 6H to 6J Various types of error information are illustrated that may be provided for presentation via an interface of a computing device, such as computing device 602 or a wearable device (e.g., wearable device 200 of FIG. 2 ). For example, in some embodiments, error information may be provided in a snore report to assist a user of the wearable device in resolving sleep detection and / or snore detection issues. For example, Figure 6H The error message 668 is shown, indicating that the user's snoring cannot be recorded due to inconsistent sleep patterns. The error message 668 may also provide instructions for resolving the error. In this example, the error message 668 may require the user to turn on the manual recording mode before going to bed in order to record the user's snoring. Figure 6K In another example, Figure 6I An error message 670 is illustrated indicating that the snoring recorded for the user may be inconclusive because the battery of the wearable device is depleted. The error message 670 may require the user to charge the wearable device to a certain battery level before going to bed. In yet another example, Figure 6J An error message 672 is shown indicating that the user's snoring cannot be recorded due to excessive background noise. The user can then take steps to reduce the background noise so that the user's snoring can be more completely recorded. Many variations are possible.

[0063] Figure 6K An interface 674 is illustrated with options for configuring a wearable device (e.g., wearable device 200 of FIG. 2 ). Interface 674 can be accessed via a computing device (such as computing device 602 or a wearable device). As shown, interface 674 can provide options to turn on automatic snore detection or manual snore detection. Interface 674 can also provide an option to disable snore detection entirely. Further, interface 674 can indicate a current noise level 676 detected by the wearable device. Many variations are possible.

[0064] Figure 7An example method 700 according to an embodiment of the present technology is illustrated. Naturally, the method 700 may include additional, fewer, or alternative steps, and unless otherwise stated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0065] At block 704, audio data may be captured over a period of time. The audio data may capture sounds occurring when the user of wearable device 702 is determined to be asleep. Depending on the embodiment, the audio data may be captured by one or more microphones in wearable device 702 (e.g., wearable device 200 in FIG. 2 ), one or more independent microphones separate from wearable device 702, or a combination thereof. At block 706, the captured audio data may be segmented into one or more audio frames, each representing a portion of the captured audio data. At block 708, a corresponding feature set may be extracted from each audio frame. For example, the feature set extracted from the audio frame may include Mel-Frequency Cepstral Coefficients (MFCCs), which describe the frequency spectrum represented by the audio frame. At block 710, the feature set may be classified as one or more sounds detected in the captured audio data, as described herein. At block 712, after the one or more sounds are classified, the captured audio data may be discarded (or deleted). In some embodiments, each audio frame may be immediately deleted once analyzed and classified. Many variations are possible.

[0066] Figure 8A An example method 800 according to an embodiment of the present technology is illustrated. Naturally, the method 800 may include additional, fewer, or alternative steps, and unless otherwise indicated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0067] At block 802, one or more breathing phase patterns may be determined over a time period using audio data captured by at least one microphone, the audio data including a plurality of snoring sounds. At block 804, breathing phase patterns included in the time period may be determined based at least in part on sensor data captured by one or more sensors in an electronic device. At block 806, a first breathing phase pattern represented by the audio data and a second breathing phase pattern represented by the sensor data are determined to be correlated. At block 808, a determination is made that both the first breathing phase pattern represented by the audio data and the second breathing phase pattern represented by the sensor data correspond to a user wearing the electronic device.

[0068] Figure 8BAn example method 850 according to an embodiment of the present technology is illustrated. Naturally, the method 850 may include additional, fewer, or alternative steps, and unless otherwise indicated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0069] At block 852, when a user of the electronic device is determined to be asleep, an audio signal may be captured using at least one microphone. At block 854, at least one audio frame may be determined from the audio signal. The audio frame may represent a frequency spectrum detected by the at least one microphone over a period of time. At block 856, one or more sounds associated with the at least one audio frame may be determined. At block 858, sleep-related information may be generated. The sleep-related information may identify the one or more sounds as potential sources of sleep disruption.

[0070] Figure 9 An example method 900 according to an embodiment of the present technology is illustrated. Naturally, the method 900 may include additional, fewer, or alternative steps, and unless otherwise stated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0071] At block 904, one or more sensors in a wearable device 902 (e.g., wearable device 200 of FIG. 2 ) may capture sensor data over a certain time period. For example, the sensor data may be data captured by one or more plethysmography (PPG) sensors in wearable device 902. In another example, the sensor data may be data captured by one or more accelerometers in wearable device 902. Similarly, at block 904, one or more microphones may capture audio data during the same time period as the sensor data. The audio data may be captured by a snoring entity. Depending on the embodiment, the audio data may be captured by one or more microphones in wearable device 902, one or more independent microphones separate from wearable device 902, or a combination thereof. At block 906, a respiratory phase may be determined (or estimated) based on the sensor data captured by the one or more sensors in wearable device 902. At block 910, a respiratory phase may be determined (or estimated) based on the audio data captured by the one or more microphones. At block 912, a determination is made as to whether the respiratory phase represented by the sensor data corresponds (or correlates) with the respiratory phase represented by the audio data, as described above. If a correspondence exists, it may be determined that the snoring captured in the audio data originates from the user of wearable device 902. At block 914, a probability is measured of the likelihood that the snoring captured by the audio data is associated with the user of wearable device 902. Many variations are possible.

[0072] Figure 10An example method 1000 according to an embodiment of the present technology is illustrated. Naturally, the method 1000 may include additional, fewer, or alternative steps, and unless otherwise indicated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0073] At block 1004, one or more sensors in a wearable device 1002 (e.g., wearable device 200 of FIG. 2 ) may capture sensor data over a period of time. For example, the sensor data may be data captured by one or more accelerometers and magnetometers in the wearable device 1002. At block 1006, one or more wrist and sleep postures may be determined for the user of the wearable device 1002. For example, the wrist and sleep postures may be determined based on the captured sensor data, as described above with reference to FIG. Figure 4 At block 1008, the user's measurements (e.g., height) may be used to refine the user's wrist and sleep posture. At block 1010, the distance between the wearable device 1002 and the user's mouth may be determined (or estimated) based on the user's wrist and sleep posture, as described above with reference to FIG. Figure 4 Described. Many variations are possible.

[0074] Figure 11 An example method 1100 according to an embodiment of the present technology is illustrated. Naturally, the method 1100 may include additional, fewer, or alternative steps, and unless otherwise stated, these steps may be performed separately or in parallel within the scope of the various embodiments discussed herein.

[0075] At box 1102, one or more sensors in a wearable device (e.g., wearable device 200 of Figure 2) can capture audio and sensor data over a period of time. For example, the audio data can be recorded (or captured) by one or more microphones in the wearable device. The audio data can record sounds detected while the user of the wearable device is asleep. For example, the sensor data can be captured by one or more accelerometers and magnetometers in the wearable device. At box 1104, motion noise recorded in the audio data can be segmented (or deleted) from the audio data to improve snoring detection accuracy. In some embodiments, motion noise can be identified by correlating motion detected by sensors in the wearable device (such as one or more accelerometers and a plethysmogram (PPG) sensor). In some embodiments, motion noise can be segmented by pre-processing the audio data, as described above with reference to Figure 9At block 1106, the audio and sensor data may be used to detect snoring sounds produced by the user during sleep and a snoring level measurement, as described above. At block 1108, the snoring level measurement may be corrected based on the distance between the user's mouth and the microphone recording the snoring sounds originating from the user's mouth, as described with reference to FIG. Figure 10 At block 1110, snoring intensity and duration measurements may be determined for the user, as described above. At block 1112, a snorogram may be generated that visualizes the user's sleep state, sound level measurements, and changes in snoring intensity over a period of time, such as Figure 6A Many variations are possible.

[0076] Naturally, the various embodiments described herein may have many users, applications, and / or variations. For example, in some cases, a user may choose whether to opt in to utilizing the disclosed technology. The disclosed technology may also ensure that various privacy settings and preferences are maintained and prevent the disclosure of private information.

[0077] Throughout this specification, multiple instances can implement the components, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as separate operations, one or more individual operations can be performed simultaneously and do not require the operations to be performed in the order shown. The structures and functions presented as independent components in the example configurations can be implemented as combined structures or components. Similarly, the structures and functions presented as individual components can be implemented as independent components. These and other variations, modifications, additions and improvements are within the scope of the subject matter in this article.

[0078] Although the overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. These embodiments of the subject matter may be referred to herein individually or collectively by the term "invention" for convenience only, and are not intended to voluntarily limit the scope of this application to any single disclosure or concept, if in fact more than one is disclosed.

[0079] The embodiments described herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description is not to be construed in a limiting sense, and the scope of the various embodiments is defined solely by the appended claims and the full scope of equivalents to which such claims are entitled.

[0080] It will be appreciated that a "module," "system," "data store," and / or "database" may include software, hardware, firmware, and / or circuitry. In one example, one or more software programs comprising instructions executable by a processor may perform one or more functions of the modules, data stores, databases, or systems described herein. In another example, circuitry may perform the same or similar functions. Alternative embodiments may include more, fewer, or functionally equivalent modules, systems, data stores, or databases and still be within the scope of the present embodiments. For example, the functionality of the various systems, modules, data stores, and / or databases may be combined or partitioned in different ways.

[0081] As used herein, the term "or" may be interpreted as meaning inclusive or exclusive. In addition, multiple instances may be provided for the resources, operations, or structures described herein as a single instance. Additionally, the boundaries between the various resources, operations, modules, and data stores are somewhat arbitrary, and specific operations are described in the context of specific illustrative configurations. Other allocations of functionality are conceivable and may be within the scope of the various embodiments of the present disclosure. In general, structures and functions presented as independent resources in the example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are considered to be illustrative rather than restrictive.

[0082] Unless specifically stated otherwise or understood otherwise in the context of use, conditional language, such as "may," "could," "might," and "might," is generally intended to convey that some embodiments include, while other embodiments do not, certain features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that one or more embodiments in any way require features, elements, and / or steps or that the one or more embodiments must include logic for deciding whether to include or perform such features, elements, and / or steps in any particular embodiment, with or without user input or prompting.

[0083] Although the present invention has been described in detail for purposes of illustration based on what are presently considered to be the most practical and preferred embodiments, it is to be understood that such detail is solely for that purpose and that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it is to be understood that the present invention contemplates that, to the extent possible, one or more features of any embodiment may be combined with one or more features of any other embodiment.

Claims

1. An electronic device comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the electronic device to: activating and deactivating at least one microphone based on a breathing phase pattern determined for a user determined to be asleep; capturing an audio signal when the at least one microphone is activated; determining that an audio frame of the audio signal includes one or more sounds indicative of snoring; responsive to determining that the audio frame includes the one or more sounds indicative of snoring, discarding the audio frame; as well as Sleep-related information is generated, wherein the information identifies the one or more sounds as instances of snoring.

2. The electronic device according to claim 1, wherein The audio frame of the audio signal determined to include one or more sounds indicative of snoring occurred while the audio signal was being captured.

3. The electronic device according to claim 1, wherein The at least one microphone for capturing the audio signal is included in one or both of the electronic device or a computing device separate from the electronic device.

4. The electronic device according to claim 1, wherein The instructions further cause the electronic device to execute: The generated sleep-related information is provided via an interface accessible via the electronic device or a computing device separate from the electronic device.

5. The electronic device according to claim 4, wherein The sleep-related information provides one or more of: a total time the user was determined to be sleeping, one or more time periods during which the user was determined to be awake, or one or more time periods during which the user was determined to be snoring. The electronic device according to claim 5 , wherein: The sleep-related information provides one or more of the following: the total time the user was snoring, the percentage of the time the user was snoring relative to the total time the user was determined to be asleep, or the percentage range of the time the user was snoring relative to the total time the user was determined to be asleep.

7. The electronic device according to claim 5, wherein: The sleep-related information provides a classification of the one or more time periods during which the user was determined to be snoring based on snoring intensity.

8. The electronic device according to claim 7, wherein: The classification labels the one or more time periods during which the user was determined to be snoring as first level snoring, second level snoring, or third level snoring.

9. The electronic device according to claim 5, wherein: The sleep-related information identifies a correlation between one or more time periods during which the user is determined to be awake and the one or more sounds identified as potential sleep interrupters.

10. The electronic device according to claim 5, wherein The sleep-related information provides one or more of: a baseline noise level while the user is asleep, an average noise level while the user is asleep, a maximum noise level while the user is asleep, or a total number of sounds detected while the user is asleep.

11. The electronic device according to claim 1, wherein Activating and deactivating at least one microphone based on a breathing phase pattern determined for a user determined to be asleep includes: activating the at least one microphone when the user inhales according to the breathing phase pattern; and The at least one microphone is disabled when the user exhales according to the breathing phase pattern.

12. A method comprising: capturing audio signals via one or more microphones while the user is asleep; determining, via one or more processors, that an audio frame of the audio signal includes one or more sounds that are potentially disruptive to sleep; in response to determining that the audio frame includes one or more sounds that are potentially disruptive to sleep, discarding the audio frame via the one or more processors; as well as Sleep-related information is generated for the user via the one or more processors, wherein the sleep-related information identifies one or more sources of the one or more sounds.

13. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform a method comprising: Capturing audio signals while the user is asleep; determining that an audio frame of the audio signal includes one or more sounds that are potentially disruptive to sleep; responsive to determining that the audio frame includes one or more sounds that are potentially disruptive to sleep, discarding the audio frame; as well as Sleep-related information of the user is generated, wherein the sleep-related information identifies one or more sources of the one or more sounds.

Images