Determining breathing rates based on audio streams and user conditions

Abstract

The disclosure relates to determining a breathing rate based on an audio stream and a user condition. The system may receive an input indicative of a user condition. The system may also receive an internal audio stream from an in-ear microphone of the head-mounted system and an external audio stream from an external microphone. The system may determine a respiration rate of the user based on the internal audio stream, the external audio stream, and the input indicative of the condition of the user. In some implementations, the respiration rate may be determined from respiration signals in the internal audio stream and / or the external audio stream. The respiration signal may measure the respiration of the user. In some implementations, the system may invoke a machine learning model to determine the breathing signal from the internal audio stream and / or the external audio stream based on the user condition.

Description

[0001] Related patent applications

[0002] This patent application claims the benefit of priority to U.S. Provisional Application No. 63 / 593,186, filed on October 25, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to health monitoring, and more particularly to determining a user's breathing rate based on an audio stream and a user condition. Other aspects are also described. Background Art

[0004] Respiratory rate is a measure of a person's breathing and may be calculated as the number of breaths per unit time. A breathing cycle may include inhalation and exhalation performed by a person. A person's respiratory rate may increase with exercise and other activities and may decrease with rest. A person's respiratory rate may be one of several measurements that may be used to determine a person's health and activity. Summary of the invention

[0005] Specific implementations of the present disclosure include utilizing inputs that provide situational awareness to the system so that the system can determine the user's breathing rate based on different utilization rates of internal audio streams and / or external audio streams. Some specific implementations may include a breathing detection system that is configured to receive input indicating a user's condition, an internal audio stream from an in-ear microphone of a head-mounted system, and an external audio stream from an external microphone of the head-mounted system. For example, the head-mounted system may include earplugs that include an in-ear microphone and an external microphone. In another example, the head-mounted system may include earplugs that include an in-ear microphone; and headphones or goggles that include an external microphone. The in-ear microphone may be configured to pick up sounds in the user's ear cavity, and the external microphone may be configured to pick up sounds outside the head-mounted system. The input may include user input and / or sensor input for indicating the user's condition. For example, the input may come from a matching device such as a smart phone, a smart watch, a wearable device, a head-mounted system, or another mobile device utilized by a user. User input may include a user indicating a user condition, such as a user indicating a type of exercise being performed (e.g., push-ups or running) or an indication of activity such as the user walking, meditating, eating, drinking, or resting. Sensor input may include an indication of a user condition via one or more sensors, indicating a user's vital signs (e.g., heart rate, temperature), a user's movement (e.g., repetitive, such as walking, running, jumping, or lifting), or a user's location (e.g., indoors or outdoors).

[0006] The breathing detection system may determine the user's breathing rate based on the internal audio stream, the external audio stream, and an input indicating a user condition. For example, the breathing rate may be determined based on the user condition based on one or more breathing signals from the internal audio stream and / or the external audio stream that measure the user's breathing. In some specific implementations, one or more breathing signals may be weighted for the breathing rate to measure the user's breathing. In some specific implementations, the breathing detection system may call a machine learning model to determine the breathing signal from the internal audio stream and / or the external audio stream. The machine learning model may determine the breathing signal based on the user condition based on the utilization of the internal audio stream and the external audio stream indicating the user condition. The breathing detection system may determine the user's breathing rate based on the breathing signal. Other aspects are also described and protection is claimed for other aspects.

[0007] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced by all suitable combinations of the various aspects summarized above and disclosed in the detailed description below and specifically pointed out in the claims section. Such combinations may have specific advantages not specifically set forth in the above summary of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various aspects of the present disclosure are described by way of example and are not limited to the illustrations of the various drawings, in which similar reference numerals indicate similar elements. It should be noted that reference to "one" or "an" aspect in the present disclosure is not necessarily the same aspect, and it means at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one aspect of the present disclosure, and for a given aspect, not all elements in the drawing may be required.

[0009] Figure 1 is an example of a breathing detection system for determining a user's breathing rate based on an audio stream and the user's condition.

[0010] Figure 2 is an example of an internal audio stream from an in-ear microphone and an external audio stream from an external microphone received during meditation.

[0011] Figure 3 is an example of an internal audio stream from an in-ear microphone and an external audio stream from an external microphone received during a workout.

[0012] Figure 4 is an example of an enhancement system for enhancing an audio stream for use in determining a user's breathing rate.

[0013] Figure 5 is a flow diagram of an example of a process for determining a user's breathing rate based on an audio stream and a user's condition. DETAILED DESCRIPTION

[0014] The user's breathing rate may be determined using an audio stream from a microphone. For example, the user may wear earbuds with a microphone that generates an audio stream. The audio stream may include a breathing signal from which the breathing rate may be calculated. For example, the breathing signal may include a breathing-aligned spectral signature associated with the user's breathing (e.g., cycles of inhalation and exhalation).

[0015] In some cases, an internal audio stream from an in-ear microphone may provide a stronger breathing signal for determining breathing rate than an external audio stream from an external microphone. For example, during meditation, an in-ear microphone may be more suitable for detecting a user's breathing. In other cases, an external audio stream from an external microphone may be more suitable for detecting a user's breathing. For example, during exercise, an in-ear microphone may detect more interfering source signals, such as the sound from a user running on a treadmill, causing the internal audio stream to be distorted. Therefore, the external microphone is more suitable for detecting the user's breathing. What is needed is a system that can reliably determine a user's breathing rate.

[0016] Specific implementations of the present disclosure solve problems such as these by utilizing inputs that provide situational awareness to the system so that the system can determine the user's breathing rate based on different utilization rates of internal audio streams and / or external audio streams. Some specific implementations may include a breathing detection system that is configured to receive input indicating a user's condition, an internal audio stream from an in-ear microphone of a head-mounted system, and an external audio stream from an external microphone of the head-mounted system. For example, the head-mounted system may include earplugs that include an in-ear microphone and an external microphone. In another example, the head-mounted system may include earplugs that include an in-ear microphone; and headphones or goggles that include an external microphone. The in-ear microphone may be configured to pick up sounds in the user's ear cavity, and the external microphone may be configured to pick up sounds outside the head-mounted system. The input may include user input and / or sensor input for indicating the user's condition. For example, the input may come from a matching device such as a smart phone, a smart watch, a wearable device, a head-mounted system, or another mobile device utilized by the user. User input may include a user indicating a user condition, such as a user indicating a type of exercise being performed (e.g., push-ups or running) or an activity such as the user walking, meditating, eating, drinking, or resting. Sensor input may include an indication of a user condition via one or more sensors, indicating a user's vital signs (e.g., heart rate, temperature), a user's movement (e.g., repetitive, such as walking, running, jumping, or lifting), or a user's location (e.g., indoors or outdoors).

[0017] The breathing detection system may determine the breathing rate of the user based on the internal audio stream, the external audio stream, and the input indicating the user's condition. For example, the breathing rate may be determined according to the user's condition based on one or more breathing signals that measure the user's breathing from the internal audio stream and / or the external audio stream. In some specific implementations, one or more breathing signals may be weighted for the breathing rate to measure the user's breathing. In some specific implementations, the breathing detection system may call a machine learning model to determine the breathing signal from the internal audio stream and / or the external audio stream. The machine learning model may determine the breathing signal based on the user's condition based on the utilization of the internal audio stream and the external audio stream indicated by the user's condition. The breathing detection system may determine the user's breathing rate based on the breathing signal. Therefore, the breathing detection system can reliably determine the user's breathing rate in many different situations.

[0018] In some implementations, to detect the user's breathing (e.g., rate and / or phase), the breathing detection system can utilize audio streams from both the inward-facing microphone and the outward-facing microphone on the earbud. This can enable the breathing detection system to resolve a wider array of aggressor signals associated with sounds (e.g., music, phone calls, speech, and other ambient noises) that can affect each of these microphones differently. For example, the breathing detection system can utilize blind source separation on both channels (e.g., from the inward-facing microphone and the outward-facing microphone) to extract the spectral signature of breathing from the noise.

[0019] In some implementations, using two audio streams can provide situational awareness to discern between: (1) high noise, more distorted sensor data and (2) low noise, better sensor signal clarity conditions. Using multiple audio channels can enable separation of source signals from nuisance source signals to generate a respiration-related signal. In some cases, the system can enhance the respiration signal using blind source separation to reduce the effects of nuisance source signals such as background noise (e.g., speech and music) and / or nuisance source signals generated by earbuds (e.g., music or speech from earbud speakers). Reducing such nuisance source signals can improve the audio quality of the recovered audio sensor data to enable accurate estimation of respiration rate and other respiration-related metrics.

[0020] Various aspects of the present disclosure will now be explained with reference to the accompanying drawings. Whenever the shapes, relative positions, and other aspects of the components described are not clearly defined, the scope of the present invention is not limited to the components shown, which are only used for illustrative purposes. In addition, although many details are set forth, it should be understood that some aspects of the present disclosure can be implemented without these details. In other cases, well-known circuits, structures, and technologies are not shown in detail to avoid obscuring the understanding of the description.

[0021] Figure 1 is an example of a breathing detection system 100 for determining a user's breathing rate based on an audio stream and a user's condition. In various specific implementations, the breathing detection system 100 may receive an input 102 that provides situational awareness to enable the breathing detection system 100 to determine a person's breathing rate based on different utilizations of an internal audio stream and an external audio stream. The breathing detection system 100 may include a head-mounted system 104, an input circuit, and a machine learning model. For example, the head-mounted system 104 may include earplugs, goggles, or glasses, which may also include virtual reality (VR), augmented reality (AR), or mixed reality. In addition, the input circuit may include a first filter circuit 106, a second filter circuit 108, a source separator 110, and a feature extractor 112. In another example, the machine learning model may include a recursive network 114 and a fully connected layer 116.

[0022] The head-mounted system 104 may include an in-ear microphone and an external microphone. The in-ear microphone may generate an internal audio stream. For example, the in-ear microphone may be configured to pick up sounds in the ear cavity of the user and generate an internal audio stream based on the sounds. In addition, the external microphone may generate an external audio stream. For example, the external microphone may be configured to pick up sounds outside the head-mounted system 104 and generate an external audio stream based on the sounds. In some cases, the head-mounted system 104 may also include a speaker for playing audio data (such as music, sounds from a phone call, and / or noise reduction). In one example, the head-mounted system 104 may include an earplug that includes an in-ear microphone, an external microphone, and / or a speaker. In another example, the head-mounted system 104 may include an earplug that includes an in-ear microphone; and a headset or goggles that include an external microphone. In another example, the head-mounted system 104 may include a pair of earplugs, each of which includes an in-ear microphone, an external microphone, and / or a speaker.

[0023] The input circuit may receive an internal audio stream and an external audio stream as sources in separate channels. For example, the first filter circuit 106 may receive the internal audio stream as a first source in a first channel, and the second filter circuit 108 may receive the external audio stream as a second source in a second channel. The first filter circuit 106 and the second filter circuit 108 may each include analog and / or digital circuits to perform low-pass filtering in order to generate a time spectrum or time-frequency representation of the audio data (sound) picked up by the corresponding microphone in an audible range (e.g., 0 kHz to 2 kHz). For example, the first filter circuit 106 may include low-pass filtering to generate a time-frequency representation of the sound from the internal audio stream, and the second filter circuit 108 may include low-pass filtering to generate a time-frequency representation of the sound from the external audio stream. For example, referring to Figure 2, showing a time-frequency representation of an inner audio stream (graph "A") and a time-frequency representation of an outer audio stream (graph "B").

[0024] The source separator 110 may receive a time-frequency representation of the sound from the internal audio stream and the external audio stream in a separate channel. In some specific implementations, the source separator 110 may perform additional signal conditioning, such as amplifying the time-frequency representation in the separate channel. The feature extractor 112 may receive the time-frequency representation in the separate channel from the source separator 110. The feature extractor 112 may perform digital signal processing (DSP) to extract features from the internal audio stream and / or the external audio stream based on the time-frequency representation. For example, the feature extractor 112 may utilize DSP algorithms such as Mel filter bank energy (MFE), Mel frequency cepstral coefficients (MFCC) or spectrograms according to various parameters such as specified frame length, frame stride, frequency band, normalization and / or background noise. The feature extractor 112 may extract features from the time-frequency representation for inference generation by a machine learning model.

[0025] The machine learning model may receive features from the feature extractor 112. The machine learning model may be, for example, or may include one or more of a neural network (e.g., a gated recurrent network, a multilayer perceptron, a convolutional neural network, a recurrent neural network, a deep neural network, or other neural network), a decision tree, a vector machine, a Bayesian network, a cluster-based system, a genetic algorithm, a deep learning system separate from a neural network, or other machine learning models. For example, the machine learning model may include a recurrent network 114 to receive features from the feature extractor 112. The recurrent network 114 may implement a gated recurrent network with a bidirectional flow (e.g., a 2-layer GRU), which enables outputs from one or more nodes to affect subsequent inputs to the same node (e.g., the recurrent network 114 may utilize internal states to process input sequences). The machine learning model may also include a fully connected layer 116 to receive outputs from the recurrent network 114. For example, the fully connected layer 116 may include a multilayer perceptron that implements a feedforward neural network that learns features to generate inferences.

[0026] Input 102 may include user input (e.g., from a user) and / or sensor input (e.g., from one or more sensors) for indicating a user condition. For example, input 102 may be provided via head mounted system 104 or another device such as a companion device (e.g., a smart phone, smart watch, wearable device, or other mobile device utilized by a user). User input may include a user directly indicating a user condition, such as by the user inputting the type of exercise being performed (e.g., push-ups or running) or an activity such as walking, meditation, eating, drinking, or resting. For example, a user may input information into a device via an application running on the device to indicate a user condition, such as by typing in the type of exercise or activity, or selecting an icon associated with the type of exercise or activity. Sensor input may indicate a user condition via measurements from one or more sensors of the device. Measurements may indicate a user's vital signs (e.g., heart rate, temperature), a user's movement (e.g., repetition, such as walking, running, jumping, or lifting), or a user's location (e.g., indoors or outdoors, or within a geo-fence). For example, the one or more sensors may include one or more gyroscopes, accelerometers, inertial measurement units (IMUs), proximity sensors, ambient light sensors, location sensors (e.g., global positioning systems and / or Wi-Fi), pedometers, heart rate sensors, and / or blood pressure sensors. Sensor inputs may enable a user to indirectly indicate a user condition, such as by sensing, through one or more sensors, a type of exercise performed by a user (e.g., push-ups or running) or an indication of an activity such as the user walking, meditating, eating, drinking, or resting. For example, one or more sensors may enable an application running on the device to sense that the user is performing that type of exercise or activity.

[0027] The breathing detection system 100 may determine the user's breathing rate (and / or phase) based on the internal audio stream, the external audio stream, and the input 102 indicating the user's condition, such as the number of breaths per unit time, each breath including an inhalation and an exhalation. The breathing rate may be generated by the output generated by the machine learning model based on the input through the input circuit. For example, the input circuit such as the feature extractor 112 may receive the input 102 indicating the user's condition. The input 102 may cause the feature extractor 112 to extract features from the audio stream in different ways according to the indicated user condition. In some cases, the user condition may cause the breathing detection system 100 (e.g., via the machine learning model receiving the feature input) to give a greater weight to one of the audio streams (e.g., the internal audio stream) and a smaller weight to another audio stream (e.g., the external audio stream) in the audio stream. For example, when the intrusion source signal in the internal audio stream is below a threshold (e.g., indicated by a user condition associated with meditation), the breathing detection system 100 may apply a greater weight to the internal audio stream. In another example, when there are more aggressor signals in the internal audio stream than in the external audio stream (e.g., indicated by a user condition associated with running), the breathing detection system 100 may place a greater weight on the external audio stream. In other cases, the user condition may cause one of the audio streams (e.g., the internal audio stream) to be gated (e.g., blocked) and cause another of the audio streams (e.g., the external audio stream) to be passed to the machine learning model. Thus, the breathing rate may be determined based on the user condition based on one or more breathing signals measuring the user's breathing in one or more streams. Thus, the breathing detection system 100 can reliably determine the user's breathing rate in many different situations corresponding to the user condition.

[0028] By way of example, see also Figure 2 , the breathing detection system 100 may receive an internal audio stream from an in-ear microphone (graph "A") and an external audio stream from an external microphone (graph "B"). Graphs A and B are time-spectrum analysis graphs, with time along the x-axis and spectral content or frequency on the y-axis, as shown. The "bright" portion of each column is the frequency band where the signal is strongest and therefore more easily detectable. It can be seen that these strong frequency bands repeat over time. The strong frequency bands occur between the end of the inspiration phase and the beginning of the next exhalation phase (e.g., Figure 2 ). The strong frequency band is more visible in graph A than in graph B. A reference (e.g., ground truth) graph is overlaid on graphs A and B as shown, which indicates the user's actual breathing measured using additional equipment (e.g., a sensor band worn by the user).

[0029] Please note that Figure 2The audio stream shown is during a first period corresponding to meditation. Meditation can be a user condition indicated by input 102. For example, meditation can be indicated by user input (e.g., the user selects meditation in an application) and / or by sensor input (e.g., one or more sensors indicate less user movement during the first period).

[0030] During meditation, there may be fewer distractor signals (e.g., music, phone calls, speech, and other ambient noise) in any audio stream. For example, the distractor signal in the internal audio stream may be below a threshold. Thus, the breathing detection system 100 may utilize the internal audio stream more (e.g., give more weight and / or pass the signal) and utilize the external audio stream less (e.g., give less weight and / or gate the signal) to determine a breathing signal from which a breathing rate may be calculated. For example, the breathing detection system 100 may better distinguish a breathing signal from distractor signals in the internal audio stream. Thus, the breathing detection system 100 utilizes the internal audio stream associated with a stronger breathing signal more based on the user condition determined in the first period of time.

[0031] In another example, additionally referring to Figure 3 , the breathing detection system 100 may receive an internal audio stream from an in-ear microphone (graph “C”) and an external audio stream from an external microphone (graph “D”). Figure 3 The audio stream shown in is during a second period corresponding to a type of exercise (eg, running). Figure 2 The curve A and curve B are similar. Figure 3 Graphs C and D of FIG. 1 are time-spectral analysis graphs, with time along the x-axis and spectral content or frequency on the y-axis, as shown. The "bright" portion of each column is a frequency band where a signal (e.g., a breathing signal or an intruder signal) is detected. It can be seen that these frequency bands include distortion, making it more difficult to detect a breathing signal in the internal audio stream (e.g., a strong frequency band between the end of the inspiration phase and the beginning of the next exhalation phase, such as Figure 3 ). The strong frequency band is clearer in graph D than in graph C. A reference (e.g., ground truth) graph is overlaid on each of graphs C and D as shown, indicating the user's actual breathing measured using additional equipment (e.g., a sensor band worn by the user).

[0032] Please note that Figure 3 The audio stream shown in FIG. 1 is during a second time period corresponding to a type of exercise (e.g., running). Exercise can be a new user condition indicated by input 102. For example, running can be indicated by user input (e.g., the user selects running in the application) and / or by sensor input (e.g., one or more sensors indicate faster movement and / or bouncing during the second time period).

[0033] During exercise, the internal audio stream may include greater distortion caused by an aggressor signal (such as the sound from a user running on a treadmill), thereby making the breathing signal from the internal audio stream weaker (e.g., more difficult to distinguish). For example, there may be more aggressor signals in the internal audio stream than in the external audio stream. Therefore, the breathing detection system 100 may use the internal audio stream less (e.g., give less weight and / or gate the signal) and use the external audio stream more (e.g., give more weight and / or pass the signal) to determine the breathing signal based on which the breathing rate can be calculated. For example, the breathing detection system 100 can better distinguish the breathing signal from the aggressor signal in the external audio stream. Therefore, the breathing detection system 100 makes more use of the external audio stream that is again associated with a stronger breathing signal based on the new user condition determined in the second time period. In addition, the breathing detection system 100 can reliably determine the user's breathing rate in many different situations.

[0034] In some implementations, the breathing detection system 100 can trigger a new determination of the breathing rate based on the input 102. For example, when the input 102 can indicate a new user condition, such as the user performing a new type of exercise (e.g., push-ups instead of running) or a new activity (e.g., walking, eating, or drinking instead of meditating), the breathing detection system 100 can then automatically trigger an update to the determination of the breathing rate based on the new user condition.

[0035] In some implementations, the machine learning model may include a separate machine learning model in a separate channel. For example, the machine learning model may include a first machine learning model that determines a first breathing signal based on features extracted from an internal audio stream and a second machine learning model that determines a second breathing signal based on features extracted from an external audio stream. For example, the first machine learning model may include a recurrent network and a fully connected layer in the first channel, and the second machine learning model may include another recurrent network and another fully connected layer in the second channel. The breathing detection system 100 may determine the breathing rate based on the utilization of the first breathing signal for determining the first breathing rate and the second breathing signal for determining the second breathing rate. In some cases, the breathing detection system 100 may utilize holistic learning based on the machine learning models in separate channels.

[0036] In some implementations, the breathing detection system 100 may include additional audio streams from additional microphones (e.g., additional sources in additional channels). For example, when the head-mounted system 104 includes a pair of earbuds, each earbud may generate an internal audio stream and an external audio stream, which are received by the input circuit in separate channels, followed by a machine learning model as described herein. In another example, when the head-mounted system 104 includes a companion device such as a smart phone, a smart watch, headphones, goggles, a wearable device, or other mobile device utilized by a user, the device may also generate an additional audio stream from an additional microphone (e.g., another external microphone generates another external audio stream). The additional audio stream may also be received by the input circuit in a separate channel, followed by a machine learning model as described herein.

[0037] In some specific implementations, the breathing detection system 100 may include post-processing using an internal audio stream, an external audio stream, an additional audio stream, and / or input 102. In some cases, post-processing can be used to verify the user's breathing rate. For example, the breathing detection system 100 can use additional audio streams and / or sensor inputs to verify that the determined breathing signal, the source of interference signal, and / or the breathing rate are consistent with the user's condition (e.g., sensing an elevated heart rate is consistent with running rather than meditation). In some cases, post-processing can determine a trust score associated with the breathing rate based on the user's condition. For example, post-processing can use the trust score to determine whether the breathing rate should be reported to the user (e.g., via an application running on the device) or measured again. In some cases, reporting the breathing rate to the user may require the trust score to exceed a threshold.

[0038] Figure 4 4 is an example of an enhancement system 400 for enhancing an audio stream for determining a user's breathing rate. In some implementations, the breathing detection system 100 may include the enhancement system 400. The enhancement system 400 may include an enhancement circuit 402 and / or an echo canceller 404. The enhancement circuit 402 may receive an internal audio stream and an external audio stream as sources in separate channels. In some implementations, the enhancement circuit 402 may be implemented in Figure 1 The filter circuits (e.g., the first filter circuit 106 and the second filter circuit 108) of the present invention and the source separator 110 are connected to each other. The enhancement circuit 402 may include a conditioning circuit to enhance the internal audio stream based on the external audio stream to generate an enhanced audio stream. For example, the enhancement circuit 402 may enhance the internal audio stream using a superposition based on the external audio stream. The enhanced audio stream may then be used as a replacement stream (e.g., a replacement for the internal audio stream) in the breathing detection system 100. In some specific implementations, the enhancement circuit 402 may include a conditioning circuit to enhance the external audio stream based on the internal audio stream to generate an enhanced audio stream.

[0039] The breathing rate may be determined based on the breathing signal in the enhanced audio stream. In some cases, additional adjustments may be performed by the echo canceller 404. For example, the echo canceller 404 may receive the enhanced audio stream and a reference sound. The reference sound may correspond to a sound played by a speaker of the head mounted system 104, such as music, sound from a phone call, and / or noise reduction. The echo canceller 404 may include cancellation circuitry to remove the reference sound from the enhanced audio stream (e.g., cancel the sound played by the speaker) in order to generate a denoised breathing signal without the reference sound, based on which the breathing rate may be determined.

[0040] To further describe some implementations in greater detail, reference is next made to examples of processes that may be performed by or using a system for determining a user's breathing rate based on an audio stream and a condition of the user. Figure 5 is a flow chart of an example of a process 500 for determining a respiratory rate. The process 500 may be performed using a computing device such as a Figures 1 to 4 The process 500 may be performed, for example, by executing a machine-readable program or other computer-executable instructions (such as a routine, instruction, program, or other code). The steps or operations of the process 500 or another process, method, technique, or algorithm described in conjunction with the specific implementations disclosed herein may be implemented directly in hardware, firmware, software executed by hardware, circuits, or a combination thereof.

[0041] For simplicity of explanation, process 500 is depicted and described as a series of steps or operations herein. However, steps or operations according to the present disclosure can occur in various orders and / or simultaneously. In addition, other steps or operations not presented and described herein can be used. In addition, not all illustrated steps or operations are required to implement the process according to the disclosed subject matter.

[0042] At operation 502, a system (e.g., breathing detection system 100) may receive an internal audio stream from an in-ear microphone of a head-mounted system (e.g., head-mounted system 104). For example, the head-mounted system may include an earbud that includes an in-ear microphone. The in-ear microphone may be configured to pick up sounds in an ear cavity of a user.

[0043] At operation 504, the system may receive an external audio stream from an external microphone (e.g., of a head-mounted system). For example, the head-mounted system may include earplugs that include external microphones. In another example, the head-mounted system may include headphones or goggles that include external microphones. The external microphone may be configured to pick up sounds outside the head-mounted system. In some implementations, the internal audio stream may be enhanced by the external audio stream to generate an enhanced audio stream. In some implementations, the external audio stream may be enhanced by the internal audio stream to generate an enhanced audio stream. In some implementations, an echo canceller may cancel sounds played by a speaker of the head-mounted device in the enhanced audio stream.

[0044] At operation 506, the system may determine whether input indicating a user condition is received (e.g., input 102). The input may include a user input from a user indicating a user condition, a sensor input from one or more sensors, or a combination thereof. For example, the input may come from a companion device, such as a smart phone, a smart watch, a wearable device, a head-mounted system, or another mobile device utilized by a user. The user input may include a user indicating a user condition, such as a user indicating the type of exercise being performed by the user (e.g., push-ups or running) or an indication of an activity being performed such as the user walking, meditating, eating, drinking, or resting. The sensor input may include indicating a user condition via one or more sensors, thereby indicating a user's vital signs (e.g., heart rate, temperature), a user's movement (e.g., repetitive, such as walking, running, jumping, or lifting), or a user's location (e.g., indoors or outdoors).

[0045] If no input indicating the user's condition is received ("No"), the system may return to operations 502 and 504 to continue to receive the internal audio stream and the external audio stream and monitor the input respectively. If the input is received ("Yes"), at operation 508, the system may determine the user's breathing rate based on the internal audio stream, the external audio stream, and the input indicating the user's condition. For example, the breathing rate may be determined according to the user's condition based on one or more breathing signals that measure the user's breathing from the internal audio stream and / or the external audio stream. In some specific implementations, one or more breathing signals may be weighted for the breathing rate to measure the user's breathing. In some specific implementations, the breathing detection system may call a machine learning model to determine the breathing signal from the internal audio stream and / or the external audio stream. The machine learning model may determine the breathing signal based on the utilization of the internal audio stream and the external audio stream indicated by the user's condition based on the user's condition. The machine learning model may be run by one or more processors of a system (such as a head-mounted system) including an in-ear microphone and / or an external microphone, or by a matching device that communicates with a system including an in-ear microphone and / or an external microphone. For example, the machine learning model may be run by a smart phone, smart watch, tablet, or head-mounted system, which may include earplugs, goggles, or glasses (which may also include VR, AR, or mixed reality). The breathing detection system may determine the user's breathing rate based on the breathing signal. For example, the machine learning model may determine the breathing rate based on the breathing signal. In some specific implementations, the system may store the breathing rate on a companion device that communicates with an in-ear microphone and / or an external microphone. For example, the breathing rate may be streamed to the companion device and stored in the user's health record maintained on the companion device.

[0046] In some implementations, the system may return to operation 502 to continue receiving the internal audio stream and the external audio stream, respectively, and monitoring for new inputs indicative of the user's condition. In some implementations, the system may return to operation 502 to periodically update the breathing rate (e.g., the breathing rate may be periodically updated, such as once every N minutes, where N is an integer value). Thus, process 500 may reduce the impact of background sources of disturbance (e.g., background noise, speech, music) to enhance the audio signal related to breathing and improve the estimation of related parameters such as breathing rate. Thus, the breathing detection system may reliably determine the user's breathing rate in many different situations.

[0047] In some specific implementations, a method may include: receiving an input indicating a user condition; receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone; and determining a user's breathing rate based on the internal audio stream, the external audio stream, and the input indicating the user condition. The in-ear microphone may be configured to pick up sounds in the user's ear cavity, and the external microphone may be configured to pick up sounds outside the head-mounted system. The method may include: triggering the determination of the breathing rate when the user condition indicates that the user is exercising. The breathing rate may be determined based on a breathing signal that measures the user's breathing, and the method may include: calling a machine learning model to determine a breathing signal from at least one of the internal audio stream or the external audio stream based on the user condition. In some cases, a greater weight is applied to the internal audio stream or the external audio stream based on the user condition. In some cases, a greater weight is applied to the internal audio stream when an intruder signal in the internal audio stream is below a threshold. In some cases, a greater weight is applied to the external audio stream when there are more intruder signals in the internal audio stream than in the external audio stream. In some cases, the input may include measurements from sensors indicating the user's vital signs or movement. In some cases, the input may include an indication of the user's location indoors or outdoors. In some cases, the input may include an indication of the type of exercise performed by the user. In some cases, the input may include an indication of whether the user is walking, meditating, eating, drinking, or resting. In some cases, the input may come from a wearable device of the user. In some cases, the breathing rate may be determined based on the breathing signal by distinguishing the breathing signal from an intrusive source signal caused by ambient sound outside the head-mounted system. In some cases, the breathing rate may be determined based on a machine learning model determining the breathing signal based on features extracted from the internal audio stream and the external audio stream. In some cases, the breathing rate may be determined based on a first machine learning model and a second machine learning model, the first machine learning model determining a first breathing signal based on features extracted from the internal audio stream, and the second machine learning model determining a second breathing signal based on features extracted from the external audio stream. In some cases, the head-mounted system includes a first earplug, the first earplug including an in-ear microphone and an external microphone; and a second earplug. In some cases, the head-mounted system may include earplugs, the earplugs including in-ear microphones; and headphones or eyewear, the headphones or eyewear including external microphones. The method may include: verifying the breathing rate of the user based on an additional audio stream from the additional microphone. The method may include: determining a trust score associated with the breathing rate based on the user condition. The method may include: enhancing the internal audio stream based on the external audio stream to generate an enhanced audio stream, wherein the breathing rate is determined based on a breathing signal in the enhanced audio stream.

[0048] In some specific implementations, a non-transitory computer-readable medium may store instructions that are operable to cause one or more processors to perform operations, including: receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone; calling a machine learning model to determine a breathing signal that measures the user's breathing, the machine learning model determining the breathing signal based on a user condition indicating utilization of the internal audio stream and the external audio stream; and determining the user's breathing rate based on the breathing signal. Based on the user condition, the machine learning model may give a greater weight to one of the internal audio stream or the external audio stream and a smaller weight to the other of the internal audio stream or the external audio stream. The operations may include: selecting between the internal audio stream or the external audio stream based on the user condition. The operations may include: gating one of the internal audio stream or the external audio stream based on the user condition so that the machine learning model determines the breathing signal based on the other of the internal audio stream or the external audio stream. The operations may include: eliminating the sound played by the speaker of the head-mounted system from the internal audio stream or the external audio stream. The operations may include receiving input indicative of the user condition from a headset or goggles. The machine learning model may include a recurrent network and a fully connected layer. The operations may include validating the breathing signal based on an additional audio stream from an additional microphone. The operations may include validating an intrusion source signal caused by ambient sound external to the head mounted system based on an additional audio stream from an additional microphone. The operations may include determining a trust score associated with the breathing rate based on the user condition.

[0049] In some specific implementations, a non-transitory computer-readable medium may store instructions that are operable to cause one or more processors to perform operations, including: receiving input indicating a user condition; receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone; and determining the user's breathing rate based on the internal audio stream, the external audio stream, and the input indicating the user's condition. The operations may include: storing the breathing rate on a companion device that communicates with at least one of the in-ear microphone or the external microphone. The breathing rate may be updated periodically. The operations may include: calling a machine learning model to determine the breathing rate, wherein the machine learning model is run by a system that includes at least one of the in-ear microphone or the external microphone. The operations may include: calling a machine learning model to determine the breathing rate, wherein the machine learning model is run by a companion device that communicates with at least one of the in-ear microphone or the external microphone.

[0050] As used herein, the term "circuit" refers to an arrangement of electronic components (e.g., transistors, resistors, capacitors, and / or inductors) structured to implement one or more functions. For example, a circuit may include one or more transistors interconnected to form a logic gate that together implements a logic function.

[0051] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to users.

[0052] As described above, one aspect of the present technology is to collect and use data available from specific and legitimate sources for determining a user's breathing rate based on an audio stream and the user's condition. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0053] The present disclosure recognizes that use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to determine the user's breathing rate. For example, health and fitness data can be used according to user preferences to provide insights into their overall health, or can be used as positive feedback to individuals using technology to pursue health goals.

[0054] The present disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be highlighted and easily accessible to users, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legitimate use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and privacy practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0055] Notwithstanding the foregoing, the present disclosure also contemplates implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, such as in the case of determining a user's breathing rate based on an audio stream and user conditions, the present technology may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data at any time during or after registration for the service.

[0056] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be promoted when appropriate by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differential privacy.

[0057] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without requiring access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user based on aggregated non-personal information data or an absolute minimum amount of personal information, such as content that is disposed only on the user's device or other non-personal information that may be used for content delivery services.

[0058] When utilizing various aspects of the embodiments, it will be apparent to those skilled in the art that combinations or variations of the above embodiments are possible for determining a user's breathing rate based on an audio stream and a user's condition. Although the embodiments are described in language specific to structural features and / or methodological acts, it should be understood that the appended claims are not necessarily limited to the specific features or acts described. Instead, the specific features and acts disclosed should be understood as embodiments of the claims for purposes of illustration.

Claims

1. A method comprising: receiving input indicative of a user condition; receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone of the head-mounted system; as well as A breathing rate of a user is determined based on the internal audio stream, the external audio stream, and the input indicative of the user's condition.

2. The method according to claim 1, further comprising: In case the user condition indicates that the user is exercising, the determination of the breathing rate is triggered.

3. The method of claim 1 , wherein the breathing rate is determined based on a breathing signal measuring the user's breathing, and the method further comprises: A machine learning model is invoked to determine a breathing signal from at least one of the internal audio stream or the external audio stream based on the user condition. The method according to claim 1 , wherein a greater weight is applied to the internal audio stream or the external audio stream based on the user status. 5 . The method according to claim 1 , wherein if the aggressor signal in the internal audio stream is below a threshold, a greater weight is applied to the internal audio stream. 6 . The method of claim 1 , wherein in a case where there are more aggressor signals in the internal audio stream than in the external audio stream, a greater weight is applied to the external audio stream.

7. The method of claim 1, wherein the breathing rate is determined based on the breathing signal by distinguishing the breathing signal from an interfering source signal caused by ambient sounds external to the head mounted system.

8. The method of claim 1 , wherein the breathing rate is determined based on a first machine learning model and a second machine learning model, wherein the first machine learning model determines a first breathing signal based on features extracted from the internal audio stream, and the second machine learning model determines a second breathing signal based on features extracted from the external audio stream.

9. The method according to claim 1, further comprising: The breathing rate of the user is verified based on an additional audio stream from an additional microphone.

10. The method according to claim 1, further comprising: A confidence score associated with the breathing rate is determined based on the user condition.

11. The method according to claim 1, further comprising: The internal audio stream is enhanced based on the external audio stream to generate an enhanced audio stream, wherein the breathing rate is determined according to a breathing signal in the enhanced audio stream.

12. A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising: receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone of the head-mounted system; Invoking a machine learning model to determine a breathing signal that measures the user's breathing, the machine learning model determining the breathing signal based on a user condition indicating utilization of the internal audio stream and the external audio stream; as well as A breathing rate of the user is determined based on the breathing signal.

13. A non-transitory computer-readable medium storing instructions according to claim 12, wherein based on the user condition, the machine learning model gives greater weight to one of the internal audio stream or the external audio stream and gives less weight to the other of the internal audio stream or the external audio stream.

14. The non-transitory computer readable medium storing instructions of claim 12, the operations further comprising: A selection is made between the internal audio stream or the external audio stream based on the user condition.

15. The non-transitory computer readable medium storing instructions of claim 12, the operations further comprising: An intrusive source signal caused by ambient sound outside the head mounted system is verified based on an additional audio stream from an additional microphone.

16. The non-transitory computer readable medium storing instructions of claim 12, the operations further comprising: A confidence score associated with the breathing rate is determined based on the user condition.

17. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising: receiving input indicative of a user condition; receiving an internal audio stream from an in-ear microphone of a head-mounted system and an external audio stream from an external microphone of the head-mounted system; as well as A breathing rate of a user is determined based on the internal audio stream, the external audio stream, and the input indicative of the user's condition.

18. The non-transitory computer readable medium storing instructions of claim 17, wherein the respiration rate is updated periodically.

19. The non-transitory computer readable medium storing instructions of claim 17, the operations further comprising: A machine learning model is invoked to determine the breathing rate, wherein the machine learning model is run by a system including at least one of the in-ear microphone or the external microphone.

20. The non-transitory computer readable medium storing instructions of claim 17, the operations further comprising: A machine learning model is invoked to determine the breathing rate, wherein the machine learning model is run by a companion device in communication with at least one of the in-ear microphone or the external microphone.

Images