Respiratory rate sensing

Abstract

A wearable respiratory rate sensing device The device comprising: a processor, connectable to an audio sensor configured to: receive captured in-ear audio data from the audio sensor; detect, from the captured in-ear audio data, a scenario of the wearer; select, based on the detected scenario, an in- ear audio data processing pipeline from a plurality of in-ear audio data processing pipelines; and derive, by applying the selected in-ear audio data processing pipeline to the captured in-ear audio data, a respiratory rate of the wearer.

Description

[0001] RESPIRA TORY RA TE SENSING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a wearable respiratory rate sensing device and a method for deriving a respiratory rate.

[0004] BACKGROUND

[0005] There has been a marked increase in both the number and complexity of wearable devices which, whilst intended for end-market consumers, are capable of measuring health parameters or physiological signals. Of these parameters, respiratory rate is a key marker of physical and mental health, and continuous monitoring of this has the potential to unlock a better understanding of health and well-being.

[0006] Respiration is an instinctive physiological behaviour of human beings to move oxygen through the body during inspiration and clean out waste gases (e.g., carbon dioxide) during exhalation. The speed of respiration, namely the respiratory rate (RR), is a fundamental physiological signal that relays pivotal information on the human body.

[0007] To measure RR, typically one of three methods have been adopted: (i) measurement of chest / abdomen movement; (ii) measurement of airflow during breathing; and (iii) physiological coupling. In (i), inward and outward movement of the chest wall and abdomen are measured, and the RR can be measured by counting the number of such periodic movements. In (ii), when RR can be estimated through measuring the airflow during breathing, RR can be estimated either by a carbon dioxide sensor connected to a nasal cannula or an audio sensor used to capture the sounds generated by airflow. In (iii), it has been observed that RR is closely coupled with other human physiological signals or physical behaviours, allowing indirect RR measurement. These couplings include respiratory sinus arrhythmia (RSA) [1] which implies RR is associated with heart rate variability (HRV) and locomotor respiratory coupling (LRC) [2] which indicates that RR is correlated with footsteps when walking or running.

[0008] However, performing robust and non-invasive respiratory rate monitoring in realistic daily life scenarios is still an open research problem.

[0009] The present disclosure was arrived at in light of the above considerations. SUMMARY

[0010] Accordingly, in a first aspect, embodiments of the invention provide a wearable respiratory rate sensing device, comprising: a processor, connectable to an audio sensor and configured to: receive captured in-ear audio data from the audio sensor; detect, from the captured in-ear audio data, a scenario of the wearer; select, based on the detected scenario, an in-ear audio data processing pipeline from a plurality of in-ear audio data processing pipelines; and derive, by applying the selected in-ear audio data processing pipeline to the captured in-ear audio data, a respiratory rate of the wearer.

[0011] It has been found that by first detecting the scenario of the wearer, and then selecting an in-ear audio data processing pipeline based on this detected scenario, the accuracy with which respiratory rate can be derived is increased. For example, by utilising the occlusion effect, clear features for detection can be found in the in-ear audio. A further benefit of the device is that the derivation of the respiratory rate may be ‘training-free’.

[0012] By scenario, it may be meant a context of the wearer. The processor may utilise a scenario detection module to detect, from the captured in-ear audio data, the scenario of the wearer.

[0013] The processor may be configured to only use the captured in-ear audio data to detect the scenario and derive the respiratory rate of the wearer. The device may not include an IMU, or may include an IMU but not utilise any IMU data in the detection of the scenario and the derivation of the respiratory rate.

[0014] The wearable respiratory rate sensing device may be a smartphone or other general computing device, and may be connected to the audio sensor via a wired or wireless connection. For example, the wearable respiratory rate sensing device may connect via a Bluetooth (RTM) connection to an audio sensor, for example an earbud.

[0015] In other examples, the wearable respiratory rate sensing device may include the audio sensor. For example, the wearable respiratory rate sensing device may be an earbud or pair of earbuds including an in-ear microphone connected to a processor also within the earbud.

[0016] The processor may be configured to detect, from the captured in-ear audio data, whether the wearer is in a moving scenario or a stationary (or sedentary) scenario. That is, the processor may be configured to detect whether the wearer is moving or is stationary. The processor may be configured to select a first in-ear audio data processing pipeline, when the wearer is detected as being in a stationary scenario, and a second in-ear audio data processing pipeline when the wearer is detected as being in a moving scenario. The processor may be configured to select a first set of parameters for the second in-ear audio data processing pipeline when the wearer is detected as being in a walking scenario, and a second set of parameters for the second in-ear audio data processing pipeline when the wearer is detected as being in a running scenario. The first set of parameters being different to the second set of parameters.

[0017] The first in-ear audio data processing pipeline may include a step of estimating a heart rate variability, HRV, from the captured in-ear audio data, and subsequently extracting the respiratory rate from the estimated heart rate variability. The heart rate variability may be adaptively estimated. A peak detection threshold, used in estimating the HRV, may be adaptively changed during estimation of the HRV. For example, a moving mean may be computed for each point of the captured in-ear audio data and used to adjust the peak detection threshold. The first in-ear audio data processing pipeline may include an initial step of filtering the captured in-ear audio data for motion artifacts.

[0018] There may be only one audio sensor, in that the device may include only one earbud. When the device comprises only one audio sensor for capturing in-ear audio data (e.g., one ear bud) the respiratory rate may be estimated using the heart rate variability derived from that audio sensor.

[0019] In other examples, there may be at least two audio sensors, for example two earbuds. Where there are at least two audio sensors these may provide two separate audio channels (for example a first and second, or left and right). The captured in-ear audio data may include a first and second audio channel, and the heart rate variability may be estimated using both channels separately to obtain a first estimated heart rate variability with a first standard deviation and a second estimated heart rate variability with a second standard deviation, and the estimated heart rate variability with the lower standard deviation is selected for use in deriving the respiratory rate. Alternatively, a respiratory rate may be estimated using both estimated heart rate variability, and a mean value of the two estimated respiratory rates provided as the final estimated respiratory rate.

[0020] Extracting the respiratory rate from the estimated heart rate variability may include selecting a candidate respiratory rate from a plurality of candidate respiratory rates, the selected candidate respiratory rate having a smallest difference between itself and an estimated initial respiratory rate. The estimated initial respiratory rate may be determined by applying a filter to the estimated heart rate variability, the passband of the filter being based on the respective candidate respiratory rate. For example, the respiratory rate may be estimated from the estimated heart rate variability (HRV) using the method of the fourth aspect discussed below.

[0021] The second in-ear audio data processing pipeline may include estimating the respiratory rate on the basis of an estimated stride frequency and a locomotion respiratory coupling ratio (LRC ratio), for example an LRC ratio for humans (e.g., 2:1 , 2.5:1 , 3:1 , or 4:1). For example, the second in-ear audio data processing pipeline may include: (i) estimating a stride frequency of the wearer using the captured in-ear audio data and (ii) generating a likelihood curve for a plurality of frames of the captured in-ear audio data, indicating whether a given frame is likely to contain breathing information, steps (i) and (ii) being performed in either order in series or in parallel, and the estimated stride frequency and likelihood curve being used to derive the respiratory rate. The likelihood curve may be decomposed into a plurality of components, and one or more of the plurality of components may then be selected, and used, together with the estimated stride frequency, for deriving the respiratory rate. For example, the second in-ear audio data processing pipeline may include applying a singularspectrum analysis algorithm to identify respiration-related components for use in estimating the respiratory rate. The non-respiration-related components, for example interference related components, may be excluded. The one or more components may be selected by comparing the number of peaks to a range of acceptable values, and selecting the component if the number of peaks falls within the acceptable range of values. The range of acceptable values may be defined by a lowest number of breathing events, RRmtn, and a highest number of breathing events, RRmax. RRmtnmay be defined as RRmin= (SFest■ y) / LRCJ^, and RRmaxmay be defined as RRmax= (SFest■ ) / LRC ink’ where SFest, fs, and T are the estimated step frequency, audio sampling rate, and sample numbers of the estimation window, respectively.

[0022] The respiratory rate may be estimated in the second in-ear audio data processing pipeline using in- ear audio data from a single audio sensor (e.g., when the device includes only one earbud). In other examples, the second in-ear audio data processing pipeline may receive in-ear audio data from more than one audio sensor (e.g., a left channel and a right channel).

[0023] A peak detection threshold, used in estimating the stride frequency, may be adaptively changed during estimation of the stride frequency as described above for HRV estimation. For example, a moving mean may be computed for each point of the captured in-ear audio data and used to adjust the peak detection threshold.

[0024] The processor may be configured to detect the scenario of the wearer by providing the captured in- ear audio data to a scenario detection module which may be a machine learning classifier, the output of which is a detected scenario. For example, the processor may use a trained support vector machine where at least some of the captured in-ear audio data is provided to the SVM, the output being a detected scenario. Beneficially, the SVM may be more computationally lightweight than other machine learning classifiers, thereby reducing the overall computational load of the respiratory rate sensing and improving battery life of the wearable device. In other examples, a different machine learning classifier may be configured to detect the scenario of the wearer such as a GMM (Gaussian Mixture Model) or an ANN (Artificial Neural Network).

[0025] In some examples, the captured in-ear audio data may represent an estimation window of a predefined duration (e.g., 60 seconds). The estimation window may be divided into a plurality of segments (e.g., each of 5 seconds duration), with the machine learning classifier being applied to each of the plurality of segments and resulting in a respective predicted scenario for each segment. The processor may use each respective predicted scenario in a voting process to arrive at an overall predicted scenario for the estimation window. The processor may determine whether the number of respective predicted scenarios corresponding to a single scenario exceeds a predetermined threshold. For example, the processor may determine whether at least 8 segments have respective predicted scenarios corresponding to a stationary scenario. In some examples there may be a total of 12 segments. Generally, the predetermined threshold may be 2 / 3 of a total number of segments (e.g., 8 where there are 12 total segments).

[0026] In a second aspect, embodiments of the invention provide a method of deriving a respiratory rate from in-ear audio data captured from a wearer of a device including an audio sensor, the method comprising steps of: detecting, from the captured in-ear audio data, a scenario of the wearer; selecting, based on the detected scenario, an in-ear audio data processing pipeline from a plurality of in-ear audio data processing pipelines; and deriving, by applying the selected in-ear audio data processing pipeline to the captured in-ear audio data, a respiratory rate of the wearer.

[0027] The method may include any one, or any combination insofar as they are compatible, of the optional features set out with reference to the first aspect. That is, where the processor is disclosed as being configured to perform certain functions those functions may be implemented as steps in the method of the second aspect. The device of the second aspect including an audio sensor may be the wearable respiratory rate sensing device of the first aspect including any one or any combination of the optional features set out with reference thereto.

[0028] The method may be a computer-implemented method. The method may be implemented on one or more processors, for example, it may be implemented on a wearable respiratory rate sensing device.

[0029] In a third aspect, embodiments of the invention provide a computer, containing a processor and memory, the memory containing machine executable instructions which, when implemented on the processor, cause the processor to perform the method of the second aspect including any one or any combination insofar as they are compatible of the optional features set out above.

[0030] The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.

[0031] In an fourth aspect provided by the present invention there is provided a computer-implemented method of estimating respiratory rate from a heart rate variability (HRV) signal, the computer- implemented method comprising:

[0032] (a) generating a list of candidate respiratory rates,

[0033] (b) for each candidate respiratory rate, applying a candidate specific filter to the HRV signal to obtain a respective filtered HRV signal, the passband of each filter being determined based on the respective candidate respiratory rate,

[0034] (c) for each filtered HRV signal, determining an interim respiratory rate from the filtered HRV signal, and

[0035] (d) determining an estimate for the respiratory rate based on the interim respiratory rates.

[0036] Advantageously, it has been found that by applying candidate specific filters across a list of candidate respiratory rates, a more accurate estimate for respiratory rate can be determined than conventional methods which use static filters. In other words, by dynamically adjusting the filter configuration, a more effective filter configuration can be determined for estimating respiratory rate in each case. Therefore, the method of estimating respiratory rate can be adapted to different subjects and scenarios which may exhibit different relationships between HRV and respiratory rate.

[0037] The method of this additional aspect may be utilised in the device of the first aspect of the method of the second aspect as a portion of one of the in-ear audio data processing pipelines.

[0038] The HRV signal may be defined as a measure of the variation in time intervals between heartbeats. For example, the HRV may also be referred to as a variation in the beat-to-beat interval, "cycle length variability", "R-R variability" (where R is a point corresponding to the peak of the QRS complex of an ECG wave; and RR is the interval between successive Rs), and "heart period variability".

[0039] As discussed above in relation to the first aspect, the HRV signal may be derived from audio data. The audio data may be in-ear audio data. In other examples, the HRV signal may be derived from an ECG signal, blood pressure signals, ballistocardiograms, or a pulse wave signal derived from a photoplethysmograph (PPG).

[0040] Determining the estimate for the respiratory rate based on the interim respiratory rates may include calculating a difference between each interim respiratory rate and the respective candidate respiratory rate. Accordingly, the calculated differences, which are differences in rate, may be given, for example, in BPM or Hz.

[0041] Next, one or more of the smallest local minima (or troughs) in the calculated differences between (interim and candidate respiratory rate) may be identified. The estimate for the respiratory rate may then be determined based on the candidate respiratory rate corresponding to the smallest local minima in calculated differences. For example, the estimate for the respiratory rate may be the candidate respiratory rate corresponding to the smallest calculated difference. In other examples, as explained in detail below, a plurality of the smallest differences may be determined and assessed to determine the estimated respiratory rate.

[0042] The computer-implemented method may further comprise determining an initial respiratory rate from the HRV signal. In this example, the list of candidate respiratory rates may be determined based on the initial respiratory rate. In other examples, the list of candidate respiratory rates may be a predetermined list of candidate respiratory rates.

[0043] The initial respiratory rate may be determined by performing frequency or time domain analysis on the HRV signal. For example, the initial respiratory rate may be based on a peak (or dominant) frequency component of the HRV signal. When the initial respiratory rate is determined using frequency analysis, the initial respiratory rate may be determined by: performing a Fourier transform (e.g., a Fast Fourier Transform, FFT) of the HRV signal, and identifying a highest amplitude peak in the transformed signal, the initial respiratory rate being given by the frequency corresponding to the highest amplitude peak. In other examples, when the initial respiratory rate is determined using time domain analysis, the initial respiratory rate may be determined by measuring a zero crossing rate of the HRV signal. This method may optionally comprise normalising and / or smoothing the HRV signal. The list of candidate respiratory rates may span a range from a minimum value ( Rmin) to a maximum value (RRmax), at least one of which is a dependent on the initial respiratory rate. The list of candidate respiratory rates may be a range of sample respiratory rates separated by a predetermined increment from the minimum value to the maximum value. When the respiratory rate is in BPM (breaths per minute) the predetermined increment may be e.g., 0.25 BPM, or 0.5BPM, or 1 BPM.

[0044] The minimum respiratory rate of the candidate respiratory rates may be calculated according to: RRmin = max Nmin, RRc - X) where RRc \s the initial estimate for respiratory rate, Nmin is a predetermined minimum value for the candidate respiratory rates and X is a predetermined constant. For example, Nmin may be between 3 to 12 BPM, more preferably 5 to 9 BPM, more preferably 7.5 BPM. X may be between 10 to 30 BPM, more preferably 14 to 20 BPM, more preferably 17 BPM.

[0045] The maximum respiratory rate of the candidate respiratory rates may be calculated according to: RRmax = min(Nmax, RRc + Y) where RRc \s the initial estimate for respiratory rate, Nmax is a predetermined maximum value for the candidate respiratory rates and Y is a predetermined constant. Nmax may be between 35 to 50 BPM, more preferably 40 to 45 BPM, more preferably 42.5 BPM. Y may be between 10 to 30 BPM, more preferably 14 to 20 BPM, more preferably 17 BPM. In some examples, Y and X may be the same.

[0046] The computer-implemented method may include a further step of pre-filtering the HRV signal before determining the initial respiratory rate. The pre-filtering of the signal may be performed using a bandpass filter having a passband configured according to predetermined limits (i.e., cut-off frequencies). For example, the predetermined high-pass cut-off frequency (i.e., a lower predetermined pre-filter limit) ( / ) may be between 0.1 to 0.2Hz, more preferably between 0.12 to 0.18Hz, more preferably 0.15Hz. The predetermined low-pass frequency (i.e., a higher predetermined pre-filter limit) (ft) may be between 0.3 to 0.4Hz, more preferably between 0.32 to 0.38 Hz, more preferably 0.35Hz.

[0047] By pre-filtering the HRV signal, the high frequency (HF) range of the HRV signal may be extracted. For example, the HR range may be from 0.11 to 0.4Hz, more preferably 0.15 to 0.35Hz. Beneficially, this HF range encapsulates rhythms governed by parasympathetic activity, which is closely related to respiration. Accordingly, this range corresponds to the common lowest and highest human breathing rates.

[0048] Each candidate specific filter may be a bandpass filter. The cut-off frequencies of the passband of each candidate specific filter may be calculated based on the respective candidate respiratory rate. For example, one or both of the upper and lower limits of each candidate specific filter may be based on the respective candidate respiratory rate.

[0049] Each candidate specific bandpass filter may be given by A lower cut-off frequency (in Hz) of the bandpass filter corresponding to a nth candidate respiratory rate may be given by: and a higher cut-off frequency (in Hz) of the bandpass filter may be given by: where RR^ is the nth candidate respiratory rate (in BPM).

[0050] Each interim respiratory rate may be determined by performing frequency analysis to determine a dominant (peak) frequency component in each filtered HRV signal. Alternatively or additionally, determining each interim respiratory rate may be performed using time domain analysis.

[0051] For example, determining each interim respiratory rate in step (c) may include performing a Fourier transform of the respective filtered HRV signal, and identifying a highest amplitude peak in the transformed signal. The interim respiratory rate may be given by the frequency of the highest amplitude peak.

[0052] In other examples, determining each interim respiratory rate in step (c) may include determining a zero crossing rate of the filtered HRV signal, wherein the interim respiratory rate is based on the zerocrossing rate. Determining the zero-crossing rate may further include first normalising and / or smoothing the filtered HRV signal.

[0053] In some examples, each interim respiratory rate may be determined in step (c) using a first estimation method to provide a plurality of first interim respiratory rates. The method may then further include estimating a second interim respiratory rate from each filtered HRV signal using a second estimation method. In this way a plurality of second interim respiratory rates may be obtained. The estimate for the respiratory rate may then be based on the first interim rates and the second interim respiratory rates.

[0054] The first interim respiratory rates may be determined using frequency domain analysis and the second interim respiratory rates may be determined using time domain analysis.

[0055] For example, each interim respiratory rate determined in step (c) may be a first interim respiratory rate determined from the filtered HRV signal in the frequency domain. The computer-implemented method may then further include:

[0056] (i) for each filtered HRV signal, determining a second interim respiratory rate from the filtered HRV signal in the time domain;

[0057] (ii) identifying a plurality of smallest calculated differences, each smallest calculated difference being a local minima of the calculated differences between the first interim respiratory rates and the candidate respiratory rates,

[0058] (iii) for at least each candidate respiratory rate corresponding to the plurality of smallest calculated differences, determining a second difference between the candidate respiratory rate and the respective second interim respiratory rate, and

[0059] (iv) providing a candidate list of summed differences by: for each calculated second difference, summing the second difference and the respective calculated difference between the first interim respiratory rates and the candidate respiratory rates, wherein determining the estimate for the respiratory rate comprises: identifying the smallest summed difference in the candidate list, and extracting the candidate respiratory rate corresponding to the smallest summed difference.

[0060] By determining the interim respiratory rates using two different methods (i.e. , in the frequency domain and in the time domain) a more accurate estimation for respiratory rate may be determined overall.

[0061] In some examples, the method may include determining a difference (i.e., a second difference) between each candidate respiratory rate and each respective second interim respiratory rate, thereby obtaining a second-differences-list. The second differences corresponding to the plurality of smallest calculated differences may then be selected for summing with the smallest calculated differences. In other words, the difference between each second interim respiratory rate and the respective candidate respiratory rate may be determined for every candidate respiratory rate and then the differences corresponding to the smallest first differences may then be determined (and summed with the smallest first differences).

[0062] In other examples, the method may include determining a second difference between each candidate respiratory rate and each respective second interim respiratory rate for only the candidate respiratory rates corresponding to the smallest first differences (each smallest calculated corresponding to a local minima of the calculated differences).

[0063] For example, between 2 and 6, more preferably between 3 and 5, more preferably 3 local minima may be identified as the smallest calculated differences. Next, the corresponding 2 to 6, or 3 to 5, or 3 differences in the second difference-list may then be identified and respectively summed with the corresponding first calculated difference. Finally, the candidate respiratory rate corresponding to the smallest of the summed differences may then be determined as a final estimate for the respiratory rate.

[0064] More specifically, if the calculated differences comprise more than a desired number of local minima (e.g., three) then the smallest local minima may be selected as the smallest calculated differences as described in the previous paragraph. However, if the calculated differences includes fewer than the desired number of local minima, but more than zero, then all of the local minima may be selected as the smallest calculated differences. If no local minima are detected then the candidate respiratory rate corresponding to the smallest calculated different (e.g., in the time domain) may be selected as the final respiratory rate.

[0065] For example, the local minima may be determined by calculating the differentials of the calculated differences and identifying the candidate respiratory rates corresponding to a differential value of zero, thereby indicating a peak or a trough in the calculated differences. The local minima (i.e., the troughs) may be identified by evaluating the difference values around each peak or trough. Next, the smallest of the local minima (i.e., the troughs) may be identified as the plurality of smallest differences for estimating respiratory rate.

[0066] In another example, the local minima may be determined by multiplying the calculated differences by -1 and performing a peak detection on the result. The largest detected peaks may then be identified as the smallest calculated differences, wherein the locations of the peaks correspond to the locations of the local minima.

[0067] In an additional aspect of the present invention there is provided a system comprising: a heartrate sensor and a processor, the processor being configured to receive heartbeat information from the heartrate sensor, determine an HRV signal from the heartbeat information, and determine an estimate for respiratory rate from the HRV signal using the method any previous aspect.

[0068] For example, the heartrate sensor and may include an electrical sensor, an optical sensor, or an audio sensor (e.g., a microphone). For example, the sensor may be an ECG (Electrocardiography) device, a PPG monitoring device, or a microphone (e.g., an in-ear microphone).

[0069] Additional aspects of the invention may relate to systems configured to execute the computer- implemented method of any previous aspects of the invention. Specifically, the system may comprise a processor which is configured to execute the respective computer-implemented methods of the previous aspects of the invention. Additional aspects of the invention provide a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the methods of any previous aspects of the present invention; a computer-readable storage medium, storing a computer program comprising code which, when run on a computer, causes the computer to perform the methods of any of the previous aspects of the invention; and a computer system programmed to perform the methods of any previous aspect of the invention.

[0070] The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.

[0071] BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic of a device;

[0073] Figure 2 is a flow chart of a method;

[0074] Figure 3 is a flow chart of a first in-ear audio data processing pipeline;

[0075] Figure 4 is a flow chart of a second in-ear audio data processing pipeline;

[0076] Figure 5 show a series of plots comparing signals captured from an IMU, an out-ear microphone, and an in-ear microphone in various scenarios;

[0077] Figure 6(a) and Figure 6(b) show captured signals using an in-ear microphone and out-ear microphone respectively;

[0078] Figure 7 is a plot showing the distribution of RRCT;

[0079] Figure 8 is a plot showing the RR estimation performance using a fixed frequency range and RRCT- adapted frequency range, respectively;

[0080] Figures 9(a) and 9(b) are plots showing an adaptive heartbeat peak detection process;

[0081] Figures 10(a) - (c) are plots showing a dynamic breathing signal extraction process, Figure 10(a) representing an example HRV signal, an example RR detection using zero-crossings, and an example RR detection using a Fourier Transform, and Figures 10(b) and (c) showing an example of a best candidate RR searching process using peak detection on frequency-domain data and timedomain data respectively;

[0082] Figures 11 (a) and (b) are plots showing a motion artifact filtering process;

[0083] Figures 12(a) - (d) are plots showing the ground truth locomotor respiratory coupling for two wearers, whilst walking and running respectively;

[0084] Figures 13(a) and (b) are plots showing an adaptive step peak detection process; Figures 14(a) - (c) are plots showing a likelihood curve decomposition process, where Figure 14(a) is a likelihood curve, Figure 14(b) shows a component of this curve related to steps, and Figure 14(c) shows the estimated breathing signals using peak detection for estimating RR;

[0085] Figure 15 is a plot showing the influence of minimum window length and threshold on mean absolute error (MAE);

[0086] Figure 16 is a plot showing the performance of the device and method with and without motion artifact filtering when the wearer is stationary;

[0087] Figure 17 is a plot showing the impact of utilising different audio channels while stationary;

[0088] Figure 18 is a plot showing the impact of utilising different audio channels whilst moving;

[0089] Figure 19 is a plot showing the performance of the device and method for each of 18 participants;

[0090] Figure 20 is a plot showing the performance of the device and method for a range of activities;

[0091] Figure 21 is a plot showing the performance of the device and method as compared to a ground-truth respirator rate;

[0092] Figure 22 is a flow chart of a method of estimating respiratory rate from HRV signals;

[0093] Figure 23 is a flow chart of a second method of estimating respiratory rate from HRV signals;

[0094] Figure 24(a) is a bar plot of MAE and MAPE, Figure 24(b) is a Bland-Altman plot showing performance;

[0095] Figure 25(a) and (b) shows RR estimation errors for different activities and different users respectively;

[0096] Figures 26(a) - (c) provide comparisons of accuracy for (a) IMU-based, and (b-c) audio-based approaches;

[0097] Figures 27(a) - (d) are plots showing the errors of (a) other rhythmic activities, (b) outdoor activities, and (c-d) noise levels;

[0098] Figures 28(a) - (d) are plots showing estimation errors for different (a-b) audio channels and (c-d) moving speeds;

[0099] Figures 29(a) and (b) show errors with music: (a) across genres and (b) at different volume levels;

[0100] Figures 30(a) and (b) show errors in longitudinal in-the-wild tracking for (a) user A and (b) User B;

[0101] Figures 31 (a) - (c) are plots showing errors with (a) application of high-pass filters on (b) sedentary and (c) active scenarios;

[0102] Figures 32(a) - (c) are plots showing performance of the system components: (a) pipeline selector;

[0103] (b) HRV and (c) stride frequency; and Figures 33(a) - (d) show the impact of (a) interference filtering, (b) adaptive BPF, (c) difference list, and (d) FFT features.

[0104] DETAILED DESCRIPTION

[0105] Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art.

[0106] Device and method

[0107] Figure 1 is a schematic of a wearable respiratory rate sensing device 100. The device, in this example, an ear bud of the type known per se in the art, includes a processor 102, audio sensor 104, storage 106, and output module 108. The audio sensor is an in-ear microphone, in that it is a microphone positioned in the ear bud 100 such that when the ear bud is worn the microphone is located within or adjacent to the external auditory canal. The device may also, but does not need to, include an out-ear microphone (not shown). That is, a microphone positioned in the earbud 100 such that when the ear bud is worn the microphone is outside of the external auditory canal, e.g. adjacent to the auricle. In some examples, the device is a pair of earbuds each with a respective microphone which each capture a separate channel of audio data (left and right channel). In other examples the microphone may be a stereo microphone and capture two-channel audio data (left and right channel).

[0108] The processor is configured to receive in-ear audio data captured by the audio sensor 104, and to access data stored within storage 106. The storage 106 can include one or more of volatile memory (e.g., RAM) and non-volatile memory (e.g., solid state storage). In the manner known perse in the art, the non-volatile memory, in this example, contains software in the form of machine executable instructions. Once loaded onto the processor, these allow the processor to operate according to the software. The software configures the processor to perform the method set out in Figure 2 and discussed below.

[0109] The output module 108 can include, for example, a speaker, a network connection, and / or a display. The processor is configured to provide the estimated respiratory rate (when it is derived) as an output via the output module. In the example where the output module is a speaker, the processor may be configured to utilise a text-to-speech module to output through the speaker the respiratory rate for the user. In the example where the output module is a network connection, for example a Bluetooth (RTM) or other wireless connection, the processor may provide the estimated respiratory rate to a remote device (e.g., a smartphone, tablet, or computer). Where the output module is a display, the processor may be configured to cause the display to visually display the estimated respiratory rate. Figure 2 is a flow chart of a method implemented using the wearable respiratory rate sensing device in Figure 1 . In a first step, S202, in-ear audio data is captured by the audio sensor. This in-ear audio data is provided to the processor, which detects in step S204, using the captured in-ear audio data, a scenario of the wearer. In this example, the processor is able to detect three scenarios: that the wearer is stationary; that the wearer is walking, and that the wearer is running. The latter two may be considered as examples of a fourth scenario: that the wearer is moving. In some examples, the processor may first detect that the wearer is moving (i.e., not stationary) and subsequently detect if the wearer is walking or running.

[0110] A support vector machine (SVM) was used to train a scenario detector module, which is used by the processor to detect the scenario. The two-channel in-ear audio data is split into 5-second segments, and the MFCC (M el-frequency cepstral coefficients) features extracted from each segment as the SVM input. The SVM uses a two-stage classifier, where first it is determined whether (for that segment) the wearer is stationary or moving. Where the SVM determines the wearer is moving, it further determines whether the wearer is walking or running. In this example, 60 seconds of in-ear audio is utilised for a single respiratory rate estimation. This leads to 12 scenario detection results for each estimation window, and these scenario detection results contribute to the overall scenario detection result through voting. Specifically, it was empirically determined that at least eight segments having a same detected scenario is preferred when using the overall 60 seconds of audio to estimate the respiratory rate.

[0111] The method then moves to step S206, where if it was determined that the wearer is not moving (No), the method moves to step S208 and a first in-ear audio data processing pipeline is selected and applied to the captured in-ear audio data. If it was determined that the wearer is moving (Yes), the method instead moves to step S210 and a second in-ear audio data processing pipeline is selected and applied to the captured in-ear audio data. The result of either of these steps is an estimated respiratory rate, which is provided as an output in step S212.

[0112] Figure 3 is a flow chart of a first in-ear audio data processing pipeline S208, which includes the substeps S302 - S306. In step S302, a motion artifact (MA) filtering process is performed. In-ear audio can be strongly interfered with by motion MAs, for example head motions (drinking, speaking) and body motions. As shown in Figure 11 the in-ear audio data was found to exhibit a consistent waveform with clear heart sounds in the absence of MAs, leading to stable statics over time. Conversely, under the presence of MAs, e.g., one-time head motion, the statistical properties demonstrate significant variation. Hence, a threshold-based approach was adopted to examine the statistics for each audio segment and detect the presence of MAs. For each processing window (e.g., each 60 seconds of in-ear audio data), the audio signal is segmented into 3 second segments. For each audio segment, the standard deviation is computed. If the standard deviation is larger than a threshold then the segment is marked as containing an MA. After the entire processing window has been processed, and so all MAs marked, the unmarked pieces with a minimal length are kept for use in estimating the respiratory rate. The trade-off between RR estimation accuracy and retained windows by setting different values of the statistic threshold and minimum window length are discussed below in relation to Figure 15. Empirically, a value of 0.08 for the standard deviation threshold and 40s for the minimum window length was found to provide the most accurate estimation of respiratory rate.

[0113] In an alternative embodiment, an adaptive filter is used to denoise the motion artifact (rather than remove it). The motion artifact is detected in the same way (computation of the standard deviation, STD, for each segment and comparison to a threshold, and then a recursive least squares (RLS) algorithm [6] is used to remove the interference artefact). The RLS algorithm recursively finds filter coefficients that minimise a least squares cost function with a reference signal. For the interferencecontaining segment, the nearest segment without interference is selected as the reference signal. After filtering, the interference artefact of the segment is mostly removed which enhances RR estimation.

[0114] After the MA filtering has been performed in step S302, the resulting, filtered, in-ear audio data is subject to an adaptive HRV (heart rate variability) signal estimation in step S304. HRV signal is suitable owing to the respiratory sinus arrhythmia (RSA) phenomena, which implies respiratory rate is associated with heart rate variability. Specifically, RSA is the natural variation in heart rate that occurs due to synchronisation with respiratory cycle. Due to RSA, heart rate increases during inhalation and decreases during exhalation, resulting in a breathing related modulation of heart rate. Rhythms within the low frequency (LF) range of the HRV signal (i.e., the variation in time interval between successive heart beats), spanning from 0.04 to 0.15 Hz can serve as indicators of sympathetic modulation, and the high frequency (HF) range, ranging from 0.15 to 0.4 Hz, encapsulates rhythms governed by parasympathetic activity, which is closely related to respiration.

[0115] In step S304, the in-ear audio data is first pre-processed with application of a low-pass filter. In this example, the low-pass filter has a cutoff of 30 Hz. This removes the high-frequency noise, since it is known that heartbeats are low-frequency sounds [3]. This can be observed in Figure 5, where identified heartbeats are all (regardless of capture method) sub-30 Hz in frequency. A Hilbert transformation of the filtered in-ear audio data is then computed, with a moving mean filter, so as to obtain a heartbeat envelope. After this, a second low-pass filter is applied to emphasize heart-beat related sounds. To obtain the HRV signal (denoted here as SHRV), first the heartbeats are identified, i.e. , the heartbeat peaks as shown in Figure 9(b).

[0116] This is achieved by application of a peak detection method with an adaptive threshold. The adaptive threshold accommodates amplitude variation and morphology changes of the in-ear audio data. Specifically, a moving mean for each point of the pre-processed signal is calculated. Regions of interest (ROI) are picked between each pair of intersection points of the pre-processed signal and the moving mean signal. Heartbeat peaks are marked at the maximum of each ROI (between two intersection points), if their amplitude is larger than that of the two enclosing intersection points. This is illustrated in Figure 9(a). SHRV(an HRV signal) is then computed by calculating the time difference between successive detected peaks. An example HRV signal, which can be derived using this method, is shown in Figure 10(a) upper plot.

[0117] As the in-ear audio data includes two channels (left and right), the process can include estimating the HRV signal from each channel and selecting the channel with the lowest standard deviation. This is because heartbeats are a regular signal and the lower SNR of in-ear audio data leads to more regular heartbeat peak detection, i.e., the lower STD of the HRV signal.

[0118] Having obtained the HRV signal SHRV, an optimization problem is formulated to enable dynamic HF range localisation for subsequent analysis. This is based on the following observation: firstly, if a list of potential RR (respiratory rate) candidates are known, the best RR candidate should be the one closest to RReT; and, secondly, the best RR candidate will also be the one which, when used to set the frequency range of a bandpass filter (BPF) used to estimate the RR from SHRV, results in the best estimation of RRGT. In other words, the best RR candidate should satisfy the criterion that the difference between itself and the estimated RR using a BFP with frequency range centred around itself should be minimal. Therefore, the process aims to select the RR candidate that minimises such a difference from among a list of RR candidates. This process is discussed in more detail below in relation to Figures 22 and 23.

[0119] Expanding on this, the bandpass filter (BFP) is denoted as where h and I are the high and low cutoff frequencies respectively. In a first step of this sub-process, a predetermined BFP is used to filter the SHRVsignal. In this example the predetermined BFP has cutoffs I = 0.15 Hz and h = 0.35Hz. The filtered signal is then subject to a fast Fourier transform (FFT), and the frequency component with the highest amplitude is identified as the centre of the list of RR candidates and denoted as RRC(units: Hz). The RR candidates (units BPM) are then sampled as RRust= RRC■ 60 + n ■ 0.5, where the range of n satisfies RRmin< RR r < RRmax- RRmtn= max (7.5,RRC- 17) and RRmax= min(42.5, RRC+ 17), 7.5 BPM and 42.5 BPM being the smallest and largest RR seen empirically and

[0120] 60 being used to convert from Hz to BMP. Figure 10 is an example where RRCis 10, and the obtained list is therefore from 7.5 to 27 in 0.5 steps.

[0121] Figure 10(a) shows an example taking RRiist. Its F(h(.) is computed as I = (0.65 ■ RR^st) / 60 and h = (1.35 ■ 7?7?j1ist) / 6O. The heartrate variability signal, SHRV, is then filtered using this to obtain its respective extracted breathing signal. The RR of the extracted breathing signal is estimated using a FFT to find the frequency component with the largest peak, and the difference between the estimated RR and RRiistis calculated. The process is repeated for all Rfi^t to obtain a frequency difference list, which is further normalised (F-difference-list) as shown in Figure 10(b). The F-difference-list is normalized by rescaling the range of the list to between

[0001] . The rescaling, therefore, changes the distance between the minimum and maximum values in the list by stretching or squeezing the points along the number line. The RR^st with the smallest frequency difference can then be selected as the estimated respiratory rate.

[0122] In an improvement to this process, the steps can be repeated using zero-crossing counting on the extracted breathing signal in the time domain. This is shown in the middle plot of Figure 10(a). The obtained time difference after normalization (T-difference-list) is firstly smoothed, as it was found that it contains essential trends to lead to a near-optimal RR estimation, as shown in figure 10(c). It was empirically found that the optimal RR estimation is frequently located in one of the top-3 candidates, which achieve the top-3 smallest local minima in F-Difference-list. Therefore, the top-3 candidates of the F-Difference-list can be mapped to the T-Difference-list. Next the sum of each F-difference and each T-difference for each of these three candidates can be calculated. The candidate RR corresponding to the smallest sum of differences among these three candidates can then be chosen as the estimated respiratory rate.

[0123] In either case, the method moves to step S212 and the estimated respiratory rate is provided as an output.

[0124] Validation of this process included a preliminary study with data collected under a stationary scenario, and where the distribution of the ground truth RR (RRGT) is shown in Figure 7. A non-adaptive pipeline of RSA-based RR monitoring was followed where ground truth ECG data for heartbeat detection and HRV signal estimation was utilized. For comparison, a default frequency range of [0.15, 0.35] Hz was utilised in the non-adaptive pipeline. In the adaptive pipeline, a dynamic frequency range (RRGT- adapted) was used with values: [0.65 ■ RRGT, 1.35 ■ The resulting RR estimation performance is shown in Figure 8, with an improvement seen for the adaptive pipeline discussed above as compared to a non-adaptive (i.e., fixed) pipeline. Specifically, a large performance gain is seen through using a frequency range which is dependent on RRGT(MAE=1 .54 BPM) compare with the non-adaptive pipeline (MAE=3.06 BPM). This is because the default frequency range only performs well when RRCTis between 9 BPM (0.15 Hz) and 21 BPM (0.35 Hz). Moreover, even though RRGTis typically within this range, the non-centered HF range also degrades performance.

[0125] Figure 4 is a flow chart of a second in-ear audio data processing pipeline S210, which includes the sub-steps S402 - S406. Steps S402 and S404 can be performed in parallel, and either simultaneously or one after the other. The device of Figure 1 implements an LRC-based (locomotion respiratory coupling) RR estimation (i.e., the second in-ear audio data processing pipeline), in which the in-ear microphone is used to capture footsteps and rough patterns of breathing sounds whilst walking or running. LRC is a phase-locking between a person’s stride rhythm and their breathing rate. This means that there will typically be a consistent number of steps taken for each breath. LRC has been shown to improve the efficiency of breathing, and reduce expenditure of energy when stable ratios are used during locomotion. In humans, a number of LRC ratios have been observed, e.g., 4:3, 3:2, 2:1 , where an LRC of 2:1 means two steps are taken for one breathing event (inhalation or exhalation). Therefore, given a stride frequency and LRC ratio, a respiratory rate can be obtained.

[0126] Empirically, it was observed that the LRC ratio may change within an estimation window, especially for walking and non-regular runners. This was validated by dividing previously captured in-ear audio data into 10-second segments and calculating the mean LRC value for each segment. After this, the changing ratio between the mean LRC ratios of two adjacent segments was computed, where the higher ratio indicates a higher irregularity between the two segments. Figures 12(a) - (d) shows the changing ratio from user3 (a regular runner) and userl (a non-regular runner), whilst walking and running respectively. It can be seen that for non-regular runners, especially during walking, this assumption is false.

[0127] In step S402, an adaptive stride frequency estimation is calculated. Capturing clear step sounds can be reliably achieved using an in-ear microphone due to the occlusion effect. This is shown in Figure 13(b). First, a low-pass filter (with a 50 Hz cutoff) is applied to filter out high-frequency noise, since it is known both running and walking are low-frequency signals [3]. Then, a Hilbert transform is computed with a moving mean filter to obtain a step envelope as shown in Figure 13(a). From the step envelope, the same peak detection with adaptive threshold process discussed with respect to S304 of Figure 3 can be performed, so as to compute the stride frequency from the audio peaks. This is shown in Figure 13(a). To reduce false alarms in the peak detection, any detected peak with an interval that is too close to the previous peak can be discarded. Different thresholds are set, empirically, for running and walking, which is smaller than the shortest time interval between two steps in the collected data set (see discussion below), i.e., 0.2s and 0.3s respectively. After step detection, the step frequency is computed as step number minus one divided by the time interval between the first and last step.

[0128] In step S404, a likelihood curve decomposition process is performed which aims to decompose the extracted respiration-related information into various components for interference removal. As the sound frequency of a human breath typically falls within the range of 300 Hz to 1800 Hz [4], a bandpass filter with cutoff frequencies from 300 Hz to 1800 Hz can be applied on the captured in-ear audio data. During testing, in-ear audio data was collected from a single user whilst stationary and breathing loudly in a quiet environment. This was used to generate an MFCC (Mel-frequency cepstral coefficient) vector indicative of the features of breathing (also referred to as a ‘breathing print’). For each frame of the filtered captured in-ear audio data, a similarity between its MFCC features and the feature print is calculated. From the measured similarity, the likelihood of the frame containing a breath is estimated. It has been found that there is no need to generate a feature print for each user, instead using a single predetermined feature print for all future determinations.

[0129] The likelihoods from all frames of the in-ear audio data are used to generate a respiration likelihood curve (see Figure 14(a)). Due to the strong interference from step sounds and noises, the respiration pattern may be overwhelmed by patterns of interference. Therefore, before removing the interference patterns, the method includes decomposing the likelihood curve first. An SSA (singular-spectrum analysis) algorithm is utilised, as it can decompose the time series data into distinct components which are interpreted as trend, periodicity, and noise. SSA algorithms are also able to effectively separate the underlying components, allowing for the isolation of periodicity occurring at various time scales, in order of significance, even within highly noisy time series data. Figure 14(b) shows component 7 of the decomposed likelihood curve, which clearly corresponds to the steps taken whilst the user is walking (i.e., the accelerometer data shown in Figure 10(b) as ‘GT-acc’). Once the likelihood curve has been decomposed into periodic components, the respiration-related components are selected for RR estimation and those related to interference are excluded.

[0130] The method utilises a loose constraint to exclude respiration-unrelated components. Specifically, for each decomposed component of the likelihood curve, the number of peaks is counted. If the number of peaks for a given component falls outside of a range defined as the lowest possible number of breathing events (denoted as RRmtn and the highest possible number of breathing events (denoted as RRmax) then that component is regarded as being unrelated to respiration. To obtain RRmtnand RRmax, the largest and smallest GT LRC ratios were identified from a walking and running data set. These were denoted, respectively, as LRC ,kand LRC , and LRC^ikand LRC^ respectively.

[0131] Taking walking as an example, RRminand RRmaxcanbe computed by:

[0132] Where SFest, fs, and T are the estimated step frequency, audio sampling rate, and sample numbers of the estimation window, respectively. As can be seen, the RR estimation is not dependent on a fixed LRC but rather focuses on upper and lower bounds of possible LRC ratios. This means the method can facilitate changing LRCs within a window.

[0133] After excluding all breathing-unrelated components, the remaining components are aggregated together in step S406. This is done by summing them. The aggregated signal is then regarded as an extracted breathing pattern and a peak detection algorithm is applied in order to estimate the final RR, as illustrate in Figure 14(c). The estimated RR is provided as an output in step S212.

[0134] The method and device were developed and validated using data collected from 18 participants (9 males and 9 female). Figure 5 show a series of plots comparing signals captured from an IMU, an out-ear microphone, and an in-ear microphone in various scenarios which is illustrative of the signals captured and forming the dataset. Specifically, Figure 5 shows signals simultaneously captured from an IMU (Figure 5(a)), out-ear microphone (Figure 5(b)), in-ear microphone (Figure 5(c)), and a chest strap for reference (providing the ground truth step, heartbeat, inspiration, and expiration points). For the data shown in Figure 5, the subject was naturally breathing under different conditions, i.e., stationary sitting, walking on a treadmill, and running on a treadmill. The overlaid markers on the figures are locations of physiological / motion signals extracted from the reference chest strap.

[0135] It was observed during testing that the IMU can clearly detect steps whilst walking and running, but that breathing signals are not discernible (particularly under motion). Similarly, it was observed that the out-ear microphone is hardly able to capture any sound except for environmental noises while stationary. Whilst walking / running, footsteps can be detected, but they are interfered with by the ambient noise of the treadmill. However, Figure 5(c) shows that heartbeats are captured by the in-ear microphone due to the occlusion effect (the increased impedance in the occluded ear canal, resulting in amplification of low frequency sounds). Although low intensity light breathing sounds may be undetectable when moving, footsteps can be clearly detected in the in-ear audio data with much lower interference from ambient noise due to occlusion as compared to the out-ear microphone. In exploring the sensing capacity of breathing sounds using an in-ear microphone, the data shown in Figure 6 was collected. Figures 6(a) and 6(b) show simultaneously captured signals from an in-ear microphone and out-ear microphone respectively, whilst the user was loudly breathing during a stationary sitting context. It can be seen that both the in-ear microphone and out-ear microphone can capture the breathing sounds, whilst the in-ear microphone has the stronger intensity in the low frequency components with lower noise due to the occlusion effect. This demonstrates that, like an out-ear microphone, an in-ear microphone can detect rough patterns of breathing whilst moving.

[0136] Data collection and performance evaluation

[0137] The 18 participants underwent stationary and motion activities. The stationary activities which were performed for 5 minutes included: (1) sitting; (2) standing; (3) lying down; (4) listening to music (performed for the duration of one song); (5) working in the wild; and (6) uncontrolled cooldown after exercise. The moving activities, again performed for 5 minutes each, included: (7) walking; and (8) running. The activities encompass typical scenarios when a person uses earbuds and where breathing rate monitoring would be desired. The data collection involved controlled activities and also in-the-wild scenarios to ensure the applicability of the methods disclosed herein to real-world use. No breathing rates were imposed. Breathing after exercise was performed immediately after the user had completed their walking and / or running and so captured their natural cool-down breathing. Whilst sitting, users were asked to move their head three times to capture head motions for assessing the impact of the MA filtering algorithm. A Knowles SPU1410LR5-HQB microphone was utilised in a 3D printed earbud. The microphone was connected to an MPU 6050 IMU breakout board, and a Raspberry Pi 4B was used to capture signals from the sensors.

[0138] Motion activities were performed on a treadmill to control cadence. However, participants were given a choice of three speeds so that a comfortable pace could be selected. For walking, speeds ranged between 2 km / h and 5 km / h. For running, between 5 km / h and 8 km / h. the average sound level in the room was 58 dB when the participant was moving, and 32 dB whilst they were stationary.

[0139] Ground truth (GT) data was collected using a chest strap which recorded respiratory rate. In total, 684 minutes of data was captured with the GT data being sampled at 25 Hz and the microphone data at 22050 Hz. The collected data has a minimum, mean, and maximum breathing rate of 7 BMP, 18 BPM, and 39 BPM respectively. After data collection, some portions of the data was excluded due to poor fitting of either the GT collection devices or wearable respiratory rate sensing device. In total, 17 out of 144 pieces of data were excluded, where each piece refers to one activity from one participant. The performance of the device and method were evaluated using the following metrics: 1) Mean absolute error (MAE: the average absolute error between the GT RR (BPMGT) and the calculated RR (BPMdevice) for each window (i, i e [1, JV]) as computed SAE: the standard deviation of the MAE; and 3) Mean Average Percentage Error (MAPE): the average percentage absolute error.

[0140] Table 1 below provides the results of the scenario detector module, using the SVM on 5-second segments. The SVM was trained on 13 participants’ data (randomly chosen during training) and tested against the remaining 5 participants’ data. The module was able to classify scenarios with excellent performance, achieving 100% accuracy for determining whether a window indicates the participant to be moving or stationary (Task 10 and 99% accuracy for determining whether a window indicating the participant to be moving indicates whether they are walking or running (Task 2). With the voting of detection results across one estimation window (as discussed above), i.e., over 60s, the detection accuracy of both tasks can achieve 100%:

[0141] Table 1 : Performance of scenario detection

[0142] To assess the accuracy of the HRV estimation, the MAPE was computed between the ground truth HRV and the estimated HRV. When using the automated channel selection process discussed above, an MAPE of 5.46% was achieved.

[0143] The results for the stride frequency estimation are provided in Table 2. Overall, the system and method was found to detect strides with a MAPE of 3.2% for data originating from the left ear and 5.29% for data originating from the right ear.

[0144] Table 2: Stride frequency estimation error As was discussed above with relation to the motion artifact filtering process, when selecting the parameters for MA filtering there is a trade-off between the minimum window length retained and the threshold for MA filtering as shown in Figure 15. With a minimum window length of 60 seconds, and a threshold of 0.04, an MAE of 1.84 BPM can be obtained. However, this is achieved by removing 41% of the windows, which is undesirable for in-the-wild usage. To manage this trade-off, a window length of 40s with a threshold of 0.1 was selected, resulting in a stationary MAE of 1 .9 BPM whilst only removing 15% of the windows. This results in a relatively high stationary data retention.

[0145] Using the selected parameters from Figure 15, the results of the MA filtering process whilst a user is stationary are shown in Figure 16. It is evident that the MA filtering reduces the variance in the error and the median error as compared to estimations performed without MA filtering. This implies that the error is smaller when it is applied, i.e., MAE = 1 .9 BPM with MA filtering and MAE =2.1 BPM without MA filtering. This amounts to an 8% error improvement.

[0146] Performance was also assessed using the left and right microphone channels for RR estimation, as was the impact of automatically selecting a channel and fusing the channels. This was done for stationary and moving scenarios.

[0147] Figure 17 provides a boxplot of the error for the left and right channels and the automatic channel selection algorithm whilst the user was stationary. The performance of the two channels was found to be similar (with median errors of 1 .17 BPM). However, there is variance in the result due to different motion artifacts in the two ears. After implementing the auto selection algorithm, the median error (0.98 BPM) and variance were found to be significantly lower than that of either channel, showing the importance of this technique when two earbuds are won. However, even on each channel in isolation, the device and method performance are still good. This demonstrates that the system and method are applicable even when the user is only wearing a single earbud.

[0148] Figure 18 provides the results for the left, right, and fusion of the two channels (i.e., when a mean value of the estimated RR from each of the channels is taken as the estimated RR), for both walking and running. Results when fusion is performed during a walking scenario is better than either channel (2.25 BPM v 2.5 BPM for fusion v left / right respectively). However, when running, a lower median error was seen on the right channel (2.5 BPM) than the left channel (3.5 BPM) and fused channels (3 BPM). The right channel was also seen to have a larger spread of errors. The fusion channel was found to have a median error in between that of the left and right channels, and a narrower spread than the right channel but a wider spread than the left. As the estimations from the left and right channels were averaged, the resulting estimation was found to have an error between the errors of the two channels. Therefore, again, it is demonstrated that the device and method can function accurately with each channel in isolation. However, fusing the two channels generally improves the robustness of the estimation and decreases the variation in errors.

[0149] Using MA filtering, automatic channel selection, and fusion of the channels for the stationary and moving cases respectively, the overall performance of the device and method in estimating RR is shown in Table 3. Overall, an MAE of 2.25 ± 2.48 BPM was achieved, with an MAE of 1 .95 ± 2.49 BPM and 3.12 ± 2.45 BPM for stationary and moving scenarios, respectively. In comparison, in-ear photoplethysmogram based respiratory rate monitoring has been reported to achieve a mean error of 3 BPM when stationary [5].

[0150] Table 3: Overall performance of device and method

[0151] Figure 19 is a plot showing the performance of the device and method for each of 18 participants. As can be seen, there is little variation in the median error of each participant, and all participants (excluding participants 8 and 13) have a median error below 1.7 BPM. Participants 8 and 13 are outliers in the study, with median errors of 2.75 BPM and 3.45 BPM respectively.

[0152] Figure 20 is a plot showing the performance of the device and method for a range of activities. It shows the overall performance of the device and method when using the auto channel selection (stationary) and fusion (moving) cases. The estimation errors whilst walking and running were found to be higher than that of any of the stationary activities. Of the stationary activities, the largest errors were seen whilst listening to music, standing, and working. The larger error in standing and working are believed to be due to the increased number of motion artifacts whilst doing these activities. When standing, participants tended to fidget and move their bodies resulting in increased motion artifacts. Working is an uncontrolled activity, and so participants were more active during this task resulting in an increase in motion artifacts. As discussed above, the motion artifact removal threshold was selected to maximise window retention. Therefore, small motions like fidgeting are not likely to be removed by the MA filter and so a slightly larger error is the result.

[0153] Since the bulk of the experiments were performed indoors, assessment of the device and method’s performance outdoors in an uncontrolled environment was also undertaken on three participants. When performing this study, only normal breathing walking, running, and cooling down was assessed as these represent the most typical of the activities to occur outdoors. The average sound level during the tests was 53dB, and when performing walking and running outdoors the participants were free to select their preferred pace and move freely around the area. The results of this study are shown in Table 4, and the overall MAE was found to be 2.22 ± 1.10 BPM. Comparing this to Table 3, it is evident that the performance is similar whether the activities occur indoors or outdoors.

[0154] Table 4: Performance of device and system in outdoor settings

[0155] An important use case of the device and method is when the wearer is moving whilst listening to music, as this is a common use of case of an earbud. The performance of the device and method was investigated for one participant who walked and ran on a treadmill whilst listening to music. Under these conditions, an error of 1 .5 BPM and 2.4 BPM were obtained for walking and running, respectively. The results are comparable to those obtained without music.

[0156] In a further study, two participants wore the device for approximately an hour whilst undergoing standard daily activities (so called in-the-wild testing). The study took place in a busy office, with an average sound level of 43dB. The participants worked at their desks, listening to music, walked around the office, and undertook other activities as desired. Figure 21 is a plot showing the performance of the device and method when worn by one of the participants during this study. The boxes A, B, and C denote different activities, namely: A - working whilst sitting; B - Walking around; C - Working and listening to music. As can be seen, the device and method can accurately track RR longitudinally, even in an uncontrolled setting. The sudden errors in the plot are due to MAs, as moderate MA filtering parameters were used in the device to achieve a high retained ratio of windows, i.e., 87% for this study. Overall, for the longitudinal tracking, the device and method achieved a MAE of 1 .38 BPM, with a MAE of 0.91 BPM and 3.25 BPM for stationary and moving scenarios respectively. The ground truth RR (GT RR) was obtained using a chest strap. This is comparable to the controlled studies discussed previously.

[0157] Method of determining respiratory rate from heart rate variability

[0158] Figure 22 shows a flowchart of a method S500 for determining an estimate for respiratory rate from a heart rate variability (HRV) signal.

[0159] As discussed above, the present inventors have conceived of a method of dynamic HF range localisation to determine a more accurate estimate for the respiratory rate from a list of potential RR (respiratory rate) candidates. The HRV signal may be obtained from in-ear audio data. However, in other examples the HRV signal may be obtained from other sources such as the ECG data, blood pressure signals, ballistocardiograms, and / or PPG data (photoplethysmograph).

[0160] In a first step, S502 a list of candidate respiratory rates (FT?^) is generated. This may be a predetermined list of candidate respiratory rates. Alternatively, the candidate respiratory rates may be generated based on an initial estimate for respiratory rate as described above in relation to Figure 10.

[0161] Next, in step S504 a candidate specific filter is applied to the HRV signal for each candidate respiratory rate, wherein the passband (i.e., the cut-off frequencies) of each candidate specific filter is determined based on the respective candidate respiratory rate. As discussed above in relation to Fig. 10, each candidate specific filter;h(. ) is computed as having I = 0.65 ■ RR^lst / 60 and h = 1.35 ■ is the nth candidate specific respiratory rate.

[0162] In step S506, an interim respiratory rate is determined for each filtered HRV signal to provide a plurality of candidate respiratory rates. Specifically, the interim respiratory rate is calculated by performing a Fast Fourier Transform (FFT) on the filtered HRV signal, and identifying the highest amplitude peak in the transformed signal. The initial respiratory rate is given by the frequency of the highest amplitude.

[0163] In other examples, the interim respiratory rate may be calculated using time domain analysis such as by measuring a zero-crossing rate. Some examples (as discussed below in relation to Figure 23), multiple different methods of estimating respiratory rate may be employed to provide multiple interim respiratory rates. Finally, in step S508 an estimate for the respiratory rate is determined based on the interim respiratory rates. This includes determining the difference between each interim respiratory rate and each respective candidate respiratory rate. The estimate for the respiratory rate is then given by the candidate respiratory rate corresponding to the smallest calculated difference. In other examples, the respiratory rate is given by the interim respiratory rate corresponding to the smallest calculated difference. In yet further examples, (e.g., as described below in relation to Figure 23) a plurality of the smallest candidate differences are determined and used to select interim respiratory rates determined by a different method.

[0164] Figure 23 shows another flowchart of a method for estimating respiratory rate from an HRV signal. The method of Figure 23 is one example embodiment of the method in Figure 22 where the interim respiratory rates are measured using a second method and compared to the first interim respiratory rates to provide an improved estimate.

[0165] In a first step S602, a list of candidate respiratory rates is generated, and, in step S604, a candidate specific filter is configured for each candidate respiratory rate and applied to the HRV signal to produce a plurality of filtered HRV signals. These steps are, therefore, the same as steps S502 and S504 of Figure 22.

[0166] In step S606 a first interim respiratory rate is determined from each filtered HRV signal by performing frequency domain analysis to determine a dominant frequency component. This includes performing a Fast Fourier transform (FFT) of the filtered HRV signal, and identifying the highest amplitude peak in the transformed signal. The initial respiratory rate is given by the frequency of the highest amplitude peak.

[0167] In step S608 the difference between each first interim respiratory rate and the respective candidate respiratory rate is calculated to produce a plurality of frequency-domain differences (i.e., F- differences) as shown in Figure 10b.

[0168] In step S610 the n smallest F-differences are identified. In Fig. 10b the three smallest F-differences are identified. More specifically, the three smallest local minima (or troughs) of the F-differences are identified.

[0169] In step S612, a second interim respiratory rate is determined for each candidate respiratory rate. In contrast to step S606, the second interim respiratory rate is determined by calculated a zero crossing rate of each filtered HRV signal. In step S614, the difference between each interim respiratory rate and the respective candidate respiratory rate is calculated to produce a plurality of “time-domain” differences (i.e., T-differences) as shown in Figure 10c. NB: steps S612 and S614 may be performed before, after, or in parallel with steps S606 to S610.

[0170] Next, in step S616 the T-differences which correspond to the same candidate respiratory rates as the n smallest local minima in the F-differences (identified in step S610) are determined to obtain n candidate T-differences as shown in Figure 10c.

[0171] In S618, each candidate T-difference and each corresponding F-difference is summed to obtain n candidate sums. Next, in S620 the smallest of the n candidate sums of the T-differences and F- differences is determined. Finally, in S620 a final estimate for the respiratory rate is determined as the candidate respiratory rate corresponding to the smallest sum of the n candidate T-differences and F- differences.

[0172] In some examples, the method of Figures 22 and 23 may include an initial prefiltering step where a predetermined BFP is used to filter the SHRVsignal. Each subsequent step in Figures 22 and 23 is then performed on the pre-filtered HRV signal.

[0173] Evaluation (variant motion artefact implementation)

[0174] Further evaluation of the devices and methods was undertaken where the adaptive filter was used to mitigate motion artefacts. As before, the device and method’s performance were evaluated using the Mean Absolute Error (MAE), which is the average absolute error between the ground truth (GT) respiratory rate (RR) and the calculated RR for each window. The Mean Average Percentage Error (MAPE) was also used, which is the average percentage absolute error.

[0175] The overall performance is shown in Figure 24(a). An overall MAE of 1.71 BPM (MAPE of 9.68%) was achieved, with MAE of 1 .48 BPM (9.12%) and 2.28 BPM (11 .04%) for sedentary and active respectively. The Bland-Altman plot is provided in Figure 24(b). A very low mean error of -0.02 BMP with narrow limits of agreement of -4.8 to 4.76 was achieved. This indicates very good agreement between the calculated RR and the ground truth breathing rate measurements.

[0176] Figure 25(a) provides a box-plot of the overall performance for each activity. The performance of each sedentary activity is comparable. Slightly higher errors were found to exist when listening to music (MAE=1 .98 BMP) (this is discussed in detail below) and working (1 .56 BPM). It is believed this is because working is an uncontrolled activity, and therefore participants were more active during this task, leading to more interference artefacts. The estimation errors while walking and running were found to be good, i.e., walking (MAE = 1 .75 BMP; MAPE = 9.17%) and running (MAE = 3.12 BPM; MAPE = 14.01%). The slightly higher running errors are believed to be due to the higher footstep interference.

[0177] Figure 25(b) reports the overall performance for each participant. It is evident that the MAE whilst sedentary is consistent amongst the users, with no user exceeding 2.3 BPM error. There is more variation amongst estimation errors whilst active: the smallest MAE being 1.26 BPM for user 16, and the largest MAE being 5.25 BPM for user 7. The majority of error is believed to come from 2 users, i.e., user 7 and user 12. User 7 ran at 5 KPH, which is believed to have slightly worse performance than faster running speeds (discussed below). User 12’s running generated large amounts of noise due to their feet hitting the side of the treadmill, resulting in high-energy noise across all frequencies in the in-ear audio. However, despite this, the performance was good for the majority of users over all activities.

[0178] The performance of the device and method is compared to those of existing works. To perform this study, data was collected from 11 users in a combination of indoor and outdoor settings, whilst the subjects were sitting still, cooling down after running (sedentary), walking and running (active). Three IMU based algorithms for RR estimation under sedentary conditions were implemented: an FFT approach (FFT), a peak detection approach (Peak) and a zero-crossing rate (ZCR) approach (Figure 26(a)). For the in-ear and out-ear microphones, an algorithm was implemented for sedentary estimation which used a peak detection approach where peaks are detected from the envelope of the microphone signal (Figure 26(b)). For active, an algorithm employed by [7] (LRC) was implemented, and expanded upon to calculate RR (Figure 26(c)). From Figures 26(a) - (c), it is evident that under both sedentary and active scenarios, the device and method disclosed herein significantly outperforms the comparison methods.

[0179] Two-motion resilient HR estimation methods were implemented, to assess their applicability whilst active. Specifically, HRV was derived from a generated “ECG” signal utilising the method in [3] during walking and running, integrating it into the RSA-based RR monitoring pipeline (discussed above). Walking / running frequency components were eliminated from in-ear audio utilising the method in [8] and then a [0.1 , 0.8] Hz band-pass filter [ was applied for RR estimation. Figure 26(c) shows that these methods result in large errors when used for RR monitoring whilst active, which underscores the contribution of the LRC-based pipeline disclosed herein.

[0180] The performance of the device and method was assessed when users engaged in other activities of varying intensities involving rhythmic footsteps, including step aerobics (StepA), climbing up and down stairs (StairUD), and rowing on an indoor rower (ROW)- 5 minutes of data was recorded per activity per participants, with activities performed in an uncontrolled manner. Figure 27(a) indicates that the device and method work properly during different rhythmic activities using the LRC-based pipeline.

[0181] The performance outdoors in an uncontrolled environment was also assessed. The tests were performed on a concrete pavement outside of an academic building next to a building site with active construction ongoing. The performance was assessed under different activities, including sitting, walking, running, and cooling down. 5 minutes of data per activity for each participant was recorded. When walking and running, participants were free to select their preferred pace and move around the area. As a result, natural changes in pace throughout the experiment occurred, to test whether the device and method functions under both controlled and uncontrolled speeds. Figure 27(b) shows the results, indicating robust performance outdoors.

[0182] The performance impact of ambient noise levels was also assessed. Figures 27(c) and (d) indicate the results. It can be seen that over three ambient noise levels for sedentary (Figure 27(c)) and active (Figure 27(d)), the device and method achieved consistent results. This is believed to be due to occluding the ear canal using earbuds which attenuates ambient noise, and the occlusion effect which attenuates high frequency external noise and amplifies low frequency heartbeat and footstep sounds.

[0183] Figure 28(a) provides the overall MAE in the left and right channels and the channel chosen by the automatic channel selection algorithm discussed above (denoted by ‘two’) while sedentary. The individual performance of the two channels is similar, with an MAE of 1 .8 BPM and 1 .79 BPM for left and right respectively. After the audio channel selection algorithm has been applied, the MAE is significantly reduced (1 .42 BPM). Moreover, even from one channel, the performance is still good. This indicates that the device and method is applicable even when the user is wearing a single earbud. Figure 28(b) provides the MAE for the left, right, and fusion of two channels (i.e., the mean value of the estimated RR from two channels) whilst active. Again, it can be seen that the fusion channel has a lower error than that of the two channels, with a MAE of 2.28 BPM for the fusion compared to 2.34 BPM and 2.56 BPM for left and right respectively.

[0184] In Figure 28(c) and (d), the impact of different speeds was assessed whilst active. For both walking and running, the median error remained similar regardless of speeds. A slightly higher error was seen for running at 5 Kmph. This may be attributed to the lower SNR of low-speed running, which causes similar interference from running footsteps but induces weaker breathing sounds compared to higher speeds. However, even with this slight trend, good results were achieved across different speeds.

[0185] The impact of listening to music through the device was assessed, whilst the users were sedentary, walking, and running. For each condition, music was played on the left channel covering a range of common genres with different volumes. Figures 29(a) and (b) provide a comparison of performance per genre per activity and performance with different volumes. Performance across all genres and with different playing volumes whilst sedentary is comparable with the sedentary performance without music playing (the error of the left channel as shown in Figure 29(a) i.e. , 1 .8BPM). This is due to the minimal overlap of heart sound frequencies with the music frequency, with only 0.5% of music energy lying in the 0-30 Hz range. Whilst active, the performance across all genres is slightly higher than the one without music. Similarly, errors are 2.50 BPM and 2.92 BPM for soft and loud music respectively compared to 2.34 BPM when active without music (the error of the left channel in Figure 29(a). It is seen again that music does not significantly degrade RR estimation performance whilst active. There is again a non-overlap between music and stride frequency, meaning that music does not impact stride detection accuracy. Although music and breathing sounds have overlapping frequencies, the breathing sounds are not used directly, but rather generate a breathing template to detect the probability of breathing sounds. This means that breathing and music can be easily differentiated, mitigating the impact of music on accuracy. Users wore the device for an hour in a busy office whilst undergoing standard daily activities. The users worked at their desks, listened to music, walked around the office, and performed other activities as they wished, such as sipping coffee or using cell phones. Figure 30 shows the tracking performance for two users, i.e., continuously estimated RR using the device (one RR estimation per 30s) compared to GTRR. User A (shown in Figure 30(a)) worked and walked around the office. User B (shown in Figure 30(b)) worked at their desk and listened to music whilst working. User A achieved an MAE of 0.97 BPM and 1 .83 BPM whilst working and walking respectively, and user B achieved errors of 1 .01 BPM and 0.58 BPM whilst working and listening to music respectively. These errors are consistent with the results obtained in the laboratory study for these two users, showing excellent performance both in controlled laboratory settings and uncontrolled, real-world settings. It also demonstrates the ability to accurately track RR longitudinally, even in uncontrolled settings.

[0186] The impact of active noise cancellation (ANC) was also assessed. This was performed by application of a high-pass filter to the collected data (with the low cutoff mimicking the hardware filters on a commercial ANC system) and then running the processing pipelines. The results of this study are shown in Figure 31 . Under both active and sedentary conditions, the median error is mostly unaffected by the filter, with the slightly higher error in sedentary due to the very low frequency of heartbeat sounds. However, overall, this study demonstrates that the method and device functions accurately on earbuds utilising ANC. Concretely, although the signal energy of heartbeat sounds are being filtered out, due to the high quality occlusion, signal components after attenuation on low frequency sounds are still obtained which can be reliably used for estimation. Further, if ANC filters are implemented in software, the filtering can be applied selectively based on the application of the acquired sound.

[0187] Figure 32(a) provides the results of the pipeline selector module using SVM on 5s segments. The SVM was trained on 13 users’ data (randomly chosen during training) and tested on the remaining 5 users to ensure user independence on the train and test sets. A 5-fold cross validation was implemented, and reported is the average results over 5 folds. The device and method was able to select pipelines with excellent performance, achieving 100% accuracy for determining whether a window is active or sedentary (SVM-Stage-1) and 99% accuracy for determining whether an active window is low-intensity or high-intensity rhythmic footstep activities (SVM-Stage-ll). With majority voting of results across an estimation window, the detection accuracy on both tasks is 100%.

[0188] The accuracy of HRV estimation was also assessed, with the MAPE computed between the GT HRV from ECG data and the estimated HRV on a beat-to-beat level (Figure 32(b)), where performance is compared on the left channel, right channel, and under the automatic channel selection algorithm (‘two’). Using the automatic channel selection algorithm, the best performance was achieved with an MAPE of 3%, which is competitive with reported results using ECG data on well-known datasets, e.g., the MIT-BIH Arrhythmia Database. The results for stride frequency estimation are provided in Figure 32(c). The device and method detect strides using an MAPE of less than 3% for both walking and running using channel fusion. Which is again competitive with literature on in-ear step counting.

[0189] An ablation study was performed, investigating the impact of the various components of the method on prediction accuracy. First, as shown in Figure 33(a), the impact of the interference artefact filtering was examined. The MAE without interference artefact filtering is 1 .63 BPM, and with the filter is 1 .45 BPM, showing a 12% decrease in error by application of the filter. It can also be seen that interference artefact filtering decreases variance and median error in estimation, making estimations more robust. Next, adaptive breathing signal extraction was examined. Figure 33(b) illustrates the performance whilst sedentary compared to the performance obtained when directly using the adaptive BPF filter generated from GTRR(discussed above). The device and method achieve an MAE of 1 .48 BPM, and with the adaptive filter from the GTRR, an error of 1 .44 BPM was obtained. It can be seen then that the design and method can lead to estimations very close to the upper bound of performance using the RSA-based RR estimation principle.

[0190] The calibration from the T-difference list was then examined, with Figure 33(c) showing the importance of the fusion of the T-Difference-List and F-Difference-List for achieving accurate estimations, with the fusion technique lowering error by 32.43% of the F-Difference-List. Figure 33(d) compares using the FFT features herein to MFCC features for creating the breathing features for RR estimation under activity. Although MFCCs are most commonly used for feature extraction, a significant performance increase is gained for both walking and running when using FFT features of the type disclosed herein. Finally, the impact of the LRC range selection was examined. The device and method leverage the common LRC ranges for walking and running in humans respectively. To further test robustness, the device and method were tested whilst active using an increased LRC range for walking and running, i.e., ±0.2. The results show that the performance is comparable, i.e., 0.2 BPM difference for walking and running.

[0191] ***

[0192] The systems and methods of the above embodiments may be implemented in a computer system (in particular in computer hardware or in computer software) in addition to the structural components and user interactions described.

[0193] The term “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above described embodiments. For example, a computer system may comprise a central processing unit (CPU), input means, output means and data storage. The computer system may have a monitor to provide a visual output display. The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network.

[0194] The methods of the above embodiments may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described above.

[0195] The term “computer readable media” includes, without limitation, any non-transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape, optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.

[0196] While the disclosure has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the disclosure set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the disclosure.

[0197] In particular, although the methods of the above embodiments have been described as being implemented on the systems of the embodiments described, the methods and systems of the present disclosure need not be implemented in conjunction with each other, but can be implemented on alternative systems or using alternative methods respectively.

[0198] The features disclosed in the description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the disclosure in diverse forms thereof.

[0199] While the disclosure has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the disclosure set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the disclosure.

[0200] For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations.

[0201] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0202] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

[0203] References

[0204] [1] Gary G Berntson, John T Cacioppo, and Karen S Quigley. 1993. Respiratory sinus arrhythmia: Autonomic origins, physiological mechanisms, and psychophysiological implications. Psychophysiology 30, 2 (1993) 183 - 196

[0205] [2] Dennis M Bramble and David R Carrier. 1983. Running and breathing in mammals. Science 219, 4582 (1983), 251 - 256

[0206] [3] Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo. 2023. hEARt: Motion-resilient Heart Rate Monitoring with in-ear microphones. In 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 200 - 209

[0207] [4] Ana Oliveira and Alda Marques. 2014. Respiratory sounds in healthy people: a systematic review. Respiratory medicine 108, 4 (2014), 550 - 570.

[0208] [5] Andrea Ferlini, Alessandro Montanari, Chulhong Min, Hongwei Li, Ugo Sassi, and Fahim Kawsar. 2022. In-Ear PPG for Vital Signs. IEEE Pervasive Computing 21 , 1 (Jan. 2022), 65 - 74

[0209] [6] J Cioffi and Thomas Kailath. 1984. Fast, recursive-least-squares transversal filters for adaptive filtering. IEEE Transactions on Acoustics, Speech, and Signal Processing 32, 2, (1984), 304- 337

[0210] [7] Fei Gu, Jianwei Niu, Sajal K Das, Zhenxue He, and Xin Jin. 2017. Detecting breathing frequency and maintaining a proper running rhythm. Pervasive and Mobile Computing 42 (2017), 498- 512.

[0211] [8] Xue Sun, Jie Xiong, Chao Feng, Wenwen Deng, Xudong Wei, Dingyi Fang, and Xiaojiang Chen. 2022. Earmonitor: In-ear Motion-resilient Acoustic Sensing Using Commodity Earphones. Proceedings of the ACM on Interactive, Mobile, wearable and Ubiquitous Technologies 6, 4 (Dec. 2022) 1-22.

Claims

CLAIMS1 . A wearable respiratory rate sensing device, comprising: a processor, connectable to an audio sensor and configured to: receive captured in-ear audio data from the audio sensor; detect, from the captured in-ear audio data, a scenario of the wearer; select, based on the detected scenario, an in-ear audio data processing pipeline from a plurality of in-ear audio data processing pipelines; and deriving, by applying the selected in-ear audio data processing pipeline to the captured in-ear audio data, a respiratory rate of the wearer.

2. The device of claim 1 , wherein the processor is configured to detect, from the captured in-ear audio data, whether the wearer is in a moving scenario or a stationary scenario.

3. The device of claim 2, wherein the processor is configured to select a first in-ear audio data processing pipeline when the wearer is detected as being in a stationary scenario, and a second in- ear audio data processing pipeline when the wearer is detected as being in a moving scenario.

4. The device of claim 3, wherein the processor is configured to select a first set of parameters for the second in-ear audio data processing pipeline when the wearer is detected as being in a walking scenario, and a second set of parameters for the second in-ear audio data processing pipeline when the wearer is detected as being in a running scenario.

5. The device of claim 3 or 4, wherein the first in-ear audio data processing pipeline includes a step of estimating a heart rate variability, HRV, from the captured in-ear audio data, and subsequently extracting the respiratory rate from the estimated heart rate variability.

6. The device of claim 5, wherein the heart rate variability is adaptively estimated.

7. The device of claim 6, wherein a peak detection threshold, used in estimating the HRV, is adaptively changed during estimation of the HRV.

8. The device of any of claims 3 - 7, wherein the first in-ear audio data processing pipeline includes an initial step of filtering the captured in-ear audio data for motion artifacts.

9. The device of any of claims 5 - 8, wherein the captured in-ear audio data includes a first and second audio channel, and the heart rate variability is estimated using both channels separately to obtain a first estimated heart rate variability with a first standard deviation and second estimated heart rate variability with a second standard deviation, and the estimated heart rate variability with the lower standard deviation is selected for use in deriving the respiratory rate.

10. The device of any of claims 5 - 9, wherein extracting the respiratory rate from the estimated heart rate variability includes selecting a candidate respiratory rate, from a plurality of candidate rates, the selected candidate respiratory rate having a smallest difference between itself and an estimated initial respiratory rate, wherein each estimated initial respiratory rate is determined by applying a filter to the estimated heart rate variability, the passband of the filter being based on the respective candidate respiratory rate.11 . The device of any preceding claim, wherein the second in-ear audio data processing pipeline includes: (i) estimating a stride frequency of the wearer using the captured in-ear audio data and (ii) generating a likelihood curve for a plurality of frames of the captured in-ear audio data, indicating whether a given frame is likely to contain breathing information, steps (i) and (ii) being performed in either order in series or in parallel, and the estimated stride frequency and likelihood curve being used to derive the respiratory rate.

12. The device of claim 11 , wherein the likelihood curve is decomposed into a plurality of components, and one or more of the plurality of components are then selected and used, together with the estimated stride frequency, for deriving the respiratory rate.

13. The device of claim 12, wherein the one or more components are selected by comparing the number of peaks to a range of acceptable values, and selecting the component if the number of peaks falls within the acceptable range of values.

14. The device of any preceding claim, wherein the processor is configured to detect the scenario of the wearer by providing the captured in-ear audio data to a machine learning classifier, the output of which is a detected scenario.

15. A method of deriving a respiratory rate from in-ear audio data captured from a wearer of a device including an audio sensor, the method comprising steps of: detecting, from the captured in-ear audio data, a scenario of the wearer; selecting, based on the detected scenario, an in-ear audio data processing pipeline from a plurality of in-ear audio data processing pipelines; and deriving, by applying the selected in-ear audio data processing pipeline to the captured in-ear audio data, a respiratory rate of the wearer.

16. A computer-implemented method of estimating respiratory rate from a heart rate variability (HRV) signal, the computer-implemented method comprising:(a) generating a list of candidate respiratory rates,(b) for each candidate respiratory rate, applying a candidate specific filter to the HRV signal to obtain a respective filtered HRV signal, the passband of each filter being determined based on the respective candidate respiratory rate,(c) for each filtered HRV signal, determining an interim respiratory rate from the filtered HRV signal, and(d) determining an estimate for the respiratory rate based on the interim respiratory rates.

17. The computer-implemented method of claim 16 wherein determining the estimate for the respiratory rate includes: calculating a difference between each interim respiratory rate and the respective candidate respiratory rate, and identifying a smallest calculated difference, wherein the estimate for the respiratory rate is based on the candidate respiratory rate corresponding to the smallest calculated difference.

18. The computer-implemented method of claim 17 wherein the respiratory rate is the candidate respiratory rate corresponding to the smallest calculated difference.

19. The computer-implemented method of any of claims 16 to 18 further comprising: determining an initial respiratory rate from the HRV signal, wherein the list of candidate respiratory rates is determined based on the initial respiratory rate.

20. The computer-implemented method of any of claims 16 to 19 wherein the list of candidate respiratory rates spans a range from a minimum value (RR-mtn) to a maximum value (RRmax) at least one of which is a dependent on the initial respiratory rate.21 . The computer-implemented method claim 20 wherein the initial respiratory rate is determined by: performing a Fourier transform of the HRV signal, and identifying a highest amplitude peak in the transformed signal, the initial respiratory rate being given by the frequency of the highest amplitude peak.

22. The computer-implemented method claim 20 wherein the initial respiratory rate is determined by: determining a zero-crossing rate of the HRV signal in the time domain, the initial respiratory rate being given by the zero-crossing rate.

23. The computer-implemented method of any of claims 16 to 23 wherein each candidate specific filter is a bandpass filter, wherein a lower cut-off frequency of the bandpass filter corresponding to a nth candidate respiratory rate is given by:60and a higher cut-off frequency of the bandpass filter is given by:1.35 - BflJn h - 60 - where RR^ is the nth candidate respiratory rate (in BPM).

24. The computer-implemented method of any of claims 16 to 23 wherein determining each interim respiratory rate in step (c) includes: performing a Fourier transform of the respective filtered HRV signal, and identifying a highest amplitude peak in the transformed signal, the interim respiratory rate being given by the frequency of the highest amplitude peak.

25. The computer-implemented method of any of claims 16 to 24 wherein each interim respiratory rate determined in step (c) is determined from the filtered HRV signal in the frequency domain, and the computer-implemented method further includes:(i) for each filtered HRV signal, determining a second interim respiratory rate from the filtered HRV signal in the time domain;(ii) identifying a plurality of smallest calculated differences, each smallest calculated difference being a local minima of the calculated differences between the first interim respiratory rates and the candidate respiratory rates,(iii) for at least each candidate respiratory rate corresponding to the plurality of smallest calculated differences, determining a second difference between the candidate respiratory rate and the respective second interim respiratory rate,(iv) providing a candidate list of summed differences by: summing each second difference and the respective calculated difference, wherein determining the estimate for the respiratory rate comprises: identifying the smallest summed difference in the candidate list, and extracting the candidate respiratory rate corresponding to the smallest summed difference.

Images