Anti-motion-interference ear-wearing heart rate and breath real-time monitoring method and system

Abstract

The invention discloses an anti-motion interference ear-wearing heart rate and breath real-time monitoring method and system. The method comprises the following steps: for a target user, acquiring audio data and IMU data in an ear canal by using an ear-wearing wearable device; extracting heartbeat features and breathing features from the audio data; inputting the breathing characteristics into a breathing waveform reconstruction model to obtain a reconstructed breathing waveform; inputting the heartbeat features into a frequency spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrogram, and converting the reconstructed electrocardiogram spectrogram from frequency domain representation to time domain representation to obtain a time domain electrocardiogram waveform; preprocessing the IMU data to obtain an IMU thoracic cavity fluctuation waveform; and based on the time domain electrocardiogram waveform, the reconstructed respiration waveform and the IMU thoracic cavity fluctuation waveform, through peak-to-peak value detection, obtaining an estimation result of the respiration frequency and the heart rate. According to the method, the influence of motion interference can be effectively eliminated, and the estimation accuracy of heart rate respiration is improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and more specifically, to an ear-worn heart rate and breathing real-time monitoring method and system that is resistant to motion interference. Background Art

[0002] With the improvement of quality of life and the development of sensor technology, people are increasingly keen on using sports monitoring equipment to record their own exercise status. During exercise, heart rate and breathing rate are key indicators to guide the effectiveness of exercise. Under the guidance of these indicators, people can better choose the exercise that suits them, determine the time and intensity of their own exercise, and thus achieve better exercise and fitness effects under the guidance of scientific data.

[0003] In the prior art, the common heart rate monitoring sensor is the PPG (photoplethysmography) sensor, which generally needs to fit closely to the skin and has weak resistance to motion interference. The common respiratory monitoring sensor is the IMU (inertial measurement unit), which monitors breathing by monitoring the rise and fall of the chest. For example, patent application CN202110267923.3 provides a human respiratory rate monitoring device integrated in a mask, which is integrated with a mask using a relatively independently arranged breathing valve and a detection module. The breathing valve is connected to the breathing valve hole of the mask by snap-fitting, and the detection module is connected to the mask by gluing. The mask is replaceable, and the breathing valve and detection module can be easily disassembled and reused. However, this solution cannot achieve simultaneous monitoring of heart rate and respiratory rate, and the mask will burden the user's breathing during exercise.

[0004] Patent application CN201410630915.0 provides an ear-worn heart rate monitoring device and method. The monitoring device includes: a sensing unit, including a light source and a brightness sensor, which is arranged in the auricle of a human body, and is used to detect the pulse wave photoplethysmography signal of the human body and transmit it to an acquisition and processing unit; an acquisition and processing unit, which is used to collect and calculate the pulse wave photoplethysmography signal to obtain heart rate detection information; a data sending and receiving unit, which is used to send heart rate detection information and receive configuration information or control instructions; and a battery and power supply unit, which is used to provide power for the ear-worn heart rate monitoring device. This solution adopts an ear-worn wearing method, which blocks ambient light through the structure of the human ear itself, has no oppressive body contact, and is suitable for long-term wearing. However, this solution cannot realize the simultaneous monitoring of heart rate and respiratory rate in a state of motion, and uses the peak value of the digital signal sequence to obtain the heartbeat interval and then obtain the heart rate, which cannot resist motion interference.

[0005] In summary, the existing technology mainly has the following defects:

[0006] 1) Among existing sports monitoring solutions, wristband smart watches based on PPG are the first choice. However, body movement can cause the PPG sensor to shift relative to its original position, change the path of light in the sensor, and produce motion artifacts. In addition, wristband sports watches cannot monitor breathing during exercise.

[0007] 2) In the monitoring solution based on breathing belt and sports chest belt, RR (heart rate) is mainly monitored by wearing a breathing belt or sports chest belt on the abdomen. Breathing belt cannot effectively counteract the impact of exercise, and sports chest belt needs to be worn close to the skin, which is cumbersome to wear, easily stained with sweat, and difficult to clean, so it is not suitable for ordinary people.

[0008] 3) In the monitoring method of ear-worn devices, IMU-based respiratory monitoring infers the respiratory rate by capturing the weak head movements generated when a person breathes in a static state. However, this monitoring method is easily disturbed by large movements and cannot achieve effective monitoring in a moving state, or the audio signal processing method used is not robust enough to accurately detect the respiratory rate in a moving state. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an ear-worn heart rate and respiration real-time monitoring method and system that is resistant to motion interference.

[0010] According to a first aspect of the present invention, a method for real-time heart rate and respiration monitoring with ear-worn technology that is resistant to motion interference is provided. The method comprises:

[0011] For target users, use ear-worn wearable devices to collect audio data and IMU data in the ear canal;

[0012] Extracting heartbeat features and breathing features from the audio data;

[0013] Inputting the respiratory characteristics into a respiratory waveform reconstruction model to obtain a reconstructed respiratory waveform;

[0014] Inputting the heartbeat feature into a spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrum diagram, and converting the reconstructed electrocardiogram spectrum diagram from a frequency domain representation to a time domain representation to obtain a time domain electrocardiogram waveform;

[0015] Preprocessing the IMU data to obtain an IMU chest cavity fluctuation waveform;

[0016] Based on the time-domain electrocardiogram waveform, the reconstructed respiratory waveform and the IMU chest rise and fall waveform, the estimation results of the respiratory frequency and the heart rate are obtained through peak-to-peak value detection.

[0017] According to a second aspect of the present invention, there is provided an ear-worn heart rate and respiration real-time monitoring system resistant to motion interference, comprising an ear-worn wearable device and a terminal device, wherein the ear-worn wearable device is used to collect audio data and IMU data in the ear canal of a target user, and transmit the data to the terminal device; the terminal device executes: extracting heartbeat features and breathing features from the audio data; inputting the breathing features into a breathing waveform reconstruction model to obtain a reconstructed breathing waveform; inputting the heartbeat features into a spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrum diagram, and converting the reconstructed electrocardiogram spectrum diagram from a frequency domain representation to a time domain representation to obtain a time domain electrocardiogram waveform; preprocessing the IMU data to obtain an IMU chest cavity fluctuation waveform; and obtaining estimation results of the breathing frequency and heart rate through peak-to-peak value detection based on the time domain electrocardiogram waveform, the reconstructed breathing waveform and the IMU chest cavity fluctuation waveform.

[0018] Compared with the prior art, the advantages of the present invention are that the provided ear-worn heart rate and respiration real-time monitoring solution that is resistant to motion interference adopts the wearing form of ear-worn devices and uses a multi-sensor collaborative method to achieve simultaneous monitoring of heart rate and respiration rate, which can extract effective heartbeat information and respiration information and effectively eliminate the influence of motion interference. The wearable device of the present invention is in the form of ear-worn type, which is less invasive, easier to wear, and easier to clean than wearable clothing and chest straps.

[0019] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 is a flow chart of a method for real-time heart rate and respiration monitoring with ear-worn anti-motion interference according to an embodiment of the present invention;

[0022] Figure 2 2 is a schematic diagram of the overall framework of a method for real-time heart rate and respiration monitoring with ear-worn anti-motion interference according to an embodiment of the present invention;

[0023] Figure 3 is an overall framework diagram of an earhook headset according to an embodiment of the present invention;

[0024] Figure 4 is a structural diagram of an ear-hook device according to an embodiment of the present invention;

[0025] Figure 5 is an overall framework diagram of a neck-hanging headset according to an embodiment of the present invention;

[0026] Figure 6 is an appearance diagram of a neck-hanging headset according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a main control board of a neck-hanging headset according to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of the result after empirical wavelet decomposition according to an embodiment of the present invention;

[0029] Fig. 9 is a schematic diagram of an ECG spectrum mapping model according to an embodiment of the present invention;

[0030] Fig.10 is a schematic diagram of a respiratory waveform mapping model according to an embodiment of the present invention;

[0031] Fig.11 It is a schematic diagram of a user personal interface, a personal information drop-down interface, and a sports record interface according to an embodiment of the present invention;

[0032] Fig.12 It is a schematic diagram of an application homepage interface, a sports progress interface, a sports pause interface, and a sports end recording interface according to an embodiment of the present invention;

[0033] Fig.13 Schematic diagram of a heart rate too fast warning interface, a breathing too fast warning interface, and an exercise intensity prompt interface according to an embodiment of the present invention;

[0034] Fig.14 is a schematic diagram of overall performance of a user according to an embodiment of the present invention;

[0035] Fig.15 is a schematic diagram of an empirical cumulative distribution function according to an embodiment of the present invention;

[0036] Fig.16 is a schematic diagram of the results of long-term tracking of heart rate and respiration according to one embodiment of the present invention;

[0037] In the attached figure, Max Pool-maximum pooling; Conv-convolution; Global Average Pooling-global average pooling; Fully Connected Layer-fully connected layer; Transposed Conv-transposed convolution; Conv Block-convolution block. DETAILED DESCRIPTION

[0038] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] In general, the present invention first designs an ear-worn wearable device to collect the user's audio data and IMU data. After obtaining the user data, the basic input of the deep learning model is obtained through signal preprocessing. Finally, the reconstructed ECG (electrocardiogram) curve and respiratory waveform curve are obtained through the model, and then the heart rate extraction algorithm and respiratory rate extraction algorithm are applied to obtain the estimated results of heart rate and respiratory rate.

[0044] Specifically, combined Figure 1 and Figure 2 As shown, the provided ear-worn heart rate and respiration real-time monitoring method with resistance to motion interference comprises the following steps:

[0045] Step S110, designing an ear-worn wearable device for collecting audio data and IMU data of the user.

[0046] To adapt to different wearing habits of users, ear-worn wearable devices with different configurations can be designed. For example, two types of wearable devices, ear-hook and neck-hook, are designed. The ear-hook device contains an in-ear microphone and an ear IMU, and the neck-hook device contains an in-ear microphone and a neck IMU. When worn, the neck IMU is close to the chest cavity, and the ear IMU is located on the user's head. Both sets of devices can obtain audio data and IMU data in the ear canal.

[0047] (1) Ear-hook wearable devices

[0048] In one embodiment, the ear-hook wearable device is an ear-hook headset. The device includes an ESP32 microcontroller, several sensors, and a self-designed 3D support shell. Figure 3 The overall framework of the ear-hook headset is shown, in which the connection and communication methods of the ESP32 microcontroller with multiple sensors and a mobile terminal are as follows.

[0049] The ESP32 microcontroller and the MEMS (micro-electromechanical system) microphone sensor are connected using the I2S digital audio bus. ESP32 obtains the audio data collected by the microphone through the SCK (Serial Clock), WS (Word Select), and SD (Serial Data) pins.

[0050] The ESP32 microcontroller and the IMU sensor are connected using the I2C bus. ESP32 communicates with the IMU sensor via the SCL (SerialClock) and SDA (SerialData) pins to read the acceleration, gyroscope (and magnetometer) data.

[0051] A wireless connection is used between the ESP32 microcontroller and the mobile terminal. For example, using a wireless connection based on Wi-Fi, the ESP32 microcontroller continuously sends audio data to the mobile terminal through the HTTP protocol.

[0052] The structure of the earhook headset is as follows Figure 4 As shown, the device is equipped with a miniature microphone sensor at the ear entrance, which can continuously collect audio signals in the user's ear canal. At the same time, the IMU sensor in the device can also continuously collect the user's body posture information, and then transmit various data to the mobile terminal device via Wi-Fi for subsequent heart rate and respiratory rate detection.

[0053] (2) Neck-hanging wearable devices

[0054] In one embodiment, the neck-hanging (or hanging) wearable device is a neck-hanging headset. The neck-hanging headset includes a neck-hanging ring, an earplug connected to the neck-hanging ring, a main control board circuit, a battery unit, an IMU sensor, and a microphone sensor. The microphone sensor is arranged in the earplug, and the main control circuit board is arranged at one end of the neck-hanging ring. The IMU sensor and the battery unit are arranged at the other end of the neck-hanging ring. Figure 5 As shown in the figure, the device includes an ESP32 microcontroller and several sensors. The connection and communication method between the ESP32 microcontroller, multiple sensors and a mobile terminal is as follows.

[0055] The ESP32 microcontroller and the MEMS microphone sensor are connected using the I2S digital audio bus. ESP32 obtains the audio data collected by the microphone through the SCK (Serial Clock), WS (Word Select), and SD (Serial Data) pins.

[0056] The ESP32 microcontroller and the IMU sensor are connected using the I2C bus. ESP32 communicates with the IMU sensor bidirectionally through the SCL (SerialClock) and SDA (SerialData) pins to read the acceleration, gyroscope, and magnetometer data.

[0057] A wireless connection is used between the ESP32 microcontroller and the mobile terminal. For example, using a wireless connection based on Wi-Fi, the ESP32 microcontroller continuously sends audio data to the mobile terminal through the HTTP protocol.

[0058] The overall appearance of the neckband headset is as follows Figure 6 As shown, the main control board structure is as follows Figure 7 As shown, there is a miniature microphone sensor at the ear of the device, which can continuously collect audio signals in the user's ear canal. At the same time, the IMU sensor in the device can also continuously collect the user's body posture information, and then transmit various data to the mobile terminal device via Wi-Fi for subsequent heart rate and respiratory rate detection.

[0059] It should be understood that the ear-worn device designed by the present invention mainly uses an in-ear microphone and an IMU to collect user data and transmit it to a mobile device for calculation. In addition to the ear-hook and neck-hook types, other forms can also be used, for example, using multiple devices to obtain a microphone and an IMU respectively. In addition, the earplug and the neck hanging ring can be connected wirelessly or with a data cable. The wireless ear-worn design ensures stable wearing and stable signal transmission, thereby reducing interference during exercise.

[0060] Step S120, processing the collected audio data to extract heartbeat features and breathing features.

[0061] There are multiple sound sources aliasing in the audio signal. Signal decomposition methods can use empirical mode decomposition (EMD) and discrete wavelet transform (DWT) and the like. Considering that EMD cannot effectively separate information of different modes and is prone to modal aliasing, DWT uses a fixed ratio of sub-bands to divide the frequency and lacks the ability of adaptive filtering. Therefore, in one embodiment, empirical wavelet transform (EWT) is used to adaptively select the frequency band to overcome the modal aliasing problem caused by the discontinuity of the signal time-frequency scale.

[0062] Specifically, after obtaining the audio data, we first use empirical wavelet decomposition to obtain data in different frequency bands. For the signal f(t), we obtain the normalized FFT spectrum of the audio, search for M maximum points of the spectrum, and sort them in descending order. Then, we use the minimum point between two adjacent maximum points as the boundary ω. n To divide the spectrum, where ω0 = 0, ω n =π, each segment can be expressed as Λ n =[ω n-1 ,ω n ]. n As the center, define a converter T n , with a width of 2τ n Using EWT, the signal is decomposed into approximate coefficients and detail coefficients. The approximate coefficients usually only contain some trends of the signal, which is not useful for HR (heart rate) and RR (respiratory rate) estimation. Therefore, for Given the following empirical wavelet function:

[0063]

[0064] in:

[0065] β(x)=x 4 (35-84x+70x 2 -20x 3 )(2)

[0066] EWT can be viewed as a filter bank, with detail coefficients It is obtained by the inner product with the empirical wavelet:

[0067]

[0068] Therefore, the empirical coefficient is:

[0069]

[0070] like Figure 8As shown in the figure, after EWT processing, the corresponding component coefficients are selected according to the specific frequency band. The components in the 1-3Hz frequency band contain the heartbeat period (Heartbeart period coefficients), which is a sine-like curve, and can be used to determine the rough position of the heartbeat. The components in the 3-20Hz frequency band contain the harmonic information of the heartbeat sound (Heartbeartharmonic coefficients). This curve has more details, which is convenient for supplementing details when reconstructing ECG. The sampling rate is set to 10KHz, and the last detail coefficient is the breathing sound coefficient (breathing sound coefficients), which contains information from 20 to 5000Hz, covering the frequency distribution of breathing sound, so it can be used as input for subsequent breathing frequency extraction.

[0071] Although EWT can extract the harmonic information of the heartbeat sound, it is still disturbed by some noise. Therefore, the extraction of the heartbeat is further realized through deep learning methods. For example, the original audio data, the period of the heartbeat, and the heartbeat harmonic coefficients are downsampled to 1kHz, and then they and the respiratory sound coefficients are converted to Mel Spectrogram and the logarithm is taken. For example, the fast Fourier transform window lengths of the heartbeat and respiratory signals are 128 and 2048 respectively, the short-time Fourier transform window lengths are 128 and 2048 respectively, the window overlap lengths are 51 and 256 respectively, and the number of Mel filters is 64 and 128 respectively. Then the first three are stacked as the input of the heart rate estimation model, and the remaining one is used as the input of the respiratory estimation model. The respiratory estimation model refers to the respiratory wave reconstruction model that includes the spatial feature extraction module and the temporal feature extraction module in the following text.

[0072] Step S130, processing the collected IMU data to obtain a chest cavity fluctuation waveform reflecting the user's movement information.

[0073] IMU signals are used to estimate respiratory rate. Since the sound of human breathing is weak at rest, the sound collected by the microphone in the ear is very weak, which can easily lead to poor respiratory rate estimation. Therefore, IMU can be used to capture subtle movements of human breathing, such as chest heave and head movement caused by breathing, as an additional respiratory rate estimation signal.

[0074] The human breathing frequency is mainly in the range of 10 to 40 BPM, so the IMU signal can be bandpass filtered or wavelet decomposition can be performed. For example, the bandpass filter range is 0.2 to 0.8 Hz. After filtering, the IMU signal filters out most of the motion noise and random noise, and retains the low-frequency breathing information.

[0075] It should be understood that the pre-processing of audio data and IMU data can adopt various signal processing methods such as wavelet transform, empirical mode decomposition, bandpass filtering, etc., as long as the information containing the frequency of heartbeat and breathing information can be extracted.

[0076] Step S140, inputting the heartbeat features into a spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrum diagram.

[0077] In the signal preprocessing, a three-channel stacked heart rate estimation input spectrum is obtained. In one embodiment, an ECG spectrum reconstruction model is designed to map the multi-channel MEL spectrogram to the STFT spectrogram of ECG and eliminate noise in the audio spectrogram. Fig. 9 It is an ECG spectrum mapping model, which uses UNet as the base model and combines the channel attention mechanism to effectively extract features from the spectrum. In the encoder, the feature map is extracted by the channel attention mechanism after repeated 3x3 convolution, LeakReLU activation function, and normalization layer. For example, SENet is used to compress the spatial features of each channel into a scalar after global average pooling, and after passing through two fully connected layers, the channel attention weight is obtained by the Sigmoid function. Using the channel attention mechanism, the model adaptively selects effective feature channels from the original spectrum and EWT additional features, which improves the model performance. In the decoder, the data passes through continuous upconvolution blocks, halving the number of feature maps, and after each upconvolution, the feature map is merged with the feature map corresponding to the encoder, and then passes through the convolution block and batch normalization. In the last layer, a 1x1 convolution is used to map the feature map to the spectrum of a single output channel.

[0078] Step S150, inputting the respiratory characteristics into the respiratory waveform reconstruction model to obtain a reconstructed respiratory waveform.

[0079] The distribution of breathing sounds varies depending on individual characteristics and physical activities, and is often masked by background noise. Therefore, a breathing curve reconstruction model (or breathing curve mapping model) is designed to effectively reconstruct the breathing waveform from the audio spectrogram, such as Fig.10 As shown, the respiratory waveform reconstruction model includes a spatial feature extraction module (SpatialFeatureExtraction) and a temporal feature extraction module (TemporalFeatureExtraction).

[0080] (1) Spatial feature extraction module

[0081] Log-Mel Spectrogram first passes through a layer of convolutional module to obtain shallow feature information. The structure of the convolutional module is similar to Fig. 9The same as in . Then it passes through two layers of deformable convolution (DCN) modules to extract the deep feature representation of breathing sound. Given a convolution kernel with K sampling positions, w k and p k They represent the weight and predefined offset of the kth position respectively. 3x3 convolution means K=9. x(p) and y(p) represent the input feature map and output feature map at position p respectively. The formula of deformable convolution is as follows:

[0082]

[0083] Among them, Δp k is the learnable offset of the kth position, Δp k is a real number with an unlimited range of values. k Obtained by applying a single convolutional layer on the same input feature map, with an output of 2K channels and kernel weights initialized to 0. The learning rate of this single convolutional layer is set to 0.1 times that of other regular layers. After two layers of DCN, the output is connected to the upconvolution block. The structure of the upconvolution block is similar to that of the heart rate estimation model. The difference is that this upconvolution block only expands the time dimension and continuously halves the frequency bins, finally obtaining an intermediate feature with a frequency bin dimension of 1, 512 channels, and a time length of 250. This feature is then fed into the timing information capture module.

[0084] (2) Temporal feature extraction module

[0085] A complete breath consists of two phases: inhalation and exhalation. In the real breathing curve, they are represented by the rising edge and falling edge of the curve respectively, and in the audio time-frequency domain, they are represented by two continuous energy concentration areas. The sounds of inhalation and exhalation are very similar, and convolution cannot reconstruct the phase of the breathing curve very well, that is, the model mistakenly reconstructs inhalation and exhalation as the falling edge and rising edge of the breathing curve. Bidirectional LSTM can be used to capture the long-term relationship between breathing, so that the model can determine whether the current position is inhalation or exhalation based on context information. The number of layers of Bi-LSTM is 8, and the dimension of the LSTM hidden layer is 256. Connect the output to a fully connected layer and reduce the output channel of LSTM to 1 dimension to obtain the reconstructed breathing curve.

[0086] In summary, the present invention considers the recovery of respiratory waveform signals. The respiratory rate is usually obtained from the fluctuation of the respiratory waveform, which reflects the fluctuation of the chest breathing and thus reflects the user's breathing condition. The present invention can restore audio data and IMU data into a real and effective respiratory waveform signal.

[0087] Step S160 , estimating the user's breathing frequency and heart rate using the reconstructed electrocardiogram spectrogram and breathing waveform.

[0088] Figure 2 The model training or reasoning can utilize the respiratory waveform and ECG collected by the sports chest strap in the prior art to train the spectrum reconstruction model and the respiratory waveform reconstruction model.

[0089] After model inference, the reconstructed ECG spectrogram and respiratory waveform are obtained. Due to the lack of phase information in the reconstructed ECG spectrogram, the Griffin-Lim algorithm is used to convert the frequency domain representation back to the time domain. A sliding window with a length of 10 seconds and a step size of 5 seconds is applied to segment the reconstructed ECG and respiratory curves. Peak-to-peak detection is used for each window to calculate the time interval between peaks. The z-scores of these intervals are then calculated and outliers are filtered out. The HR (heart rate) and RR of each window are determined by the average of the time intervals after filtering out outliers. Finally, a moving average filter is applied to smooth the HR and RR curves.

[0090] For the pre-processed IMU signal, the amplitude is first monitored. If it exceeds a certain amplitude, it is judged that the user is in motion. At this time, the IMU signal noise is too much, so the IMU signal estimation is discarded. If it does not exceed a certain value, it is judged that the user is in a resting state, and the RR of the filtered IMU signal is calculated and averaged with the RR value calculated by the audio as the final RR value.

[0091] Furthermore, the estimated results of HR and RR are displayed on the user's device, such as a mobile phone, tablet computer, etc. In terms of software, a convenient and easy-to-use mobile application software has been developed. By connecting to the smart ear-worn device, the user can view the physiological index data during exercise in real time, see Figure 11-13 This application software has an intuitive and user-friendly interface to ensure that users can easily obtain the information they need. Users can get personalized sports health advice and guidance based on their personal physical condition and exercise records, so as to conduct sports training more scientifically.

[0092] See also Fig.11 As shown in the figure, the user's personal homepage interface displays the individual's physical health status and exercise records. Each exercise record includes the type, duration and date of the exercise, as well as physiological data related to the exercise process, such as heart rate changes, breathing rate changes, etc. Such records help users understand their exercise performance and health status, and provide a reference for formulating future exercise plans.

[0093] See also Fig.12As shown in the figure, on the homepage of the app, users can select the type and duration of exercise they need to do, and obtain real-time heart rate, breathing rate and other data through smart Bluetooth headphones. On the page during exercise, users can learn about the current exercise duration, heart rate, breathing rate and heart rate variability in real time. In addition, users can pause exercise recording at any time and save the record at the end.

[0094] See also Fig.13 As shown, by real-time monitoring of the user's heart rate and breathing rate, it is possible to determine the user's exercise stage, such as the warm-up or fat burning stage, and issue a warning according to the situation. For example, if the heart rate is too high or the breathing is too rapid, the user is warned and advised to adjust or end the exercise in time. Such a function helps users better understand their physical condition during exercise and avoid excessive exercise or health risks. Overall, the present invention can provide users with a full range of exercise monitoring and guidance services, allowing them to exercise more safely and scientifically and achieve better fitness results.

[0095] In order to further verify the effect of the present invention, experiments were carried out.

[0096] Fig.14 The MAE of HR and RR estimates for different users and activities is given, where Fig.14 (a) is the estimate of HR, Fig.14(b) is the RR estimation. Due to device placement issues, the data of user 4 and user 9 when cycling and walking were excluded. The following findings can be drawn from the results. First, there is no correlation between HR and RR estimates. For HR estimation, the average MAE of users U2~U5, U11 and U12 exceeds 6BPM. But their RR estimation is quite accurate, with an average MAE of less than 2.26RPM. This difference is mainly caused by the different physical characteristics of heartbeat and breathing. The masking effect significantly amplifies the low-frequency heartbeat sound, while the effect on the high-frequency breathing sound is small. Therefore, the reduced bite caused by improper earplug closure has a more serious impact on the heartbeat sound, resulting in a higher MAE for heart rate estimation, while the performance of RR estimation remains stable. Second, even the same activity has different effects on HR and RR estimation. The present invention provides excellent HR estimation at rest, with an average MAE of 0.93BPM, but the RR estimation is poor. This is due to the reduced breathing intensity at rest, which makes breathing difficult to distinguish from background noise. In contrast, the present invention has good RR estimation, with an average MAE of 1.44RPM, but poor HR estimation in rowing. This is due to the synchronization of breathing with the rowing motion, which stabilizes the RR, while the body noise generated by rowing masks the low-frequency heartbeat, compromising the HR measurement as described in Section 2. When running, the present invention shows the highest MAE for both heart rate and RR due to the strong footsteps that mask the heartbeat. In addition, the body movement during running causes the hardware to vibrate, which creates additional interference and masks the breathing sound.

[0097] Next, the empirical cumulative distribution functions (ECDFs) of the HR and RR estimates were also calculated, see Fig.15 As shown, Fig.15 (a) is the ECDF of HR, Fig.15 (b) is the ECDF of RR. It can be seen that in the worst case, the absolute errors of HR and RR estimation of about 60% of the samples are less than 12.5BPM and 2.5RPM respectively.

[0098] Furthermore, among the activities considered, running and rowing had a significant effect on HR estimates, while resting and cycling had the smallest effects. For RR estimates, rowing and cycling had the smallest effects compared to the other three activities.

[0099] The long-term tracking performance was evaluated while users performed various aerobic activities under real-world conditions. Users could row, bike, run, and walk in a gym, with rest periods between activities depending on their activity level. Fig.16 The long-term tracking results shown in Fig.16 (a) is the long-term tracking of HR estimation, Fig.16(b) is the long-term tracking of RR. The average MAE of HR and RR estimates are 3.46 BPM and 2.40 RPM, and the MAPE values ​​are 2.78 and 12.89, respectively. The MAE of HR estimation in each activity stage is 4.00, 3.47, 3.93, and 3.44 BPM, while the MAE of RR estimation is 1.35, 0.88, 4.24, and 2.50 RPM. HR performs well. Although some fluctuations are observed in RR estimation, the overall trend is acceptable.

[0100] It should be noted that, without violating the spirit and scope of the present invention, those skilled in the art may make appropriate changes or modifications to the above embodiments. For example, the number of input channels of the U-Net spectrum mapping model can be set according to actual needs and is not limited to 3-channel input. The reconstruction model can be a spectrum-to-spectrum mapping model, or a spectrum-to-curve mapping model. The specific structure of the model, such as the number of convolution blocks, activation function, convolution kernel size, etc., can be flexibly set.

[0101] In summary, compared with the prior art, the present invention has the following advantages:

[0102] 1) The ear-worn motion-noise-resistant heart rate and respiratory rate monitoring solution proposed in the present invention utilizes a microphone sensor and an IMU sensor to collect heart sounds and respiratory sounds in the user's ear canal, and body movement information of the head or neck caused by breathing. It also combines signal processing and deep learning technology to suppress various noises introduced by motion, reconstruct ECG curves and respiratory curves, and ultimately obtain estimation results of heart rate and respiratory rate, thereby improving the accuracy of respiratory rate estimation.

[0103] 2) In signal processing, the present invention utilizes a variety of different signal processing methods, such as empirical mode decomposition, discrete wavelet transform, and bandpass filtering, to obtain information on heartbeat and breathing in different frequency bands in the audio, and converts them into spectrograms respectively, and then uses a deep learning model to map them into ECG spectrum and respiratory waveform respectively, and then uses an inverse short-time Fourier transform to invert the ECG spectrum into an ECG waveform, and finally uses a motion peak monitoring algorithm to calculate the heart rate and respiratory rate.

[0104] 3) The present invention designs two different sets of hardware devices, namely ear-hook type and neck-hook type devices, which are suitable for the wearing habits of different users, and are conducive to users choosing their favorite wearing style. Compared with the existing wearing method, it is more convenient and less intrusive to the user. The ear-hook type has a compact structure design, small size, and is easy to wear. In addition, the ear-worn devices have various forms and can better meet the needs of multiple users. The neck-hook type is more comfortable to wear and the data quality obtained is better.

[0105] 4) Compared with sports watches, the present invention can monitor more physiological indicators, including not only heart rate but also respiratory rate, and can combine multiple sensors to achieve more accurate respiratory monitoring. Compared with sports chest straps, the present invention does not need to contact the skin, is less invasive, easier to clean and maintain, and more convenient to wear.

[0106] 5) During exercise, body movement, clothing friction, footsteps, external environmental noise, etc., will bring a lot of interference signals. The present invention can effectively eliminate the influence of motion interference and extract effective heartbeat information and breathing information.

[0107] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0108] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0109] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0110] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0111] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0113] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0114] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0115] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for real-time heart rate and respiration monitoring by ear-wearing with resistance to motion interference, comprising: For target users, use ear-worn wearable devices to collect audio data and IMU data in the ear canal; Extracting heartbeat features and breathing features from the audio data; Inputting the respiratory characteristics into a respiratory waveform reconstruction model to obtain a reconstructed respiratory waveform; Inputting the heartbeat feature into a spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrum diagram, and converting the reconstructed electrocardiogram spectrum diagram from a frequency domain representation to a time domain representation to obtain a time domain electrocardiogram waveform; Preprocessing the IMU data to obtain an IMU chest cavity fluctuation waveform; Based on the time-domain electrocardiogram waveform, the reconstructed respiratory waveform and the IMU chest rise and fall waveform, the estimation results of the respiratory frequency and the heart rate are obtained through peak-to-peak value detection.

2. The method according to claim 1, characterized in that The breathing feature and the heartbeat feature are obtained according to the following steps: Taking a logarithmic Mel-spectrogram of the audio data to obtain a heartbeat feature of a first channel; Decomposing the audio data by wavelet to obtain heartbeat cycle coefficients, heartbeat harmonic coefficients and breathing sound coefficients; For the heartbeat cycle coefficient and the heartbeat harmonic coefficient, respectively, logarithmic Mel spectrograms are obtained to obtain the heartbeat feature of the second channel and the heartbeat feature of the third channel, and the heartbeat features are combined with the heartbeat feature of the first channel to obtain the heartbeat feature; The breathing sound coefficient is subjected to a pre-emphasized logarithmic Mel-spectrogram to obtain the breathing feature.

3. The method according to claim 1, characterized in that The spectrum reconstruction model is built based on the UNet model. In the encoder, the feature map is extracted by the channel attention mechanism after repeated convolution, LeakReLU activation function and normalization layer. In the decoder, the data passes through continuous upconvolution blocks to halve the number of feature maps. After each upconvolution, the feature map is merged with the feature map corresponding to the encoder, and then passes through the convolution block and batch normalization. In the last layer, 1x1 convolution is used to map the feature map to the spectrum of a single output channel.

4. The method according to claim 1, characterized in that: The respiratory curve reconstruction model includes a time feature extraction module and a spatial feature extraction module. The time feature extraction module includes a bidirectional long short-term memory network and a fully connected layer. The bidirectional long short-term memory network is used to capture the long-term relationship between breathing. In the fully connected layer, the output channel of the bidirectional long short-term memory network is reduced to 1 dimension to obtain the reconstructed respiratory waveform; the spatial feature extraction module includes a convolution layer and a deformable convolution module. The convolution layer is used to extract shallow feature information of breathing sound, and the deformable convolution module is used to extract deep features of breathing sound. For a given convolution kernel with K sampling positions, the formula of deformable convolution is: Among them, Δp k is the learnable offset of the kth position, w k and p k denote the weight and predefined offset of the kth position respectively, and x(p) and y(p) denote the input feature map and output feature map at position p respectively.

5. The method according to claim 2, characterized in that: Decomposing the audio data by wavelet to obtain the heartbeat cycle coefficient, heartbeat harmonic coefficient and breathing sound coefficient comprises the following steps: For the audio data, using empirical wavelet decomposition to obtain data of different frequency bands; The corresponding component coefficient is selected according to the set frequency band, the heartbeat cycle coefficient is determined by the components of the 1-3 Hz frequency band, the heartbeat harmonic coefficient is determined by the components of the 3-20 Hz frequency band, and the breathing sound coefficient is determined by the information of 20-5000 Hz.

6. The method according to claim 1, characterized in that The preprocessing of the IMU data includes: performing wavelet decomposition or bandpass filtering on the IMU data to filter out motion noise and random noise, and obtain the IMU chest cavity fluctuation waveform.

7. The method according to claim 1, characterized in that The ear-worn wearable device is an ear-hook headset, which includes an ear hook, an earplug, a main control board, an IMU sensor and a microphone sensor, and the microphone sensor is arranged at the ear inlet of the earplug.

8. The method according to claim 1, characterized in that: The ear-worn wearable device is a neck-hanging headset, which includes a neck hanging loop, an earplug connected to the neck hanging loop, a main control board circuit, a battery unit, an IMU sensor and a microphone sensor, wherein the microphone sensor is arranged in the earplug, the main control circuit board is arranged at one end of the neck hanging loop, and the IMU sensor and the battery unit are arranged at the other end of the neck hanging loop.

9. An ear-worn heart rate and breathing real-time monitoring system with anti-motion interference, comprising an ear-worn wearable device and a terminal device, wherein: The ear-worn wearable device is used to collect audio data and IMU data in the ear canal of a target user and transmit the data to the terminal device; the terminal device executes: extracting heartbeat features and breathing features from the audio data; Inputting the respiratory characteristics into a respiratory waveform reconstruction model to obtain a reconstructed respiratory waveform; The heartbeat features are input into a spectrum reconstruction model to obtain a reconstructed electrocardiogram spectrum diagram, and the reconstructed electrocardiogram spectrum diagram is converted from a frequency domain representation to a time domain representation to obtain a time domain electrocardiogram waveform; the IMU data is preprocessed to obtain an IMU chest fluctuation waveform; based on the time domain electrocardiogram waveform, the reconstructed respiratory waveform and the IMU chest fluctuation waveform, the estimation results of the respiratory frequency and heart rate are obtained through peak-to-peak value detection.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images