Internet of things based method and system for monitoring and early warning of respiratory state of chronic obstructive pulmonary disease patient

Abstract

The application discloses a kind of based on slow obstructive pulmonary patient respiratory state monitoring and early warning method and system of Internet of Things, belong to medical Internet of Things and speech signal processing technical field. Including: obtaining the continuous speech signal generated in the process of free conversation of target patient;Determine a plurality of exhalation support speech units and the speech pause between adjacent units according to continuous speech signal;Determine the respiratory support mode parameter according to exhalation support speech unit and speech pause, including average exhalation support speech duration, sound time ratio and speech pause statistical value;According to respiratory support mode parameter and individualized baseline parameter containing time period specificity baseline and day type baseline, determine whether the respiratory state of target patient is abnormal. Through the application, the problem that existing speech monitoring scheme cannot utilize respiratory-speech coupling time structure for non-sense respiratory state evaluation is solved.

Description

Technical Field

[0001] This invention relates to a method and system for monitoring and early warning of respiratory status in COPD patients based on the Internet of Things (IoT), belonging to the technical fields of medical IoT and voice signal processing. Background Technology

[0002] Chronic obstructive pulmonary disease (COPD) is a common chronic respiratory disease. Daily monitoring of the disease and early warning of acute exacerbations are of great significance for improving patient prognosis and reducing mortality.

[0003] Currently, home monitoring of COPD mainly relies on portable pulmonary function meters or wearable physiological monitoring devices. However, these devices usually require patients to actively cooperate in performing specific forced expiratory tests or wear them for extended periods, resulting in poor patient compliance and making long-term continuous monitoring difficult. In recent years, some studies have attempted to use speech signals for respiratory status assessment. The main approach is to extract the acoustic features of speech (such as fundamental frequency, energy, and spectral characteristics) to identify specific breath sounds, cough sounds, or analyze changes in speech quality caused by long-term disease.

[0004] However, most existing voice-based monitoring schemes ignore the inherent physical coupling between speech and respiratory airflow when humans speak naturally. When people are talking freely, the airflow support capacity of a single exhalation directly determines the duration of continuous speech, while silence between sentences often reflects the physiological need for ventilation.

[0005] Existing technologies have failed to effectively extract and utilize the breath-speech coupling temporal structure features in everyday free conversation, making it impossible to accurately and objectively quantify and assess a patient's true respiratory support capacity and changes in respiratory status in everyday communication scenarios where the patient is unaware of their surroundings.

[0006] Therefore, a new solution is needed to address this problem. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and system for monitoring and early warning of respiratory status in COPD patients based on the Internet of Things, which solves the problem that existing voice monitoring schemes cannot use the respiratory-voice coupling time structure to assess respiratory status without being noticed.

[0008] The technical problem to be solved by this invention is achieved by the following technical solution:

[0009] A method for monitoring and early warning of respiratory status in COPD patients based on the Internet of Things, characterized by comprising:

[0010] Acquire continuous speech signals generated by the target patient during free conversation;

[0011] Based on the continuous speech signal, determine multiple expiratory support speech units of the target patient during free conversation, as well as the speech intervals between adjacent expiratory support speech units;

[0012] Based on the plurality of expiratory support voice units and the voice intervals, the respiratory support mode parameters of the target patient are determined, wherein the respiratory support mode parameters include at least one of the following: average expiratory support voice duration, voice time ratio, and voice interval statistics.

[0013] Based on the respiratory support mode parameters and the individualized baseline parameters of the target patient, it is determined whether the respiratory status of the target patient is abnormal. The individualized baseline parameters include time-specific baselines and day-type baselines.

[0014] The present invention is further configured such that: determining multiple expiratory support speech units of the target patient during free conversation based on the continuous speech signal includes:

[0015] Detect voiced segments in the continuous speech signal and extract the fundamental frequency of the voiced segments;

[0016] When the baseband interruption is detected, the continuous speech signal is detected to have a decanted sound feature during the baseband interruption. The decanted sound feature includes a first type of decanted sound feature and a second type of decanted sound feature. The first type of decanted sound feature is a high-frequency friction energy feature of a first frequency band with a duration greater than or equal to a second threshold and less than or equal to a first threshold.

[0017] The second type of unvoiced sound feature is an energy surge that lasts longer than the fifth threshold and less than the second threshold, exceeding the preset decibel value of the ambient noise floor and accompanied by a sudden change in the zero-crossing rate.

[0018] If the unvoiced sound feature exists during the fundamental frequency interruption and fundamental frequency recovery is detected, the fundamental frequency interruption is marked as a voiceless consonant bridging segment, and the voiced sound segments before and after the fundamental frequency interruption are connected into the same expiratory support speech unit.

[0019] If the unvoiced sound feature is not present during the fundamental frequency interruption, and the duration of the fundamental frequency interruption is greater than or equal to the third threshold, the fundamental frequency interruption is marked as a breathing pause boundary, and the voiced sound segments before and after the fundamental frequency interruption are respectively determined as independent expiratory support speech units; if the duration of the fundamental frequency interruption is less than the third threshold, the fundamental frequency interruption is marked as environmental noise interference and ignored.

[0020] The present invention is further configured such that: the determination of multiple expiratory support speech units of the target patient during free conversation further includes:

[0021] For each of the expiratory support speech units, extract the voiceprint embedding vector;

[0022] Calculate the cosine similarity between the voiceprint embedding vector and the baseline voiceprint embedding vector of the target patient;

[0023] The exhalation support speech units whose cosine similarity is less than a preset threshold are marked as non-target speech units and discarded.

[0024] The exhalation support speech units whose cosine similarity is greater than or equal to the preset threshold are marked as target speech units;

[0025] When there is a discarded non-target speech unit between two adjacent target speech units, the time period occupied by the non-target speech unit is marked as the interactive listening period, and the interactive listening period is excluded from the calculation of subsequent breathing support mode parameters.

[0026] The present invention is further configured such that: determining the respiratory support mode parameters of the target patient based on the plurality of expiratory support voice units and the voice intervals includes:

[0027] Set the timeout threshold for each individual's speaking turn;

[0028] When the silence duration between two adjacent target speech units is greater than or equal to the timeout threshold, the silence duration is marked as a non-verbal resting state, and the current speaking round of the target patient is determined to have ended. The silence duration is not included in the calculation of physiological apnea.

[0029] When the silence duration between two adjacent target speech units is less than the timeout threshold, the silence duration is marked as a physiological apnea.

[0030] Within a single consecutive speaking round of the target patient, the total effective vocalization duration of the target speech unit and the total duration of the physiological apnea are obtained;

[0031] The vocalization time ratio is determined based on the total effective vocalization time and the total physiological apnea time. The vocalization time ratio is the total effective vocalization time divided by the sum of the total effective vocalization time and the total physiological apnea time. The vocalization time ratio represents the proportion of the target patient's effective vocalization time in a single consecutive speaking round to the total time of the single consecutive speaking round.

[0032] The present invention is further configured such that: the statistical value of the speech intermittent includes the median of physiological ventilation pauses; the step of determining the respiratory support mode parameters of the target patient based on the plurality of expiratory support speech units and the speech intermittent further includes:

[0033] The phonation duration of multiple target speech units is obtained, and the average expiratory support speech duration is determined.

[0034] Obtain the silence duration marked as the physiological apnea, and determine the median of the physiological apnea.

[0035] The present invention is further configured such that the individualized baseline parameters are established in the following manner:

[0036] During the preset learning period, respiratory support mode parameter samples of the target patients were collected at different time periods and on different days.

[0037] The statistical distribution of respiratory support mode parameters for each time period and each day type is determined based on the sample, and the individualized baseline parameters are defined by the median and interquartile range of the statistical distribution.

[0038] The present invention is further configured to include:

[0039] Obtain the pitch variability and energy dynamic range of the target speech unit;

[0040] When the pitch variability and the energy dynamic range both exceed the individualized high arousal threshold of the target patient, the respiratory-associated acoustic signal corresponding to the target speech unit is detected. The individualized high arousal threshold is determined based on the statistical analysis of the historical speech features of the target patient in a stable state.

[0041] The respiratory-associated acoustic signal is processed by a higher-order cumulant to obtain a target signal after suppressing Gaussian noise, and the quasi-periodic component of the second frequency band is extracted from the target signal.

[0042] When the duration of the quasi-periodic component is greater than or equal to the fourth threshold, the respiratory acoustic signal is determined to contain wheezing characteristics; or, when a high-frequency inspiratory noise with a duration of less than 200 milliseconds is detected, the respiratory acoustic signal is determined to contain rapid inspiratory sound characteristics.

[0043] If the respiratory acoustic signal contains at least one of the wheezing feature or the rapid inhalation feature, the target speech unit is marked as a pathological panic speech unit; if the respiratory acoustic signal does not contain the wheezing feature or the rapid inhalation feature, the target speech unit is marked as an emotional interference speech unit and removed.

[0044] The present invention is further configured to include:

[0045] When the number of the exhalation support voice units collected within a preset time period is lower than a preset threshold, a voice guidance task is initiated to the target patient through the associated smart speaker to obtain standardized voice samples as the evaluation benchmark for the day.

[0046] Alternatively, during periods without speech activity, cough events and wheezing sounds can be monitored, and the frequency of coughs and the total duration of wheezing can be counted as co-validation indicators for the parameters of the respiratory support mode.

[0047] A respiratory status monitoring and early warning system for COPD patients based on the Internet of Things, characterized in that it includes:

[0048] The speech signal acquisition module is used to acquire continuous speech signals generated by the target patient during free conversation;

[0049] The voice activity detection and breathing bridging module is used to determine, based on the continuous voice signal, multiple expiratory support voice units of the target patient during free conversation, as well as the voice intervals between adjacent expiratory support voice units;

[0050] The voiceprint verification and round marking module is used to perform voiceprint verification on the exhalation support voice unit to distinguish between the target voice unit and the non-target voice unit, and to mark the interactive listening period;

[0051] The respiratory support mode analysis module is used to determine the respiratory support mode parameters of the target patient based on the plurality of expiratory support voice units and the voice intervals. The respiratory support mode parameters include at least one of the following: average expiratory support voice duration, voice time ratio, and voice interval statistics.

[0052] The judgment module is used to determine whether the respiratory status of the target patient is abnormal based on the respiratory support mode parameters and the individualized baseline parameters of the target patient. The individualized baseline parameters include time-specific baselines and day-type baselines.

[0053] The beneficial effects of this invention are: by determining the expiratory support speech unit and speech interval from the continuous speech signal of the target patient's daily free conversation, a respiratory support mode parameter including the average expiratory support speech duration, speech time ratio and speech interval statistics is constructed. This effectively utilizes the coupling time structure of speech and respiratory airflow, and can objectively quantify the patient's single expiratory support capacity and physiological ventilation needs in a natural communication scenario without the patient's awareness.

[0054] Meanwhile, by combining individualized baseline parameters, including time-specific and day-type baselines, to determine abnormalities, the system fully considers the fluctuations in the basic respiratory status of COPD patients under different work and rest periods and different day types, effectively reducing the false alarm rate caused by environmental factors and physiological rhythms, and improving the accuracy and reliability of respiratory status abnormality warnings. Attached Figure Description

[0055] Figure 1 This is a flowchart of the method of the present invention.

[0056] Figure 2 This is a flowchart of the expiratory support voice unit determination and breathing bridging sub-module of the present invention.

[0057] Figure 3 This is a flowchart of the sub-flowchart for calculating the ratio of speaking turn marking to speaking time in this invention.

[0058] Figure 4 This is the system architecture diagram of the present invention. Detailed Implementation

[0059] To facilitate a clear understanding of the technical means, creative features, objectives, and effects of this invention, the invention will be further described below in conjunction with specific illustrations.

[0060] To address the technical problem that existing home monitoring solutions for COPD cannot utilize the respiratory-voice coupling time structure in everyday conversational speech for respiratory status monitoring and early warning, this application provides an Internet of Things-based method for monitoring and warning the respiratory status of COPD patients. (Refer to...) Figure 1 This includes the following steps:

[0061] S1: Acquire continuous speech signals generated by the target patient during free conversation.

[0062] Speech signals are continuously acquired by a multi-microphone array deployed in the daily activity space of the target patient. The multi-microphone array contains no fewer than four microelectromechanical system microphones, which are arranged in a ring with a spacing of 8 cm. The sampling rate is 16 kHz and the quantization bit depth is 16 bits.

[0063] A multi-microphone array uses a delay-summation beamforming algorithm to form a directional pickup beam pointing towards the target patient's mouth and nose, thereby enhancing the acoustic signal from the target patient's direction and suppressing interference signals from other directions. After beamforming, a high-frequency pre-emphasis filter is added for gain compensation in the 4 kHz to 8 kHz frequency band to counteract the high-frequency attenuation of the signal during air propagation. The transfer function of this high-frequency pre-emphasis filter is H(z) = 1 - α·z⁻¹, where the pre-emphasis coefficient α ranges from 0.95 to 0.97. An adaptive noise cancellation algorithm is used to remove background noise components from the speech signal in real time; this algorithm employs a normalized least mean square algorithm. In an extended configuration, microphones can be deployed at multiple permanent locations of the target patient to construct a distributed microphone network. A sound source localization algorithm automatically identifies the target patient's current location and selects the nearest microphone node for pickup. The sound source localization algorithm uses a generalized cross-correlation phase transform algorithm to calculate the time difference of arrival.

[0064] S2: Based on continuous speech signals, identify multiple expiratory support speech units during the target patient's free conversation, as well as the speech intervals between adjacent expiratory support speech units.

[0065] S21: Detect voiced segments in a continuous speech signal and extract the fundamental frequency of the voiced segments, while detecting whether there are unvoiced features during the fundamental frequency interruption.

[0066] refer to Figure 2 The continuous speech signal is processed by framing, with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. A Hamming window is added to reduce spectral leakage. The framed signal is then analyzed in parallel.

[0067] In the first analysis channel, the fundamental frequency is extracted using the autocorrelation method or the cepstral method. When the fundamental frequency is greater than or equal to 70 Hz and less than or equal to 300 Hz and a stable harmonic structure exists, this period is marked as a voiced segment.

[0068] The system provides targeted enhancement to the high-frequency band where the energy of the fricative is concentrated. In this embodiment, this high-frequency band is defined as the first band, which is a high-frequency band from 4 kHz to 8 kHz. This band range is determined based on the spectral characteristics of the high-frequency frictional noise generated by airflow passing through the narrow part of the vocal tract during the production of the fricative. Although the fricative has no vocal cord vibration or fundamental frequency, its energy is mainly concentrated in the wide frequency range of 4 kHz to 8 kHz. By detecting the high-frequency frictional energy in this band, the fricative band can be accurately distinguished from environmental noise, thereby ensuring the accuracy of the boundary recognition of the expiratory support speech unit.

[0069] Simultaneously, the first type of voiceless consonant feature, namely the voiceless fricative feature, is detected in the second analysis channel. During the fundamental frequency interruption, the high-frequency frictional energy of the 4 kHz to 8 kHz broadband band is calculated. When the energy in this band exceeds the preset decibel value of the ambient noise floor and the duration is greater than or equal to the second threshold of 50 milliseconds and less than or equal to the first threshold of 250 milliseconds, it is marked as a voiceless fricative segment. Voiceless fricatives, such as / s / , / ʃ / , / f / , / h / , and / x / in the International Phonetic Alphabet, although lacking vocal cord vibration and fundamental frequency, require continuous airflow through the narrow part of the vocal tract to generate high-frequency friction, belonging to the speech continuation part supported by a single exhalation. 50 milliseconds is a typical dividing line in the physics of articulation between the plosive release of voiceless plosives and the sustained friction of voiceless fricatives. The plosive segment of a voiceless plosive usually does not exceed 50 milliseconds, while the duration of the frictional segment of a voiceless fricative is usually greater than 50 milliseconds.

[0070] The real-time estimation of the preset ambient noise floor value in decibels is achieved as follows: During periods without speech activity, the system continuously acquires ambient acoustic signals and uses a minimum statistics-based algorithm to track the minimum energy envelope value of each frequency band. This algorithm continuously monitors the energy values ​​of each frequency band and records their minimum values ​​within a preset time window (e.g., 0.5 to 1.0 seconds), using this minimum value as the real-time benchmark for the current ambient noise floor. Based on this, "exceeding the preset ambient noise floor value" in the unvoiced sound feature detection refers to a signal energy exceeding this real-time noise floor benchmark by more than 20 decibels.

[0071] The system also includes a third analysis channel to detect second-type unvoiced sound features, namely unvoiced plosives. It detects instantaneous energy surges, characterized by an energy jump exceeding a preset decibel level of ambient noise within a 5-10 millisecond time window, accompanied by a sudden change in the zero-crossing rate. The zero-crossing rate is significantly higher in unvoiced sound periods, including plosive periods, than in voiced sound periods, and can be used as an auxiliary criterion. When this energy surge is detected and its duration is greater than the fifth threshold of 5 milliseconds but less than the second threshold of 50 milliseconds, it is marked as an unvoiced plosive segment. Unvoiced plosives, such as / p / , / t / , and / k / in the International Phonetic Alphabet, are extremely short plosives. Although their duration is very short and they do not involve vocal cord vibration, they still require instantaneous expiratory airflow to drive them, belonging to the speech continuation supported by a single exhalation. Fundamental frequency interruptions less than 5 milliseconds are usually caused by ambient transient impulse noise or microphone background noise and do not possess the phonetic characteristics of unvoiced plosives; therefore, they are used as noise filters.

[0072] S22: Perform a breathing bridging decision based on the results of the voiceless consonant feature detection, and mark the fundamental frequency interruption as a voiceless consonant bridging segment or a breathing pause boundary.

[0073] If a fricative segment exists during a fundamental frequency interruption and the fundamental frequency subsequently recovers, the fundamental frequency interruption is marked as a fricative bridging segment, and the voiced segments before and after the fundamental frequency interruption are smoothly connected and merged into the same expiratory support speech unit. If a plosive segment is detected during a fundamental frequency interruption and the fundamental frequency recovers within 50 milliseconds, the fundamental frequency interruption is marked as a plosive bridging segment, and the voiced segments before and after the fundamental frequency interruption are smoothly connected and merged into the same expiratory support speech unit.

[0074] If no unvoiced features are present during a fundamental frequency interruption and the duration of the interruption is greater than or equal to the third threshold of 250 milliseconds, the fundamental frequency interruption is marked as a breathing pause boundary, and the voiced segments before and after the interruption are identified as independent expiratory support speech units. If no unvoiced features are present during a fundamental frequency interruption and the duration of the interruption is less than the third threshold of 250 milliseconds, the interruption is marked as ambient noise interference and ignored.

[0075] The resulting expiratory support speech unit duration accurately reflects the single expiratory support capacity, avoiding the fragmentation of voiced sounds caused by voiceless consonants.

[0076] S23: Perform voiceprint verification on each expiratory support speech unit to distinguish between target speech units and non-target speech units, and mark the interactive listening period.

[0077] refer to Figure 3For each expiratory support speech unit confirmed through respiratory bridging, a voiceprint embedding vector is extracted. This voiceprint embedding vector is a 256-dimensional speaker identity representation vector. The extraction of the voiceprint embedding vector uses the ECAPA-TDNN model, which is a speaker recognition model pre-trained on the open-source speech dataset VoxCeleb and then deployed in an edge computing gateway. The cosine similarity between this voiceprint embedding vector and the target patient's baseline voiceprint embedding vector is calculated. The baseline voiceprint embedding vector is obtained during the system initialization phase through guided speech acquisition. Specifically, during the first three days after system deployment, the target patient is guided to read standard text through an associated smart speaker, and the voiceprint embedding vectors of multiple expiratory support speech units are extracted and averaged as the baseline voiceprint embedding vector.

[0078] Breathing support speech units with a cosine similarity less than a preset threshold are marked as non-target speech units and discarded. In this embodiment, the preset threshold is set to 0.70. This threshold is determined through statistical analysis of the voiceprint embedding vectors of a large number of different speakers. Under this threshold condition, the cosine similarity of the same speaker is distributed above 0.85, the cosine similarity of different speakers is distributed below 0.55, and 0.70 is between the two distributions, achieving a low false recognition rate. Breathing support speech units with a cosine similarity greater than or equal to the preset threshold are marked as target speech units. When there are discarded non-target speech units between two adjacent target speech units, the time period occupied by the non-target speech units is marked as the interactive listening period, and the interactive listening period is excluded from the calculation of subsequent breathing support mode parameters.

[0079] S3: Determine the respiratory support mode parameters for the target patient based on multiple expiratory support voice units and voice intervals.

[0080] The respiratory support mode parameters include at least one of the following: average expiratory support speech duration, speech time ratio, and speech intermittent statistics. Speech intermittent statistics characterize the distribution of speech intermittents between adjacent expiratory support speech units during free conversation of the target patient. Speech intermittent statistics include, but are not limited to, at least one of the following: median of physiological apnea, mean of physiological apnea, standard deviation of physiological apnea, or interquartile range of physiological apnea. In this embodiment, the speech intermittent statistics are preferably the median of physiological apnea.

[0081] A timeout threshold of 2.5 to 3.0 seconds is set for each individual speaking turn. When the silence duration between two adjacent target speech units is greater than or equal to this timeout threshold, the silence duration is marked as a nonverbal resting state, and the current speaking turn of the target patient is considered to have ended. This silence duration is not included in the calculation of physiological apnea. In actual conversation, a silence duration exceeding 2.5 to 3.0 seconds usually indicates the end of the current speaking turn. When the silence duration between two adjacent target speech units is less than this timeout threshold, the silence duration is marked as a physiological apnea.

[0082] The total effective vocalization time and the total physiological pause duration of the target speech unit are obtained within a single consecutive speaking round of the target patient. The vocalization time ratio is determined based on the total effective vocalization time and the total physiological pause duration. The vocalization time ratio is calculated according to the following formula (1):

[0083]

[0084] in, This indicates the total effective vocalization time of the target patient within a single consecutive speaking round, expressed in seconds. This represents the total duration of silence marked as physiological apnea within a given round, in seconds. The vocalization time ratio represents the proportion of effective vocalization time of the target patient within a single consecutive speaking round to the total time of that round. Long periods of silence spanning multiple rounds, i.e., nonverbal resting periods, are not included in the calculation of the vocalization time ratio.

[0085] The phonation duration of multiple target speech units was obtained to determine the average expiratory support speech duration, and the silence duration marked as physiological ventilatory pauses was obtained to determine the median of physiological ventilatory pauses.

[0086] S4: Determine whether the respiratory status of the target patient is abnormal based on the respiratory support mode parameters and the individualized baseline parameters of the target patient.

[0087] Individualized baseline parameters include time-specific baselines and day-type baselines, established as follows: During a pre-defined learning period, respiratory support mode parameter samples were collected from target patients at various time periods and day types. The statistical distribution of these parameters was determined based on the samples, and the individualized baseline parameters were defined using the median and interquartile range of the statistical distribution. The time-specific baseline divides the day into multiple time periods: 6:00 AM to 9:00 AM, 9:00 AM to 12:00 PM, 12:00 PM to 5:00 PM, and 5:00 PM to 10:00 PM, establishing baselines for each time period. The day-type baseline distinguishes between weekdays and rest days, establishing separate baselines. Since COPD patients' baseline respiratory status naturally declines on rainy days with low air pressure or poor air quality, using a baseline from sunny weather would lead to frequent false alarms. Therefore, the day-type baseline is further divided into sunny day baselines and rainy day baselines to accommodate normal differences in patients' baseline respiratory status under different weather conditions.

[0088] In this embodiment, a stable state is defined as follows: The system enters a baseline learning period by default for the first 14 days after deployment, during which no data intervention or judgment is performed, and the patient's overall condition is considered relatively stable. If the coefficient of variation of respiratory support mode parameter samples collected within the first 14 days is less than 10% and the absolute value of the daily parameter variation is less than 0.5%, this period is identified as a stable state sample and included in the baseline calculation.

[0089] This step includes two parallel early warning logic lines. The first is a graded early warning logic based on parameter trends: it compares the respiratory support mode parameters of the current time window with the individualized baseline parameters of the corresponding time period and day type, maintains a multi-day sliding window of no less than 7 days, calculates the daily slope of change of each parameter, and judges that there is an abnormality in the respiratory status when the parameters show a continuous deterioration trend. The specific judgment criteria are as follows: a level 1 early warning is triggered when the parameter deviates from the baseline by more than 1.5 times the interquartile range or the trend slope is continuously negative for more than 3 days; a level 2 early warning is triggered when the deviation is continuous for 3 days or the trend slope is continuously negative for more than 5 days and the cumulative change exceeds 15%; a level 3 early warning is triggered when the deviation is continuous for 5 days and the vocalization time decreases compared to the synchronization.

[0090] The second logic is an emergency warning system based on pathological panic events, which performs joint screening of pathological panic and emotional hyperarousal. It acquires the pitch variability and energy dynamic range of the target speech unit. Pitch variability is the standard deviation of the fundamental frequency within the target speech unit (in Hertz), and energy dynamic range is the ratio of maximum instantaneous energy to minimum instantaneous energy (dimensionless). When both pitch variability and energy dynamic range simultaneously exceed the individualized hyperarousal threshold of the target patient, the respiratory-associated acoustic signal corresponding to the target speech unit is detected. This individualized hyperarousal threshold is determined based on the statistical analysis of the target patient's historical speech characteristics in a stable state, specifically three times the mean pitch variability and the mean energy dynamic range in a stable state.

[0091] The wheezing and gasping sounds produced by airway narrowing in COPD patients have acoustic energy concentrated in a specific mid-frequency band, which is defined as the second frequency band in this embodiment. Specifically, the second frequency band is the mid-frequency band from 400 Hz to 900 Hz. This frequency range is determined based on the acoustic characteristics of the wheezing and gasping sounds. The wheezing sound produced by airway narrowing in COPD patients is a quasi-periodic narrowband signal, and its fundamental frequency and harmonic components are mainly concentrated in the range of 400 Hz to 900 Hz; at the same time, the gasping sound segment also exhibits a stable quasi-periodic structure within this frequency band.

[0092] To extract wheezing features, higher-order cumulant processing is performed on respiratory acoustic signals. The specific processing procedure is as follows. For a respiratory acoustic signal sequence x(n) of length N, the discrete calculation formula for its third-order cumulant is:

[0093]

[0094] Where C3(τ1, τ2) represents the third-order cumulant function, τ1 and τ2 are two independent time delay variables, ranging from 0 to L-1, where L is the preset maximum delay length. Data points exceeding the signal length N are padded with zeros. Since the third-order cumulant of Gaussian noise is always zero, calculating the third-order cumulant can effectively filter out environmental Gaussian white noise.

[0095] The bispectral feature B(f1, f2) is obtained by performing a two-dimensional Fourier transform on the third-order cumulant sequence, and its calculation formula is as follows:

[0096]

[0097] Here, f1 and f2 are two frequency variables, B(f1, f2) is a bispectral function, and B is a complex matrix.

[0098] Quasi-periodic components in the 400 Hz to 900 Hz frequency band are extracted from the bispectral matrix. Specifically, a cross-sectional analysis is performed along the main diagonal direction f1 = f2 in the bispectral matrix, and the amplitude spectrum sequence along this diagonal is extracted. The ratio of the peak value of this amplitude spectrum sequence in the 400 Hz to 900 Hz frequency band to the average value across the entire frequency band is calculated. When this ratio is greater than 2.5 and the duration of the quasi-periodic component is greater than or equal to a fourth threshold of 200 milliseconds, the respiratory acoustic signal is determined to contain wheezing characteristics. This ratio threshold of 2.5 was determined through statistical regression analysis of the bispectral characteristics of clinical wheezing and non-wheezing samples from COPD patients. Under this threshold condition, the sensitivity and specificity of wheezing detection reach clinically acceptable levels. When high-frequency inspiratory noise with a duration of less than 200 milliseconds is detected, the respiratory acoustic signal is determined to contain rapid inspiratory sound characteristics.

[0099] If the respiratory acoustic signal contains at least one of the characteristics of wheezing or rapid inspiration, the target speech unit is marked as a pathological panic speech unit. Since acute exacerbation of COPD accompanied by wheezing or rapid inspiration is a clinically critical event, there is no need to wait for the trend to worsen; therefore, a Level 3 warning is triggered directly. If the respiratory acoustic signal does not contain the characteristics of wheezing or rapid inspiration, the target speech unit is marked as an emotionally disruptive speech unit and removed; no warning is triggered.

[0100] S5: Implement the intelligent backup monitoring steps during the silent period.

[0101] When the number of expiratory support voice units collected within 24 consecutive hours falls below a preset threshold of 30, a voice guidance task is initiated to the target patient via a linked smart speaker to obtain standardized voice samples as the daily assessment baseline. The voice guidance task involves asking the target patient to read standardized text or count from 1 to 20. The system records the duration of the expiratory support voice units and the data on breath pauses when the target patient completes the guidance task, which serve as the minimum assessment baseline for the respiratory support mode parameters for the day.

[0102] S6: Implement enhanced monitoring steps for co-monitoring of cough and wheezing sounds.

[0103] Coughing events and wheezing segments were monitored during periods of no speech activity. Coughing events were initially identified using time-frequency features of transient high energy, wide spectrum, and durations ranging from 50 to 200 milliseconds. Further analysis of Mel-frequency cepstral coefficients was performed, and a pre-trained support vector machine classifier was used to distinguish coughing events from other transient noises such as door slams and clapping. Wheezing segments were identified using time-frequency features of quasi-periodic narrowband signals with frequencies ranging from 400 Hz to 900 Hz and durations ranging from 1 to 3 seconds. The frequency of coughs and the total duration of wheezing per unit time were statistically analyzed as co-validation metrics for respiratory support mode parameters.

[0104] S7: Performs edge computing and privacy protection.

[0105] All signal processing, voiceprint verification, parameter calculation, and trend analysis are completed on a local edge computing gateway. This gateway is based on an ARM architecture processor and integrates a neural network computing acceleration unit with a floating-point computing power of no less than one trillion operations per second to support real-time inference of the voiceprint embedding extraction model and the cough sound support vector machine classification model. A feature extraction-first, content-last-follows processing flow is adopted. Raw audio data is permanently deleted immediately after feature extraction in memory; neither the original speech signal nor the speech content is uploaded to the cloud. Only anonymized statistical values ​​and warning events are uploaded to the cloud; no playable speech data is stored in the cloud.

[0106] S8: Generate a visual report on respiratory support capabilities.

[0107] A daily summary report of respiratory support modes is automatically generated. This report includes a comprehensive respiratory support capability score, a 7-day trend chart of each respiratory support mode parameter, a visualization of the deviation of each parameter from the individualized baseline parameter, the proportion of non-verbal resting states, and records of warning events. The comprehensive respiratory support capability score ranges from 0 to 100, calculated by integrating indicators such as average expiratory support speech duration, speech-to-time ratio, median physiological apnea, and cough frequency, weighted according to preset parameters. The report is pushed to healthcare terminals in encrypted form, presenting the status of each indicator in a visual dashboard: green indicates normal, yellow indicates attention required, and red indicates abnormal.

[0108] refer to Figure 4 The present invention also provides an IoT-based respiratory status monitoring and early warning system for COPD patients, which is used to implement the above-mentioned IoT-based respiratory status monitoring and early warning method for COPD patients, including a voice signal acquisition module, a voice activity detection and respiratory bridging module, a voiceprint verification and round marking module, a respiratory support mode analysis module, and a judgment module.

[0109] The speech signal acquisition module is used to acquire continuous speech signals generated by the target patient during free conversation. The speech signal acquisition module is a multi-microphone array deployed in the target patient's daily activity space, containing no fewer than four microelectromechanical system (MEMS) microphones arranged in a ring. The speech signal acquisition module includes an adaptive beamforming and high-frequency pre-emphasis compensation subunit, used to form a directional pickup beam pointing towards the target patient's mouth and nose through delay-summation beamforming, and to perform high-frequency pre-emphasis compensation in the 4 kHz to 8 kHz frequency band.

[0110] The speech activity detection and breathing bridging module is used to determine multiple expiratory support speech units and speech intervals between adjacent expiratory support speech units during a target patient's free conversation, based on continuous speech signals. The module includes a voiced fundamental frequency detection subunit, a voiceless high-frequency friction and plosive sound detection subunit, and a breathing bridging decision subunit.

[0111] The voiced fundamental frequency detection subunit is used to detect the presence of a continuous fundamental frequency trajectory and stable harmonic structure in a continuous speech signal. The unvoiced high-frequency friction and plosive detection subunit is used to detect high-frequency friction energy in the 4 kHz to 8 kHz frequency band during fundamental frequency interruptions to identify first-type unvoiced features, i.e., unvoiced fricatives, and to detect instantaneous energy spikes combined with zero-crossing rate changes to identify second-type unvoiced features, i.e., unvoiced plosives. The breathing bridging decision subunit is used to mark the fundamental frequency interruption as a unvoiced consonant bridging segment and connect the voiced segments before and after the fundamental frequency interruption into the same expiratory support speech unit when unvoiced features are present and fundamental frequency recovery is detected; and to mark the fundamental frequency interruption as a pausing boundary and determine the voiced segments before and after the fundamental frequency interruption as independent expiratory support speech units when no unvoiced features are present and the duration of the fundamental frequency interruption is greater than or equal to a third threshold.

[0112] The voiceprint verification and turn marking module is used to verify the voiceprints of expiratory support speech units to distinguish between target and non-target speech units and to mark the interactive listening period. This module includes a target speaker verification subunit and a conversation turn marking and timeout truncation subunit. The target speaker verification subunit extracts the voiceprint embedding vector for each expiratory support speech unit, calculates the cosine similarity between the voiceprint embedding vector and the target patient's baseline voiceprint embedding vector, marks expiratory support speech units with a cosine similarity less than a preset threshold as non-target speech units and discards them, and marks expiratory support speech units with a cosine similarity greater than or equal to the preset threshold as target speech units. The conversation turn marking and timeout truncation subunit is used to mark the time period occupied by the non-target speech unit as the interactive listening period when there is a discarded non-target speech unit between two adjacent target speech units, and to exclude the interactive listening period from the calculation of subsequent respiratory support mode parameters. It also sets the timeout threshold for a single person's speaking turn and marks it as a non-verbal resting state or a physiological apnea based on the relationship between the silence duration between two adjacent target speech units and the timeout threshold.

[0113] The respiratory support pattern analysis module is used to determine respiratory support pattern parameters for a target patient based on multiple expiratory support speech units and speech pauses. These parameters include at least one of the following: average expiratory support speech duration, phonation time ratio, and median physiological apnea. The module comprises a respiratory support parameter extraction subunit and a pathological panic-emotion joint screening subunit. The phonation parameter extraction subunit is used to obtain the phonation duration of multiple target speech units to determine the average expiratory support speech duration, obtain the silence duration marked as physiological apnea to determine the median physiological apnea, and obtain the total effective phonation duration and the total physiological apnea duration of the target speech units within a single consecutive speaking round of the target patient, determining the phonation time ratio based on these two parameters.

[0114] The pathological panic-emotion joint screening subunit is used to obtain the pitch variability and energy dynamic range of the target speech unit. When both pitch variability and energy dynamic range exceed the individualized high arousal threshold of the target patient, the respiratory acoustic signal corresponding to the target speech unit is detected. The respiratory acoustic signal is processed by high-order cumulant to obtain the target signal after suppressing Gaussian noise, and the quasi-periodic component in the 400 Hz to 900 Hz frequency band is extracted from the target signal. When the duration of the quasi-periodic component is greater than or equal to the fourth threshold of 200 ms, the respiratory acoustic signal is determined to contain wheezing features. When high-frequency inspiratory noise with a duration of less than 200 ms is detected, the respiratory acoustic signal is determined to contain rapid inspiratory features. If the respiratory acoustic signal contains at least one of the wheezing features or rapid inspiratory features, the target speech unit is marked as a pathological panic speech unit. If the respiratory acoustic signal does not contain wheezing features or rapid inspiratory features, the target speech unit is marked as an emotional interference speech unit and is removed.

[0115] The judgment module is used to determine whether the respiratory status of the target patient is abnormal based on the respiratory support mode parameters and the target patient's individualized baseline parameters. Individualized baseline parameters include time-specific baselines and day-type baselines. The judgment module maintains a multi-day sliding window of no less than 7 days, calculates the daily slope of change for each respiratory support mode parameter, and judges that the respiratory status is abnormal when the parameter shows a continuous deterioration trend.

[0116] In this embodiment, the first 3 days after system deployment are the baseline voiceprint registration period. The patient is guided to read standard text through a smart speaker. The ECAPA-TDNN 256-dimensional embedding vectors of multiple expiratory support speech units are extracted and averaged to serve as the patient's baseline voiceprint embedding vector.

[0117] The first 14 days after system deployment served as a baseline learning period. Initial baselines for each time period were established through guided voice data collection, divided into four time periods: 6:00 AM to 9:00 AM, 9:00 AM to 12:00 PM, 12:00 PM to 5:00 PM, and 5:00 PM to 10:00 PM. Daily baselines for weekdays and rest days were also established. Each baseline was defined using the median and interquartile range of the respiratory support mode parameter samples for the corresponding time period and daily type.

[0118] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention, all of which fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for monitoring and early warning of respiratory status in COPD patients based on the Internet of Things, characterized in that, include: Acquire continuous speech signals generated by the target patient during free conversation; Based on the continuous speech signal, determine multiple expiratory support speech units of the target patient during free conversation, as well as the speech intervals between adjacent expiratory support speech units; Based on the plurality of expiratory support voice units and the voice intervals, the respiratory support mode parameters of the target patient are determined, wherein the respiratory support mode parameters include at least one of the following: average expiratory support voice duration, voice time ratio, and voice interval statistics. Based on the respiratory support mode parameters and the individualized baseline parameters of the target patient, it is determined whether the respiratory status of the target patient is abnormal. The individualized baseline parameters include time-specific baselines and day-type baselines.

2. The method according to claim 1, characterized in that, The step of determining multiple expiratory support speech units of the target patient during free conversation based on the continuous speech signal includes: Detect voiced segments in the continuous speech signal and extract the fundamental frequency of the voiced segments; When the baseband interruption is detected, the continuous speech signal is detected to have a decanted sound feature during the baseband interruption. The decanted sound feature includes a first type of decanted sound feature and a second type of decanted sound feature. The first type of decanted sound feature is a high-frequency friction energy feature of a first frequency band with a duration greater than or equal to a second threshold and less than or equal to a first threshold. The second type of unvoiced sound feature is an energy surge that lasts longer than the fifth threshold and less than the second threshold, exceeding the preset decibel value of the ambient noise floor and accompanied by a sudden change in the zero-crossing rate. If the unvoiced sound feature exists during the fundamental frequency interruption and fundamental frequency recovery is detected, the fundamental frequency interruption is marked as a voiceless consonant bridging segment, and the voiced sound segments before and after the fundamental frequency interruption are connected into the same expiratory support speech unit. If the unvoiced sound feature is not present during the fundamental frequency interruption, and the duration of the fundamental frequency interruption is greater than or equal to the third threshold, the fundamental frequency interruption is marked as a breathing pause boundary, and the voiced sound segments before and after the fundamental frequency interruption are respectively determined as independent expiratory support speech units; if the duration of the fundamental frequency interruption is less than the third threshold, the fundamental frequency interruption is marked as environmental noise interference and ignored.

3. The method according to claim 2, characterized in that, The method of determining multiple expiratory support speech units of the target patient during free conversation also includes: For each of the expiratory support speech units, extract the voiceprint embedding vector; Calculate the cosine similarity between the voiceprint embedding vector and the baseline voiceprint embedding vector of the target patient; The exhalation support speech units whose cosine similarity is less than a preset threshold are marked as non-target speech units and discarded. The exhalation support speech units whose cosine similarity is greater than or equal to the preset threshold are marked as target speech units; When there is a discarded non-target speech unit between two adjacent target speech units, the time period occupied by the non-target speech unit is marked as the interactive listening period, and the interactive listening period is excluded from the calculation of subsequent breathing support mode parameters.

4. The method according to claim 3, characterized in that, The step of determining the respiratory support mode parameters for the target patient based on the plurality of expiratory support voice units and the voice intervals includes: Set the timeout threshold for each individual's speaking turn; When the silence duration between two adjacent target speech units is greater than or equal to the timeout threshold, the silence duration is marked as a non-verbal resting state, and the current speaking round of the target patient is determined to have ended. The silence duration is not included in the calculation of physiological apnea. When the silence duration between two adjacent target speech units is less than the timeout threshold, the silence duration is marked as a physiological apnea. Within a single consecutive speaking round of the target patient, the total effective vocalization duration of the target speech unit and the total duration of the physiological apnea are obtained; The vocalization time ratio is determined based on the total effective vocalization time and the total physiological apnea time. The vocalization time ratio is the total effective vocalization time divided by the sum of the total effective vocalization time and the total physiological apnea time. The vocalization time ratio represents the proportion of the target patient's effective vocalization time in a single consecutive speaking round to the total time of the single consecutive speaking round.

5. The method according to claim 4, characterized in that, The statistical value of the speech intermittent includes the median of physiological ventilation pauses; the step of determining the respiratory support mode parameters of the target patient based on the plurality of expiratory support speech units and the speech intermittent further includes: The phonation duration of multiple target speech units is obtained, and the average expiratory support speech duration is determined. Obtain the silence duration marked as the physiological apnea, and determine the median of the physiological apnea.

6. The method according to claim 1, characterized in that, The individualized baseline parameters are established in the following way: During the preset learning period, respiratory support mode parameter samples of the target patients were collected at different time periods and on different days. The statistical distribution of respiratory support mode parameters for each time period and each day type is determined based on the sample, and the individualized baseline parameters are defined by the median and interquartile range of the statistical distribution.

7. The method according to claim 4, characterized in that, Also includes: Obtain the pitch variability and energy dynamic range of the target speech unit; When the pitch variability and the energy dynamic range both exceed the individualized high arousal threshold of the target patient, the respiratory-associated acoustic signal corresponding to the target speech unit is detected. The individualized high arousal threshold is determined based on the statistical analysis of the historical speech features of the target patient in a stable state. The respiratory-associated acoustic signal is processed by a higher-order cumulant to obtain a target signal after suppressing Gaussian noise, and the quasi-periodic component of the second frequency band is extracted from the target signal. When the duration of the quasi-periodic component is greater than or equal to the fourth threshold, the respiratory acoustic signal is determined to contain wheezing characteristics; or, when a high-frequency inspiratory noise with a duration of less than 200 milliseconds is detected, the respiratory acoustic signal is determined to contain rapid inspiratory sound characteristics. If the respiratory acoustic signal contains at least one of the wheezing feature or the rapid inhalation feature, the target speech unit is marked as a pathological panic speech unit; if the respiratory acoustic signal does not contain the wheezing feature or the rapid inhalation feature, the target speech unit is marked as an emotional interference speech unit and removed.

8. The method according to claim 1, characterized in that, Also includes: When the number of the exhalation support voice units collected within a preset time period is lower than a preset threshold, a voice guidance task is initiated to the target patient through the associated smart speaker to obtain standardized voice samples as the evaluation benchmark for the day. Alternatively, during periods without speech activity, cough events and wheezing sounds can be monitored, and the frequency of coughs and the total duration of wheezing can be counted as co-validation indicators for the parameters of the respiratory support mode.

9. A respiratory status monitoring and early warning system for COPD patients based on the Internet of Things, characterized in that, include: The speech signal acquisition module is used to acquire continuous speech signals generated by the target patient during free conversation; The voice activity detection and breathing bridging module is used to determine, based on the continuous voice signal, multiple expiratory support voice units of the target patient during free conversation, as well as the voice intervals between adjacent expiratory support voice units; The voiceprint verification and round marking module is used to perform voiceprint verification on the exhalation support voice unit to distinguish between the target voice unit and the non-target voice unit, and to mark the interactive listening period; The respiratory support mode analysis module is used to determine the respiratory support mode parameters of the target patient based on the plurality of expiratory support voice units and the voice intervals. The respiratory support mode parameters include at least one of the following: average expiratory support voice duration, voice time ratio, and voice interval statistics. The judgment module is used to determine whether the respiratory status of the target patient is abnormal based on the respiratory support mode parameters and the individualized baseline parameters of the target patient. The individualized baseline parameters include time-specific baselines and day-type baselines.

Images