Detection and classification of cardiovascular abnormalities based on pulse waveforms

Abstract

A method for constructing and implementing an anomaly detection system for assessing the health state of a user by analysis of PPG signals of the user measured by using wearable techniques. In one embodiment of this anomaly detection system, a convolutional neural network deep learning model is used to derive feature vectors of PPG signals, and is used to construct low-dimensional feature maps along with known cardiovascular health issues, thereby identifying possible health issues among users with unknown health states.

Description

Background Art

[0001] Since the 1870s, the value of pulse waveforms in medical applications has become apparent. FA Mahomed pointed out in 1872 that the information contained in the pulse waveform is so important and is consulted so frequently that it is necessary for medical professionals to fully understand the pulse waveform and extract as much detail as possible from it. Since then, the pulse waveform has become a routine health monitoring data stream in the medical profession and is used for monitoring of vital signs such as but not limited to heart rate, cardiac cycle, respiration, depth of anesthesia, and blood pressure. Analysis of the pulse waveform has also been used to monitor the development of applications in a wider range of health, nursing, and medicine (HWM) industries. This allows continuous monitoring of some medically relevant vital signs, such as but not limited to heart rate, respiratory rate, and oxygen saturation (SpO2). Summary of the invention

[0002] Embodiments of the claimed invention include a method of assessing a user's cardiovascular health status by interpreting photoplethysmography (PPG) signal data obtained through a wearable device using anomaly detection techniques in combination with more conventional methods of analyzing PPG signal data, thereby providing feedback to the user and interested third parties such as medical practitioners (by making the obtained results available to them). The PPG signal obtained from the wearable device requires some digital signal processing to filter, detrend and denoise the data before analysis. In one aspect, the PPG signal can be fed into an anomaly detection system after being divided into segments of equal length for training or analysis of the PPG signal data using a fully trained anomaly detection system. In some instances, conventional analysis of the PPG signal can be used to improve the feedback provided to the user or interested third party. Embodiments of the claimed invention can help medical practitioners remotely monitor patients diagnosed with cardiovascular health problems and evaluate the success of recommended medical interventions by continuously collecting and analyzing PPG signal data using wearable devices.

[0003] According to one aspect, the present invention relates to a method for identifying a user's health status using an IoT device of interconnected electronic devices and sensors. The method may include: collecting a user's PPG signal using a wearable device; preprocessing the PPG signal for conventional analysis of the PPG signal, and for extracting critical points and interpretability engineering one-dimensional (1D) features (such as but not limited to rise time, SEVR, ejection duration index, large artery stiffness index, small artery resistance, and features related to hypertension). After preprocessing, the method may utilize an anomaly detection system to generate a feature vector in a latent space, and the anomaly detection system may be exemplified by a convolutional neural network trained on pulse waveform data. Then, a dimensionality reduction method may be used to construct a low-dimensional representation (two-dimensional or three-dimensional) of the feature vector. The dimensionality reduction method may include PCA or t-SNE.

[0004] Based on the categories assigned to the PPG signals by the anomaly detection system, partitions of the two-dimensional / three-dimensional space can then be labeled as corresponding to healthy, specific conditions, or unknown. Interpretable engineering 1D features can then be created, which refer to specific physiological processes associated with health risks. These interpretable engineering features can be used in conjunction with the health output / disease output / unknown output of the anomaly detection system to resolve "unknown" anomalies as healthy or unhealthy based on whether the interpretable engineering feature has a 1D value associated with a health condition or a 1D value associated with a specific condition. In addition, in the case of an unknown label output, feedback related to the user's health status can be provided to the user or a third party.

[0005] According to one aspect, conventional analysis of PPG signals may be used to determine critical points associated with the PPG pulse waveform and its derivatives, compute a complete set of features from the critical points, determine health-related and interpretability engineering one-dimensional (1D) features from the complete set of features computed from the critical points, and derive a small subset of interpretability features related to independent aspects of health and anatomy. A quadratic moving average method may be used to determine critical points associated with PPG, VPG, and APG signals. Additionally, a complete set of features may be derived from the difference between any two critical points in the form of amplitude, time span, sub-region, and slope features. In some aspects, the interpretability engineering one-dimensional (1D) features include rise time, normalized rise time, SEVR, ejection duration, ejection duration index, large artery stiffness index, arteriolar resistance, and features related to hypertension, which are derived from a complete set of features computed from the x- and y-coordinates of the critical points. Techniques for building covariance matrices or principal component analysis are used to remove interpretable engineered one-dimensional (1D) features with covariance, or are used to select interpretable engineered one-dimensional (1D) features that retain unique health or anatomical information.

[0006] In one aspect, a convolutional neural network may be constructed from a one-dimensional or two-dimensional representation of the PPG signal (via frequency domain methods, such as Fourier spectrum) for extracting feature sets from the PPG signal for classifying these feature sets into health state categories and for placing the health states in a low-dimensional representation manifold. Input data derived from segments of the PPG signal may be preprocessed and transformed into a one-dimensional representation of the PPG signal, or used to derive a two-dimensional frequency domain representation of the PPG, or used through cardiopulmonary coupling, the two-dimensional frequency domain representation including Fourier spectrum, Lomb-Scargle periodogram. In addition, a one-dimensional or two-dimensional representation of the PPG signal data may be used as input data to train a CNN and obtain an output feature set, wherein the output feature set is mapped onto a low-dimensional representation framework using methods including t-SNE or PCA. A loss function may be used to organize the manifold by identifying health state regions in the low-dimensional representation manifold and to create continuity in the feature set categories, wherein the loss function includes triplet loss, mean squared error loss, or cross entropy loss.

[0007] In one aspect, a user's CNN feature set corresponding to unknown regions in a low-dimensional representation manifold can be compared to interpretable engineering one-dimensional (1D) features that contain unique health and anatomical information as derived through conventional analysis of PPG signals to assess the user's health state. Unknown regions in a low-dimensional representation manifold can be assigned based on the interpretable engineering one-dimensional (1D) features that contain unique health and anatomical information corresponding to those unknown regions as derived through conventional analysis of PPG signals.

[0008] Feedback may be provided to the user and interested third parties using a display including a desktop computer display, a laptop computer display, a smartphone display, a wearable device display, a phone call, a text message, an email, or a web-based dashboard. The computational aspects of the present invention may be performed remotely on a device such as, but not limited to, a smart wearable device, a smartphone, a desktop computer, a laptop computer, or may be performed using a cloud computing infrastructure. The wearable device may include a fingertip pulse oximeter, an earlobe pulse oximeter, a wrist-worn wearable device, or a smart ring. 21. The method of claim 1, wherein the dimensionality reduction method comprises PCA or t-SNE?

[0009] According to another aspect of the present disclosure, the method relates to a method for identifying a user's health status using an IoT device of interconnected electronic devices and sensors. The method includes: using a wearable device to collect a user's PPG signal, and preprocessing the PPG signal for conventional analysis of the PPG signal, thereby extracting at least one interpretable engineering one-dimensional (1D) feature from the PPG signal. An anomaly detection system can be used on the PPG signal to generate a feature vector in a latent space, and to generate a classification result of the PPG signal corresponding to a health state or corresponding to a state associated with another condition. A low-dimensional representation of the feature vector can then be created by applying PCA or t-SNE to the feature vector. The low-dimensional representation of the PPG signal can be annotated to correspond to the state classified by the anomaly detection system. Then, the classification result can be associated with the interpretable engineering 1D feature in the low-dimensional representation space. Next, the anomaly marked as unknown can be parsed as belonging to a health state or associated with another condition. Then, feedback on the user's health status can be provided to the user or a third party. In one aspect, the wearable device includes one of a fingertip pulse oximeter, an earlobe pulse oximeter, a wrist-worn wearable device, or a smart ring.

[0010] In some aspects, the interpretability engineered 1D features may include rise time, SEVR, ejection duration index, large artery stiffness index, small artery resistance, and features associated with hypertension. Additionally, the anomaly detection system includes a convolutional neural network trained on PPG signal data from other users.

[0011] In one aspect, the present invention relates to identifying a user's health status using an IoT device of interconnected electronic devices and sensors. The method may include collecting a PPG signal of the user using at least one of the wearable devices. The wearable device may include, but is not limited to, a fingertip pulse oximeter, an earlobe pulse oximeter, a wrist-worn wearable device, or a smart ring.

[0012] The next step may include pre-processing the PPG signal for conventional analysis of the PPG signal. The pre-processing may include applying a pre-processing filter such as, but not limited to, an inverse Chebyshev filter or a Butterworth filter to improve signal quality. Additionally, VPG and APG may be derived from the PPG signal. Furthermore, a quadratic moving average method may be used to extract critical points associated with the PPG, VPG, and APG signals. In one aspect, conventional analysis of the PPG signal may be used to determine critical points associated with the PPG pulse waveform and its derivatives, compute a complete set of features from the critical points, determine health-related and interpretability engineered one-dimensional (1D) features from the complete set of features computed from the critical points, and derive a small subset of interpretability features related to independent aspects of health and anatomy.

[0013] After preprocessing, determination of interpretable engineering one-dimensional (1D) features such as, but not limited to, rise time, SEVR, ejection duration index, large artery stiffness index, arteriolar resistance, and features related to hypertension may be accomplished. Thus, an anomaly detection system may be used on the PPG signal to generate a feature vector in a latent space and to generate a classification result of the PPG signal corresponding to a healthy state or corresponding to a state associated with another condition. A low-dimensional representation of the feature vector may then be constructed, wherein the low-dimensional representation of the PPG signal corresponding to the state classified by the anomaly detection system is labeled. The interpretable engineering 1D features are then associated with the classification results in the low-dimensional representation space. Next, anomalies labeled as unknown may be resolved as belonging to a healthy state or associated with another condition. Feedback regarding the user's health status may then be provided to the user or a third party.

[0014] In one aspect, the interpretable engineering one-dimensional (1D) features may include rise time, normalized rise time, SEVR, ejection duration, ejection duration index, large artery stiffness index, arteriolar resistance, and features related to hypertension derived from a complete set of features calculated from the x- and y-coordinates of the critical points. Additionally, techniques for building a covariance matrix or principal component analysis may be used to remove interpretable engineering one-dimensional (1D) features with covariance, or to select interpretable engineering one-dimensional (1D) features that retain unique health or anatomical information.

[0015] In some aspects, a convolutional neural network may be constructed from a one-dimensional representation or a two-dimensional representation (via frequency domain methods, such as Fourier spectrum) of the PPG signal. The convolutional neural network may be used to extract feature sets from the PPG signal for classification of these feature sets into health state categories and / or for placing the health states in a low-dimensional representation manifold. In addition, input data derived from segments of the PPG signal may be preprocessed and transformed into a one-dimensional representation of the PPG signal, or used to derive a two-dimensional frequency domain representation of the PPG, or used through cardiopulmonary coupling, wherein the two-dimensional frequency domain representation of the PPG includes a Fourier spectrum, a Lomb-Scargle periodogram. In some aspects, a one-dimensional representation or a two-dimensional representation of the PPG signal data is used as input data to train a CNN and to obtain an output feature set, wherein the output feature set is mapped to a low-dimensional representation framework using methods including t-SNE or PCA. Additionally, loss functions may be used to organize the manifold by identifying healthy state regions in the low-dimensional representation manifold and to create continuity among feature set categories, wherein the loss functions include triplet loss, mean squared error loss, or cross entropy loss.

[0016] In some aspects, a user's CNN feature set corresponding to unknown regions in a low-dimensional representation manifold can be compared to interpretability engineered one-dimensional (1D) features to assess the user's health state, a small subset of features containing unique health and anatomical information as derived through conventional analysis of PPG signals.

[0017] In one aspect, the present disclosure relates to an IoT system of interconnected devices and sensors. The system collects PPG signals of a user using a wearable device. The PPG signal is preprocessed to extract critical points and interpretable 1D features, such as rise time, SEVR, ejection duration index, large artery stiffness index, and arteriolar resistance. An anomaly detection system is used to generate a feature vector from a CNN, which is trained on pulse waveform data. The system can then construct a 2D or 3D representation of the feature vector by dimensionality reduction, and can then annotate the 2D / 3D space as healthy, specific condition, or unknown based on the category assigned by the anomaly detection system. Thus, the system can then create interpretable 1D features related to physiological processes associated with health risks, and then assign health status by combining the output of the anomaly detection system with the interpretable 1D features. The system can then provide feedback on the health status to the user or a third party.

[0018] In one aspect, routine analysis of a PPG signal may include determining critical points in the PPG signal and its derivatives, computing a complete set of features from the critical points, extracting interpretable 1D features from the complete set, and selecting a subset of interpretable features that are relevant to health and anatomy. The critical points may be determined using a quadratic moving average method. Features may be derived from the difference between any two critical points, including amplitude, time span, sub-region, and slope features. Interpretable 1D features may include rise time, SEVR, ejection duration index, large artery stiffness index, and arteriolar resistance.

[0019] In these aspects, the system can construct a CNN to derive a feature set from the PPG signal, classify the feature set into health state categories, and map the health state to a low-dimensional representation space. Thus, the CNN can be trained on a 1D representation or a 2D representation of the PPG signal, and a method such as t-SNE or PCA is used to map the output feature set to a low-dimensional representation space. In some aspects, a loss function is used to organize the low-dimensional representation space and identify health state areas, and the loss function includes a triplet loss, a mean squared error loss, or a cross entropy loss. An unknown CNN feature set can be compared with an interpretable 1D feature to assess the user's health state.

[0020] It should be understood that this summary is not an extensive overview of the present disclosure. This summary is exemplary and non-restrictive, and is not intended to identify key or important elements of the present disclosure, nor is it intended to describe its scope. The sole purpose of this summary is to explain and exemplify certain concepts of the present disclosure as an introduction to the following complete and exhaustive description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The features and components of the following drawings are illustrated to emphasize the overall principles of the present disclosure. For consistency and clarity, corresponding features and components throughout the drawings may be designated by matching reference characters.

[0022] Figure 1 is a schematic representation of one embodiment of the invention.

[0023] Figure 2 is an illustration of PPG, VPG and APG signals and the locations of critical points.

[0024] Figure 3 is an exemplary method for determining a subset of interpretable features related to independent aspects of health and anatomy.

[0025] Figure 4 is an illustration of how manifolds in a CNN embedding space can be used to implement an anomaly detection system.

[0026] Figure 5 is a schematic representation of an IOT device for operating the present invention.

[0027] Figure 6 is an illustration of how the quadratic moving average method can be applied to a PPG pulse waveform. DETAILED DESCRIPTION

[0028] definition

[0029] CNN - Convolutional Neural Network

[0030] IOT-Internet of Things

[0031] SIANN - Translation / Space Invariant Artificial Neural Network

[0032] PPG-Photoplethysmography

[0033] VPG - Velocity Plethysmography

[0034] APG - Acceleration Plethysmography

[0035] AWS - Amazon Web Services

[0036] t-SNE - t distribution - Stochastic Neighbor Embedding

[0037] PCA - Principal Component Analysis

[0038] Afib - Atrial Fibrillation

[0039] GAN-Generative Adversarial Network

[0040] ResNeXt - A 50-layer deep neural network model developed to identify atrial fibrillation using PPG signal data.

[0041] De-noising - The process of removing unwanted modifications or noise from signal data generated during the measurement of the signal data.

[0042] It should be understood that the present disclosure is not limited to the systems and components described herein.It should also be understood that the terminology used herein is for the purpose of describing certain embodiments only and is not intended to be limiting, as the scope of the present disclosure will be limited only by the appended claims.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present disclosure belongs. Any systems and components similar or equivalent to any systems and components described herein may be used in the implementation or testing of the present invention. All disclosures mentioned are incorporated herein by reference in their entirety.

[0044] The use of the terms "a," "an," "the" and similar referents in the context of describing the presently claimed invention (especially in the context of the claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context.

[0045] Recitation of ranges of values ​​herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein.

[0046] The pulse waveform can be measured by a technique such as photoplethysmography (PPG), which uses a variety of wearable and non-wearable pulse oximeter devices, wearable earlobe pulse oximeter devices, and smart wearable devices such as wrist-worn devices and smart rings. The PPG signal is obtained by illuminating the skin and measuring the changes in light absorption caused by perfusing blood to the dermis and subcutaneous tissue of the skin. Blood is pumped to the dermis and subcutaneous tissue of the skin with each heart cycle to produce a pressure pulse that moves through arteries and arterioles. The pressure pulse changes the volume of the arteries and arterioles of the subcutaneous tissue, which can be detected by illuminating the skin with light from a light-emitting diode (LED). The amount of light transmitted (as in the case of a fingertip pulse oximeter) or reflected (as in the case of a wrist-worn wearable device) to the photodiode can therefore be measured. The resulting PPG signal appears as a series of peaks, each of which is generated by a heart cycle.

[0047] The PPG signal obtained from the wearable device reflects the movement of blood in the blood vessels of the subcutaneous tissue, which moves from the heart to the dermis and subcutaneous tissue of the skin where the wearable device is placed. The wave-like movement of the blood pressure pulse changes the amount of light transmitted through the limb where the wearable device is placed, or changes the amount of light backscattered to the photodiode of the wearable device. This change in the light reaching the photodiode corresponds to the change in blood volume in the pressure pulse.

[0048] The PPG signal continuously captures the wave-like motion of the blood pressure pulse, which causes a pressure pulse corresponding to each heartbeat. Each heartbeat corresponds to a pulse waveform that captures the characteristics of the heart during the corresponding heartbeat. The resulting arterial pulse waveform consists of three unique components that show heart function: (1) systolic phase; (2) dicrotic phase; and (3) diastolic phase.

[0049] The systolic phase of the pulse waveform is characterized by a rapid increase in pressure, which increases until it reaches a maximum pressure called the systolic peak (S), followed by a decrease in the pressure pulse. The systolic phase begins with the opening of the aortic valve and corresponds to the ejection of blood from the left ventricle. The next component of the pulse waveform is called the dicrotic notch (N) and is widely believed to correspond to the closure of the aortic valve. The third component of the pulse waveform is called the diastolic phase. The diastolic phase represents the outflow of blood into the peripheral circulatory system and is characterized by a second peak, which has a maximum pressure reached in the diastolic phase that corresponds to the diastolic peak (D). The shape of the pulse waveform is affected by multiple factors, such as hemodynamics and physiological conditions caused by changes in the characteristics of the arterioles.

[0050] Critical points are points of interest in the selected pulse waveform, corresponding to the maximum and minimum points, or the starting and ending points, of the pulse waveform, which may contain valuable physiologically relevant information related to the function of the heart. The critical points of the beginning (O) of the pulse waveform, the maximum value associated with the systolic peak (S), the minimum value associated with the dicrotic notch (N), the maximum value associated with the diastolic peak (D), and the end point (E) of the pulse waveform (corresponding to the O point of the subsequent pulse waveform) can be determined using methods such as those proposed by Dr. Elgendi. See Elgendi M. TERMA framework for biomedical signal analysis: An economic-inspired approach. J. Biosensors, 2016, 6 (4): 55. These methods are used for conventional analysis of pulse waveforms to derive physiologically relevant features for comparison with the low-dimensional representation of CNN analysis.

[0051] Other critical points can be identified by utilizing the first derivative of the PPG signal, known as velocity plethysmography (VPG). Critical points for the maximum positive velocity in systole (w), the minimum negative velocity in systole (y), and the maximum positive velocity in diastole (z) can be determined from the VPG signal. Similarly, the second derivative of the PPG signal, known as acceleration plethysmography (APG), can be used to derive critical points associated with the a-wave through the e-wave, referred to herein as a, b, c, d, and e. The identification of critical points associated with the pulse waveform depends on the application of the biomedical analysis technique.

[0052] In Dr. Elgendi's public text Elgendi M.TERMA framework for biomedical signal analysis: An economic-inspired approach. J. Biosensors, 2016, 6 (4): 55, a framework for the analysis of PPG biomedical signals is proposed. Although the analysis of biomedical signals (including PPG signals) has been developed for more than 20 years, this public text aims to develop a standardized framework for the analysis of biomedical signals. Methods such as those proposed by Dr. Elgendi can be used to determine critical points associated with pulse waveforms.

[0053] The exemplary embodiments described herein are provided for illustrative purposes, not restrictive. Other exemplary embodiments are possible, and modifications may be made to the exemplary embodiments within the spirit and scope of the present disclosure. Therefore, the specific embodiments are not intended to limit the present disclosure. More specifically, the scope of the present disclosure is limited only by the following claims and their equivalents. The following detailed description of the exemplary embodiments will be sufficient to fully reveal the overall nature of the present disclosure, so that others can easily modify and / or adapt such exemplary embodiments for a variety of applications by applying the knowledge of technicians in the relevant technical field without excessive experiments and without departing from the spirit and scope of the present disclosure. Therefore, such adaptations and modifications are intended to fall within the meaning of the exemplary embodiments based on the teachings and guidance proposed herein and the scope of many equivalents. It will be understood that the wording or terminology herein is for the purpose of description rather than limitation, so that the wording or phrases of this specification will be understood by technicians in the relevant technical field according to the teachings herein.

[0054] In some instances, analysis of the pulse waveform and identification of critical points associated with the pulse waveform may require the addition of a pre-processing filter (such as, but not limited to, an inverse Chebyshev filter or a Butterworth filter) to improve signal quality and to facilitate identification of specific critical points (such as the dicrotic notch (N)), which are difficult to detect in some pulse waveforms.

[0055] In one aspect, identification of critical points for the pulse waveform results in a set of 13 critical points derived from the PPG, VPG, and APG signals, where each critical point has an x-coordinate and a y-coordinate, such as Figure 2 As illustrated. The method for identifying critical points associated with a pulse waveform may be performed on each pulse waveform in a series of pulse waveforms obtained for the user, thereby deriving the x-coordinate and y-coordinate of each critical point of each pulse waveform in the series of pulse waveforms. This requires separation of each pulse waveform in the series, and zeroing of each separated pulse waveform. Alternatively, a series of pulse waveforms may be used to derive a representative pulse waveform, for example, a series of 30-second pulse waveforms may be used and aggregated, thereby obtaining a single representative pulse waveform for each 30-second series.

[0056] Methods such as, but not limited to, the quadratic moving average method as described by Dr. Elgendi may be used to determine the locations of critical points in the PPG, VPG, and APG signals that separate the pulse waveform, respectively. Analysis of the PPG pulse waveform (601) is illustrated in Figure 6 In. The quadratic moving average method requires the use of two aggregation windows of different sizes to calculate the moving average of the aggregation windows on the pulse waveform. The smaller aggregation window (referred to herein as W1) is an event window that captures the peak and elbow of the pulse, which corresponds to the moving average result calculated within the event window width (602), while the larger aggregation window (referred to herein as W2) is a period window that highlights the area containing the peak and elbow, and the moving average is calculated within the period window width (603). A smoothing convolution operation is applied to the midpoints of the moving aggregation windows of both W1 (604) and W2 (605). To further illustrate how this method can be applied, consider a certain point at the beginning of the pulse waveform, where the moving average window produces two different means because the windows (W1, W2) include different areas of the pulse waveform. Since the period window W2 (period window W2 is the larger of the two windows) takes into account part of the contraction peak area, the convolution of W2 will be larger than the convolution of the event window W1. As the two windows slide toward the systolic peak, W2 will include edge points that are closer to the start of the PPG pulse waveform than W1, resulting in a smaller convolution than that of W1, as shown in Figure 6 This can be seen in the first block of the first PPG pulse (607). In the block generated by this quadratic moving average method, the convolution of W1 is greater than the convolution of W2, and the block can be considered as a peak area or elbow area (606). Figure 6The application of the quadratic moving average method to two PPG pulses is illustrated, resulting in four peak areas and elbow areas. The first PPG pulse indicates the first peak area and the second peak area (607 and 608), while the second pulse indicates the peak area (609) corresponding to the systolic peak and the elbow area (610) corresponding to the diastolic peak. Critical points are included in the peak areas and elbow areas of PPG, VPG, and APG.

[0057] The ability to obtain the x- and y-coordinates of each critical point of each pulse waveform or representative pulse waveform in a series of pulse waveforms allows the derivation of other features describing the PPG signal. In an exemplary embodiment of the present invention, a series of amplitude features can be derived by recording the difference between the y-coordinates of any and all two critical points of the PPG, VPG, and APG, resulting in a total of 211 amplitude features, as discussed below.

[0058] like Figure 2 As shown, the critical point detected in one curve can be marked in the other two curves at the same time. For example, point w(209) can be marked in the PPG, and the difference between the y coordinates of point w(209) and point S(205) can be calculated as an amplitude feature. In the PPG pulse waveform, any two critical points except point O(204) and point E(208) (the amplitudes of point O and point E are normalized to 0 in the PPG) can generate an amplitude ratio feature. Through combined calculations from 11 critical points, there are 55 (11×10 / 2=55) developed amplitude features in the PPG pulse waveform. Similarly, in VPG or APG, any two critical points can generate an amplitude ratio feature. Since the y coordinates of the O(204) point and the E(208) point in the VPG(202) and APG(203) are not 0, there are 78 (13×12 / 2=78, calculated from the combination of 13 critical points) amplitude features developed in the VPG or APG pulse waveform. In total, 211 amplitude features are obtained. In addition, a series of time span features can be derived by recording the difference between the x coordinates of any and all two critical points, resulting in a total of 77 time span features (13×12 / 2-1=77, excluding the time span between the O(204) point and the E(208) point).

[0059] The pulse waveform can also be divided into sub-regions located between different critical points, wherein any two critical points can generate an area according to the pulse waveform sub-region. Each sub-region may or may not be normalized by the total area according to the entire pulse waveform, and each sub-region may be integrated using a method such as, but not limited to, numerical integration or an integration method based on the trapezoidal rule. The sub-region features derived from the integration of the sub-regions of the pulse waveform generate a total of 77 sub-region features through combined calculations from 13 critical points (13×12 / 2-1=77, excluding the total area between the O(204) point and the E(208) point). Another set of features (which may or may not be derived from this set of critical points) is the slope between any two critical points, which reflects the rate of change of the shape of the pulse waveform between the two specified critical points. Similar to the aforementioned combined calculation, this calculation generates a total of 77 (13×12 / 2-1=77, excluding the total area between the O(204) point and the E(208) point) slope features derived from the pulse waveform.

[0060] The method for determining amplitude features, time span features, sub-region features, and slope features can be used to derive a total of 445 features that provide a detailed description of the pulse waveform. These features can be used to derive a complete set of combinations and ratios of critical point features. On the one hand, this complete set of critical point features can be used in machine learning applications, such as but not limited to disease state monitoring (including cardiovascular-focused MLA), and general anomaly detection in the HWM industry, or other health-related applications that become available in the future. In addition, features selected from this complete set of critical point features have been shown to be strongly correlated with known health conditions or disease states, and these features can be provided to users or interested third parties to assess the health of the user. Some of these features will be discussed herein because some of these features are relevant to the present invention.

[0061] The systolic phase of the PPG pulse waveform represents cardiac output, which is the product of heart rate and stroke volume from the heart. Stroke volume is determined by left ventricular filling and left ventricular function. Rise time (defined as the time from the bottom (O) of the PPG pulse waveform to the systolic peak (S)) reflects the speed of left ventricular filling and how well the left ventricular function is performed. The stronger and more elastic the myocardium, the faster the left ventricle can inject the stroke volume into the aorta, so the rise time is shorter and the healthier the subject is. Rise time can be positively affected by youth and high-intensity exercise, which corresponds to better myocardial function in young and healthy users compared to older and unhealthy users. In addition, some cardiovascular diseases (such as but not limited to aortic stenosis and aortic regurgitation, as well as mitral valve abnormalities) may have an impact on rise time. Rise time may be of interest to the user or to an interested third party because it is usually an indication of myocardial function. It has been demonstrated that the normalized rise time as derived from the PPG signal of healthy individuals represents a relative mean value of less than 0.2, while individuals with acute myocardial infarction (AMI), chronic myocardial infarction (CMI), and antiphospholipid syndrome (SAA) have higher normalized rise time means. See, for example, Angius, Gianmarco, Doris Barcellona, ​​Elisa Cauli, Luigi Meloni, and Luigi Raffo, "Myocardial infarction and antiphospholipid syndrome: a first study on finger PPG waveforms effects." In 2012 Computing in Cardiology, pp. 517-520, IEEE, 2012. The rise time may be expressed as an absolute rise time as calculated using Formula 1, or as a normalized rise time as calculated using Formula 2.

[0062] Absolute rise time = (S x -O x ) [Formula 1]

[0063] Normalized rise time = (S x -O x ) / (E x -O x ) [Formula 2]

[0064] Another feature that is important to users and interested third parties in the HWM industry is the subendocardial myocardial vitality rate (SEVR), which is calculated as an estimated ratio of myocardial perfusion to cardiac workload and is calculated as the ratio of the diastolic pressure time index (DPTI) to the systolic pressure time index (SPTI). The physiological significance of SEVR depends on background knowledge of the cardiac cycle. The systemic circulation consists of an engine and two pumps. The first pump is the left ventricle, which represents the systolic pump. The second is the aorta and the large elastic arteries, which represent the diastolic pump. During systole, the left ventricle acts as a pump to push the blood stroke volume into the aorta, and the dilated aorta stores a portion of the stroke volume. Subsequently, the large elastic arteries act as another pump, pushing the stored stroke volume into other blood vessels during diastole. Due to the extravascular compression forces in the myocardium, the coronary arteries of the heart cannot be perfused during the contraction phase (systole). Therefore, subendocardial perfusion only occurs during the diastole phase of the cardiac cycle. The low pressure from the aorta provides an opportunity to pump blood into the coronary arteries to support heart function. The pressure represented by the area between the aortic and left ventricular pressure curves in diastole affects coronary blood flow and maintains adequate subendocardial blood supply during the diastole of the cardiac cycle. If the DPTI is small, it means that the diastolic blood pressure in the aorta is reduced, so the subendocardial perfusion is reduced. The less blood is supplied to the coronary arteries, the more the heart afterload increases, resulting in heart overload.

[0065] The area of ​​the left ventricular pressure waveform in systole (from the beginning of ventricular contraction to the dicrotic notch) represents the left ventricular afterload and defines the cardiac workload. In the case of a large mean arterial pressure in the ascending aorta during systole, the left ventricle must recoil more forcefully to maintain adequate stroke volume. Therefore, the systolic area describes myocardial oxygen demand and depends primarily on left ventricular ejection time, ejection pressure, and myocardial recoil force. The pressure represented by the area between the aortic and left ventricular pressure curves in diastole affects coronary blood flow and maintains adequate subendocardial blood supply during the diastole of the cardiac cycle. This indicates the degree of cardiac perfusion: due to high pressure, the heart cannot be perfused during recoil, but diastole with low pressure provides an opportunity to pump blood into the coronary arteries, which feed blood out of the bottom of the aorta. SEVR can be calculated based on sub-region features obtained from the PPG pulse waveform. The ratio of the sub-region features ON to NE in the PPG pulse waveform is used to calculate SEVR, as shown in Formula 3.

[0066] SEVR=DPTI / SPTI=sub-region(NE) / sub-region(ON[Formula 3]

[0067] The third feature indicating myocardial health is the ejection duration index, which is calculated as the normalized time span from the bottom (O) of the PPG pulse waveform to the dicrotic notch time (N). The left ventricular ejection duration is the time from the start of left ventricular retraction until the closure of the aortic valve, and is the duration of the systolic period. The ejection duration has been used to estimate left ventricular function and recoil force. It not only indicates the strength of the myocardium similar to the rise time, but also reflects the retraction and ejection function of the left ventricle. Ventricular failure can be caused by both very short ejection duration and long ejection duration. If the left ventricle is abnormally dilated, the left ventricular muscle will become thinner and weaker, resulting in more blood to be filled and a reduced contraction speed. Therefore, the systolic period will increase and produce a prolonged ejection duration. This is called systolic dysfunction.

[0068] In contrast, in diastolic dysfunction, the ventricular muscles are stiffer and thicker, and the ventricular capacity is reduced, resulting in less blood being ejected into the aorta. The duration of ejection in this case will be shorter than normal. In addition, some other cardiovascular diseases can increase ejection duration, such as aortic stenosis (Pagoulatou, Stamatia, Nikolaos Stergiopulos, Vasiliki Bikia, Georgios Rovas, Marc-Joseph Licker, Hajo Müller, Stéphane Noble, and Dionysios Adamopoulos. "Acute effects of transcatheter aorticvalve replacement on the ventricular-aortic interaction." American Journal of Physiology-Heart and Circulatory Physiology 319, no. 6 (2020): H1451-H1458.), aortic regurgitation (Kamran, Haroon, Louis Salciccioli, Carl-Frederic Bastien, Abhishek Sharma, and Jason M. Lazar. "The association between aortic regurgitation and increased arterial wave reflection." Artery Research 6, no.1(2012):49-54.), and ascending aortic aneurysm (Salvi, Lucia, Jacopo Alfonsi, Andrea Grillo, Alessandro Pini, Davide Soranna, Antonella Zambon, Davide Pacini, Roberto Di Bartolomeo, Paolo Salvi, and Gianfranco Parati. "Postoperative and mid-term hemodynamic changes after replacement of the ascending aorta." The Journal of Thoracic and Cardiovascular Surgery (2020)). Ejection duration and ejection duration index can be calculated using formulas 4 and 5, respectively.

[0069] Ejection duration = (N x -O x ) [Formula 4]

[0070] Ejection duration index = (N x -O x ) / (E x -O x ) [Formula 5]

[0071] The stiffness index is defined as the subject's height divided by the time difference between the systolic peak and the diastolic peak, where the time difference is calculated as the time span between the critical points S and D divided by the sampling rate, as described by Formula 6. As described above, the shape of the pulse waveform is determined by the left ventricle and the aorta. However, the relationship between the left ventricle and the aorta cannot explain all the phenomena that define the blood pressure and pulse waveform, and wave reflections also have an impact on the shape of the PPG pulse waveform detected at the limbs of the body. Considering the physiological relevance of the stiffness index, it may be convenient to associate this phenomenon with a basin filled with water and a series of concentric waves that travel from the center point of the basin to the edge. After hitting the outer edge, the first wave will move back toward the center of the basin. This reverse wave will be superimposed on the second centrifugal wave, resulting in a larger wave. In a similar manner, the forward wave generated by the heart pump travels along different pipelines (aorta, arteries, arterioles, capillaries, etc.).

[0072] Some typical reflection locations include arterial branches, atherosclerotic plaques, and terminal arterioles, which define systemic vascular resistance. At the reflection location, the reflected wave is generated and travels toward the heart to be superimposed on the forward wave. Because the speed of the pressure wave is very fast, the reverse wave is usually superimposed on the same forward wave that generates it. Due to this superposition, the blood pressure wave as measured in the PPG signal is a combination of a forward pressure wave and a reverse pressure wave, the forward pressure wave moves from the heart to the limbs, and the reverse pressure wave is reflected back toward the heart. The pulse wave speed is very fast, resulting in the application of the reflected wave being almost instantaneous. The time delay between the systolic peak and the diastolic peak is related to the passage time of the pressure wave, which is from the bottom of the subclavian artery to the obvious position of the reflection and returns to the subclavian artery, and this path length can be assumed to be proportional to the height of the subject. In the case of good elasticity of the aorta at the reflection location and low arterial stiffness, the reverse wave will reach the upper limb at a lower rate. However, in cases where the arterial stiffness of the aorta at the reflection location is high (e.g. if the subject is older), the blood stroke volume will be reflected back at a faster rate due to the reduced elasticity. This means that the pulse waveform of an older subject will have a shorter systolic peak (S) when compared to a younger subject. y ) to diastolic peak (D y) time and a higher stiffness index. This corresponds to an increase in the stiffness index as a function of time, as reported in the literature and illustrated by Equation 6.

[0073] Stiffness index = height / (D y -S y ) [Formula 6]

[0074] Pulse width is another feature of interest and is defined as the width of the pulse at half the height of the systolic pulse in the PPG pulse waveform. It has been proposed that pulse width is better positively correlated with systemic vascular resistance than systolic amplitude. See, for example, Awad, Aymen A., Ala S. Haddadin, Hossam Tantawy, Tarek M. Badr, Robert G. Stout, David G. Silverman, and Kirk H. Shelley. “The relationship between the photoplethysmographic waveform and systemic vascular resistance.” Journal of clinical monitoring and computing 21, no. 6 (2007): 365-372. Systemic vascular resistance describes the resistance to blood flow throughout the vascular system. The greatest amount of resistance comes from the arterioles and arterioles, which have very thick tunica media and adventitia.

[0075] Based on the Hagen-Poiseuille law, there are three key determinants of vascular resistance: blood viscosity, blood vessel length, and blood vessel radius. Total peripheral resistance is almost entirely due to changes in the diameter of small arteries and arterioles. The smaller the radius of a blood vessel, the greater the resistance to blood flow will be. In a normal blood vessel, the smooth muscle surrounding it can modulate its diameter to retract (vasoconstriction) or expand (vasodilation), so blood pressure and blood flow can be regulated by vascular resistance. The PPG pulse represents the changes in vascular volume as the blood stroke volume passes through the wrist / fingertip arterioles. For a blood pulse with a certain volume, arterioles with good elastic walls and muscles are more likely to expand, just like the aorta with good elasticity. Therefore, the pulse has a higher amplitude and a relatively shorter width, and the pulse speed through the artery is lower. In contrast, as the elasticity of the blood vessel wall and muscle decreases and the artery becomes stiff, the diameter of the arteriole becomes more difficult to modulate. Therefore, vascular resistance increases, and the blood stroke volume must pass through the blood vessel at an increased speed. Ultimately, the pulse will preferentially widen its width rather than increase its amplitude. When blood vessel walls are damaged, their ability to expand or contract to accommodate changes in blood flow becomes impaired. This damage often results in excessive resistance in that vessel, leading to further damage to the vessel, high blood pressure, or blocking blood flow to the vascular distribution area.

[0076] As mentioned in the discussion about SEVR, there are two actions during systole, the first is the isovolumetric contraction of the left ventricle, and the second is the ventricular ejection into the aorta and the aorta dilates and stores most of the stroke volume. When interpreting the APG pulse waveform, the a wave represents the acceleration of ventricular ejection, while the b wave mainly represents the acceleration of the "buffer" reduction. Due to the time delay between the heart and the detection point, the period of increased contraction matches the T peak in the ECG signal. Therefore, the a wave corresponds to the early moments during ventricular ejection after isovolumetric contraction. The size of the a wave corresponds to the strength of the myocardium. The larger the a wave, the stronger the myocardium. In contrast, the b wave is affected by aortic stiffness and arterial stiffness. In the case of high aortic stiffness and arterial stiffness, the amount of stroke volume stored in the aorta during systole is low, and most of the stroke volume is directly pressed toward the peripheral blood vessels. This suggests that the aortic pressure effect is reduced and the absolute value of the b wave is reduced. An increase in the b / a ratio indicates an increase in aortic stiffness and arterial stiffness, suggesting that the subject has hypertension. Hypertensive subjects have been observed to have higher (less negative) b / a ratios, as discussed in Zhang, Yahui, Zhihao Jiang, Lin Qi, Lisheng Xu, Xingguo Sun, Xinmei Chu, Yanling Liu, Tianjing Zhang, and Stephen E. Greenwald. “Evaluation of cardiorespiratory function during cardiopulmonary exercise testing in untreated hypertensive and healthy subjects.” Frontiers inphysiology 9(2018):1590.

[0077] Anomaly detection in time series data is a well-studied phenomenon with a wide range of techniques applied. Recently, with the development of larger amounts of Internet of Things (IOT) data recorded on human subjects and computer performance, opportunities to train more sophisticated and complex models have opened up, such as deep neural networks that detect physiological anomalies on time series data. Anomaly detection can include elements of unsupervised machine learning, supervised machine learning, and semi-supervised machine learning, thereby characterizing some regions in the input feature space, which correspond to normal physiology, known disease states or conditions, and unknown but more likely abnormal regions that fall outside the normal physiological range. There are some available literature on training supervised deep neural networks to detect specific conditions, such as atrial fibrillation (Afib), hypertension, heart rate, and other biometrics from PPG data. As an example of a deep neural network for PPG signal data, work performed by Andrew Ng's research group proposed a 50-layer neural network model (called ResNeXt) that can directly analyze the PPG pulse waveform and can achieve a 95% test AUC for atrial fibrillation detection. See Shen, Yichen, Maxime Voisin, Alireza Aliamiri, Anand Avati, Awni Hannun, and Andrew Ng. "Ambulatory Atrial Fibrillation Monitoring Using Wearable Photoplethysmography with Deep Learning." In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1909-1916. 2019. Recent literature has demonstrated the potential of deep neural networks in PPG pulse waveform analysis.

[0078] In one aspect, the present invention is based on the use of PPG pulse waveforms and their derivatives in combination with anomaly detection methods for identifying and monitoring healthy and unhealthy states on an integrated IOT platform spread across multiple devices, including but not limited to wrist-worn wearable devices, cellular smartphones, and cloud computing infrastructure for storage, computing, and communication with users or interested third parties.

[0079] Although anomaly detection is a broad field, specific embodiments of the present invention are used as specific examples of methodology. Convolutional neural networks (CNNs) are used as specific methods commonly used to analyze visual imaging, and have been used in applications such as image recognition, image classification, image segmentation, natural language processing, and brain-computer interfaces. CNNs are also commonly referred to as translation-invariant or space-invariant artificial neural networks (SIANNs). This naming is derived from the shared weight architecture of the convolution kernel, which slides along the input features in a manner similar to the event window and period window described for the quadratic moving average method, and the convolution kernel provides a translation-equivariant response as a feature map. CNN is a regularized form of a multilayer perceptron, in which each neuron in a single layer is connected to all neurons in the next layer. Its drawback is that this full connectivity tends to overfit the data, and relies on increasing regularization to prevent such overfitting. These regularization methods may include, but are not limited to, utilizing hierarchical patterns in the data and assembling patterns of increased complexity by using smaller and simpler patterns engraved in the CNN filter.

[0080] On the one hand, CNN is at the lower extreme with respect to connectivity and complexity of neural networks, and uses relatively little preprocessing compared to other image classification algorithms, making it ideal for real-time monitoring applications, such as those expressed in the present invention. Rather than hard-designing the kernel, network learning optimizes the kernel through automated learning, making it independent of prior knowledge and human intervention to extract features. CNN is often used as part of a supervised learning approach, meaning that a CNN can be constructed by training the network with data having specific labels associated with the data, for example in this case, PPG signal data of healthy and non-healthy participants can be presented to the network along with an indication that the participant is healthy, or in the case of a participant being non-healthy, the indication indicates the disease or condition the participant is suffering from. This allows the CNN to extract features that will highlight the differences in different health states.

[0081] For example, consider the application of CNN in analyzing PPG pulse waveforms for identifying Afib, where several different features in the pulse waveform correspond to health status of Afib participants as well as non-Afib participants, as discussed in Shen, Yichen, Maxime Voisin, Alireza Aliamiri, Anand Avati, Awni Hannun, and Andrew Ng. "Ambulatory atrial fibrillation monitoring using wearable photoplethysmography with deeplearning." In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1909-1916. 2019. In one aspect, a latent space representation or manifold is generated by feature extraction using such a CNN device, which can be constructed during training on pulse waveform data. Training can be done in a supervised, semi-supervised, or unsupervised configuration. For the supervised configuration, the CNN is part of a classification network trained on a pulse waveform dataset from normal individuals, individuals at different parts of the healthy range, and individuals with one or more diagnoses of cardiovascular conditions or diseases. For the unsupervised configuration, the CNN can be trained as part of a variational autoencoder architecture or a generative adversarial network (GAN), both of which are trained on unlabeled data.

[0082] Once trained, the CNN can be used to generate outputs (410) in its latent space manifold (see Figure 4) (output layer before the classification stage), the output is the output that has not yet been transformed into a class classification. As the next step of anomaly detection, a technique that reduces the output in a latent space manifold is applied, typical techniques such as PCA (Principal Component Analysis) or t-SNE (t-distributed-stochastic neighbor embedding) (412-413). The output of this step will be a low-dimensional (e.g., two-dimensional (2D) or three-dimensional (3D)) visualization (414), and a set of domains in this space are then defined to coincide with where the available data sets appear when processed through the CNN and dimensionality reduction techniques (PCA or t-SNE). Specific domains will correspond to specific conditions or disease states, such as data from people diagnosed with sleep apnea (415) or atrial fibrillation (417), normal health states for individuals with no diagnosis of chronic diseases across age and gender demographics (416), and abnormal health states that fall outside of disease states and normal health states (418).

[0083] To provide more specific information about this method of creating a low-dimensional space to classify data in this way, the t-SNE embedding method is described in detail. The t-SNE method can be combined with the high-dimensional final convolutional layer output vector (e.g., 1024-dimensional CNN output) to transform the CNN output into two-dimensional data points in the following way: similar samples are clustered into clusters of nearby points, while non-similar data points are modeled as other points or clusters farther away. This results in clusters formed on the manifold, where each cluster corresponds to a specific health state. Considering that N PPG signals can be processed so as to output the PPG signal (x1,…,x N ) of N CNN high-dimensional objects, the t-SNE method can be used to calculate the probability (p ij ), the probability (p ij ) indicates that any two objects x i With x j This can be done using Equation 7 and Equation 8.

[0084] p i / j =exp(-||x i -x j || 2 / 2σ i 2 ) / ∑ k≠i exp(-||x i -x k || 2 / 2σ i 2 ) [Formula 7]

[0085] p ij =(p i / j +pi / j ) / 2N [Formula 8]

[0086] Formula 7 calculates the conditional probability p i|j To evaluate object x j With x i The similarity of x i With x j The degree of proximity of i Their probability density is in a Gaussian distribution centered on . In addition, the Gaussian kernel (σ i ) can be set in a way that the conditional distribution is equal to the predefined perplexity. The goal of the t-SNE technique is to learn the d-dimensional graph (y1,…,y N ), the d-dimensional graph is similar to the probability of two objects (p ij ) reflects the similarity (q ij ). Therefore, the similarity of two objects is i With y j is measured between , which can be done using Equation 9.

[0087] q ij =(1+||y i -y j || 2 ) -1 / ∑ k ∑ l≠k (1+||y k -y l || 2 ) -1 [Formula 9]

[0088] Formula 9 can be used to measure two low-dimensional points (y i With y j ) ij ), thus allowing non-similar objects to be modeled as separate from each other in the manifold. For each point y in the manifold i The location of can be determined by minimizing the asymmetric Kullback-Leibler divergence of distribution P with distribution Q, as illustrated in Equation 10. This results in a low-dimensional representation manifold that reflects the similarity between high-dimensional inputs.

[0089] KL(P||Q)=∑ i≠j p ij log(p ij / q ij ) [Formula 10]

[0090] In one aspect, additive loss functions (such as, but not limited to, triplet loss, mean squared error loss, and cross entropy loss) may be used to organize the manifold and to create continuity among classes of feature sets. A loss function (also referred to as a cost function) maps the value of one or more features or predictions to a real number that intuitively represents the cost or loss associated with the placement of the feature or set of features within the manifold. When performing optimization, the goal would be to minimize the loss function to ensure that feature sets describing a particular health state are not incorrectly assigned in the manifold. For example, considering using a triplet loss as a loss function, a triplet may be formed by deriving an anchor input (A), a positive input (P), and a negative input (N), wherein the positive input (P) describes the same health state as the anchor, and the negative input (N) describes a different health state than the anchor. The inputs may then be run through the network, and the outputs may be used in the loss function. This may be accomplished by passing the loss function This is done by describing the Euclidean distance function, as shown in Equation 11. The Euclidean distance function can then be used in the cost function , as detailed in Equation 12.

[0091]

[0092] In Formula 11, α refers to the spacing between positive and negative pairs, and f refers to the embedding function. Adding a loss function allows the data points distributed on the manifold to be organized so as to be classified according to the multiple health states represented in the manifold. The organization of the manifold will produce regions on the manifold corresponding to known healthy states and known unhealthy states, while regions within the manifold that cannot be placed in the organized regions, for example due to lack of training on data diversity, may contain unknown health states.

[0093] In such Figure 1In one aspect of the invention shown, a PPG signal may be obtained from a wearable device (101) such as, but not limited to, a fingertip pulse oximeter, an earlobe pulse oximeter, and in a smart wearable device such as, but not limited to, a wrist-worn device and a smart ring. Using methods such as, but not limited to, a Butterworth filter, an inverse Chebyshev filter, or other high-pass and low-pass filters commonly understood by those skilled in the art, desired pre-processing (such as, but not limited to, detrending and denoising) may be applied to the PPG signal (102) to improve signal quality. In some aspects, knowing the fundamental frequency of the PPG signal may help pre-process or denoise the signal. With respect to the description of this exemplary embodiment of the invention, a mean filter with, for example, a sampling window of 50 may be used as a high-pass filter (103), thereby allowing only signal data above 0.5 Hz to pass through the filter. This may be followed by a low-pass filter with, for example, a Gaussian filter (e.g., order 3 and standard deviation 0.8) that only allows signals less than 6 Hz to pass through the filter (104).

[0094] The detrended and denoised PPG signal can be used for automated feature extraction using a CNN device, as well as for conventional extraction of interpretable engineered one-dimensional (1D) features, including but not limited to rise time, SEVR, ejection duration, large artery stiffness index, small artery resistance, and features related to hypertension, which have been described in the literature. According to one aspect, with respect to automated feature extraction using a CNN device, the PPG signal is divided into equal time span segments (105), such as 30 second segments. These PPG segments can then be normalized by using a minimum-maximum scaling method to scale the amplitude of the PPG segments to a range from 0 to 1 (106), thereby generating a one-dimensional representation of the PPG signal data, such as in Figure 4 Alternatively, a two-dimensional input data set may be generated from the PPG signal by generating a frequency domain representation of the PPG signal (such as, but not limited to, a Fourier spectrum, a Lomb-Scargle periodogram) or by cardiopulmonary coupling (107), as in Figure 4 Described in more detail in .

[0095] A CNN is constructed for classification of the health status of a participant (108). The high dimensional output from the CNN apparatus can be used with methods such as t-SNE and triplet loss to automatically construct a low dimensional representation manifold having feature sets corresponding to different possible health states present in the participant (109). This training apparatus allows the assignment of unknown health states to participants to be classified in the manifold, thereby identifying any potential health risks that the participant may have (110). Thus, the derived feature set of a new participant (e.g., a user) can be derived using the CNN apparatus, and the feature set can be placed on the manifold to assess the health status of the new participant (111).

[0096] Where the participant's data corresponds to a feature set described in the manifold (112) and the feature set corresponds to a health state presented in the manifold (113), then positive feedback may be reported to the participant or an interested third party indicating that the participant is healthy (114).

[0097] In the event that a participant corresponds to a feature set that deviates from a healthy feature set as described in the manifold, the assignment of the feature set in the manifold, as expressed in the manifold, may help identify which health issues may be present in the participant.

[0098] In the event that the participant's feature set does not correspond to a feature set of known unhealthy states (115), as expressed in the manifold, the participant or an interested third party may be provided feedback regarding a specific unhealthy state that corresponds to the participant's feature set (116).

[0099] With interpretability engineering one-dimensional (1D) features (e.g., Figure 3 Feedback (derived from conventional analysis of the PPG signal shown) may also be provided to the user or interested third party (117).

[0100] In the case where the feature set does not correspond to any feature set expressing in the manifold that describes a known healthy state or an unhealthy state and corresponds to an unknown region in the manifold, then the feature set corresponding to this participant can be classified as an unknown health state of interest within the manifold. The resulting feature set can then be compared to interpretable engineering one-dimensional (1D) features, such as those derived by conventional methods of pulse waveform analysis.

[0101] On the other hand, a PPG signal may be obtained from a wearable device (101). Required preprocessing may be applied to the PPG signal (102) to improve the signal quality for identifying critical points. Similar to the above-mentioned exemplary embodiment of the present invention, a mean filter with a sampling window of 50 is used as a high-pass filter (103), followed by a Gaussian filter (order 3 and standard deviation 0.8) as a low-pass filter (104). Each pulse waveform may be separated from the PPG signal (119), and may be followed by zeroing of the pulse waveform. This is followed by determination of the first and second derivatives of the pulse waveform, thereby obtaining the pulse waveforms of PPG, VPG, and APG, respectively. Methods such as, but not limited to, the quadratic moving average method as described by Elgendi may be applied to each PPG, VPG, and APG pulse waveform (121) for determining critical points associated with the PPG, VPG, and APG pulse waveforms (122).

[0102] The critical points can be used to determine a complete set of features (123) derived from the differences in the locations of the critical points. Amplitude (124), time span (125), sub-region (126), and slope features (127) can be derived to generate this complete set of features, and these features can be normalized (128) and used to derive interpretable engineered one-dimensional (1D) features (129), including but not limited to rise time, SEVR, ejection duration, large artery stiffness index, small artery resistance, and features related to hypertension, which have been described in, for example, Figure 3 Described in the text illustrated.

[0103] As stated in the foregoing exemplary embodiments of the present invention, the CNN device may obtain a feature set for an unknown health state that is not equivalent to a known healthy state or a known unhealthy state described in the manifold (118). Interpretable engineering one-dimensional (1D) features may be derived for participants who may have a feature set of unknown health states to determine whether there are potential health problems in the participants (130), and will be discussed in more detail below. In the event that the interpretable engineering one-dimensional (1D) features show that the participant is healthy, the contrast information may be used to improve the understanding of the CNN manifold by expanding the accepted feature set corresponding to the health state (131). In addition, positive feedback may be reported to the participant or an interested third party, indicating that the participant is healthy (114). In the event that an interpretable engineering one-dimensional (1D) feature indicates that there may be a potential health issue, the specific interpretable engineering one-dimensional (1D) feature may be recorded as being associated with a specific feature in the manifold as obtained by the CNN device (132), and the participant or interested third party may be notified that the participant has a specific interpretable engineering one-dimensional (1D) feature that indicates a health issue (117).

[0104] In one aspect, conventional methods of analyzing PPG signals for deriving critical points can be used to determine interpretable engineered one-dimensional (1D) features for comparison with automatically derived CNN features. The PPG pulse waveform (201) can be used to determine five critical points, such as Figure 2 The first and second derivatives of the PPG signal can be derived to obtain VPG (202) and APG (203), thereby determining the remaining eight critical points.

[0105] The critical points corresponding to the start O (204) of the PPG pulse waveform, the systolic peak S (205), the dicrotic notch N (206), the diastolic peak D (207), and the end E (208) of the PPG pulse waveform can be determined from the PPG signal. The critical points corresponding to the maximum positive velocity w (209) of the systolic period, the minimum negative velocity y (210) of the systolic period, and the maximum positive velocity z (211) of the diastolic period can be determined from the VPG signal. The critical points corresponding to the a wave to the e wave in the APG signal (herein referred to as critical point a (212), critical point b (213), critical point c (214), critical point d (215), and critical point e (216)) can be determined from the APG signal.

[0106] In one aspect, the quadratic moving average method disclosed by Elgendi can be used to identify blocks in PPG, VPG, and APG pulse waveforms for determining the locations of critical points. The block containing each particular critical point can be identified, and the maximum or minimum point in each block of interest will correspond to the critical point associated with that particular block. Using this method, the x- and y-coordinates of each critical point of a given pulse waveform can be determined.

[0107] Comparison of engineered one-dimensional (1D) features for interpretability (as derived by conventional analysis of PPG signals) with the CNN derived feature set requires a clear understanding of these features and their classification of healthy versus unhealthy states. The ability to obtain the x- and y-coordinates of each critical point in a series of pulse waveforms or a representative pulse waveform allows the derivation of additional features describing the PPG signal.

[0108] In one aspect, the critical points derived from PPG, VPG, and APG can be used to extract a complete set of features (301), such as Figure 3 As shown. The difference (302) between the y coordinates of any two critical points can be calculated to derive the amplitude feature (303) corresponding to the pulse waveform. The difference (304) between the x coordinates of any two critical points can be calculated to derive the time span feature (305) corresponding to the pulse waveform. PPG, VPG and APG pulse waveforms can be divided into sub-partitions between different critical points, wherein any two critical points can generate an area (306) based on the pulse waveform sub-partition. Each sub-area may or may not be normalized by the total area based on the entire pulse waveform, and each sub-area can be integrated using a method such as but not limited to numerical integration or an integration method based on the trapezoidal rule to derive the sub-area feature (307). The difference between the x coordinate and the y coordinate of any two critical points can be calculated (308) to derive the slope feature (309) corresponding to the pulse waveform.

[0109] There are some disclosures in the literature that focus on deriving health-related and interpretable engineered features (310) from a complete set of amplitude, time span, sub-region, and slope features. These interpretable engineered one-dimensional (1D) features have been associated with disease and anatomy in the scientific literature, and are all interpretable and useful in understanding the risk of disease development. Health-related and interpretable features (311) can be calculated for a representative population of human participants, and methods such as but not limited to dimensionality reduction techniques or PCA for building a covariance matrix can be used to remove interpretable engineered one-dimensional (1D) features, or select engineered one-dimensional (1D) features that preserve unique health-related information (312).

[0110] Allowing for the construction of a covariance matrix, which generalizes the concept of variance to multiple dimensions. This allows the elimination of interpretable engineered one-dimensional (1D) features with high covariance to reduce the set of engineered features to only those that do not covary with other features, thereby identifying features that preserve unique information about cardiovascular health and anatomy. On the one hand, PCA can be used to eliminate features that do not contain unique health-related information, which is the process of calculating the principal components of a data set and performing basic transformations on the data using these principal components. This results in a small subset of interpretable engineered one-dimensional (1D) features related to independent aspects of health and anatomy (313). Mathematical formulas for calculating this small subset of features are available in the scientific literature, and these mathematical formulas can be used to derive interpretable engineered features (314). Health-related features can be derived through conventional analysis of the PPG signal, such as but not limited to rise time (315), normalized rise time (316), SEVR (317), ejection duration (318), ejection duration index (319), stiffness index (320), b / a ratio (321), pulse wave velocity (PWV) (322) and augmentation index (323) as well as future available interpretable engineered one-dimensional (1D) features.

[0111] In one aspect, the PPG signal can be used to generate an input data sequence for a CNN deep learning algorithm. This is followed by a discussion of some data preparation methods that need to be considered. A series of PPG pulse waveforms are collected (e.g., 3000 pulses), which can then be sorted and concatenated along a timeline to obtain a sequence of 3000 pulses. This pulse sequence can then be divided into a series of smaller sequences of fixed length, such as 100 smaller sequences of 30 pulses per sequence. These smaller sequences can then be iteratively fed into the CNN for training. The PPG signal can also be divided into 30 second segments of equal time span (105).

[0112] After detrending and denoising, the PPG segments may be normalized by using a min-max scaling method to scale the amplitude of the PPG segments to a range from 0 to 1, thereby generating a one-dimensional representation of the PPG signal data, while the time scale remains unnormalized (401), as shown in Figure 4As shown. This one-dimensional data sequence will be used as an input data set for the CNN (402). In one aspect, an alternative input data set can be generated from the PPG signal by generating a frequency domain representation of the PPG signal (such as but not limited to a Fourier spectrum, a Lomb-Scargle periodogram) or as a two-dimensional representation of the pulse waveform through cardiopulmonary coupling (403). The CNN deep learning algorithm can then be trained on the frequency domain representation of the PPG signal instead of the PPG signal input for constructing the CNN manifold for identifying the health state (404). Regardless of whether the PPG signal or the frequency domain representation of the PPG signal is used as the input data, the input data can be used to construct the CNN and train the CNN to derive a feature set to identify the user's health state (405).

[0113] To train the CNN, a convolution procedure (406) is performed on the input data, followed by a subsampling procedure (407). Each procedure generates a series of feature maps (408), thereby associating features according to the user's health state. Multiple steps including convolution may be included, and a subsampling step (409) may be required before a sufficiently good CNN can be derived. The latent space output (410) from the fully trained CNN model and the classification output (411) can then be used to construct a low-dimensional manifold by incorporating methods such as, but not limited to, principal component analysis or t-SNE, thereby generating a lower dimensional projection on the manifold (412). In addition, the use of a loss function (e.g., a triplet loss (413)) allows identification of multiple health states in the manifold, thereby allowing multiple health states on the manifold to be mapped (414).

[0114] On the one hand, Figure 5 As shown, the system is implemented on an IOT device (501), which includes a wearable device (502) for monitoring PPG signals, such as but not limited to a fingertip pulse oximeter (503), an earlobe pulse oximeter (504), a wrist-worn wearable device (505), and a smart ring (506). The PPG signal data (507) obtained from the wearable device will require a memory (508), which may include a remote data storage device (509) (such as but not limited to a hard disk (510), a flash drive (511), or an external hard disk (512)), and a cloud infrastructure (513) for storing data (such as but not limited to Google Drive (514) or Amazon Web Services (AWS) S3 storage bucket (515)).

[0115] The computing operations will be performed on an apparatus (516) that relies on remote computing (517) on a device such as, but not limited to, a personal computer (518), a smartphone (519), and a wearable device (520). A cloud infrastructure may be used to implement the computing operations (521) using services such as, but not limited to, AWS (522). For the purpose of providing feedback about the user's health status to the user or an interested third party (523), a number of user interfaces (524) may be utilized, including a device with a screen display (525) to report feedback to the user, such as, but not limited to, a desktop computer display (526), ​​a smartphone display (527), and a wearable device display (528). Alternative communication methods (529) may be used to provide feedback about the user's health status to the user or an interested third party, such as, but not limited to, a phone call (530), text message (531), email communication (532), or through a web-based dashboard (533).

[0116] In one aspect, the present invention provides a method for detecting early changes in cardiovascular health using a wrist-worn wearable device. A monitored user having a stable rise time feature (the rise time feature initially maintains a stable value) will be classified by a neural network (411) that is used to analyze the user's features as being part of a "normal" category (416) on which the model is trained. However, if the user's rise time begins to change due to early signs of heart failure (because the disease affects the heart muscle), the rise time feature will gradually increase along with other cardiovascular features derived from the wearable device (such as heart rate and heart rate variability). This change will follow a pattern that moves the user from a healthy portion (416) of a lower dimensional manifold (412) (created using a model using t-SNE or PCA) toward a direction on the lower dimensional manifold (412) associated with heart failure. When this becomes a statistically significant deviation from the deviation seen for healthy users, the system generates an alert that can be sent to a care provider (523) to perform further follow-up and determine the cause of this observation.

[0117] Additionally, the system may provide a comprehensive method for detecting cardiovascular changes that are statistically significantly distinguishable from changes observed in training data of healthy people (405). Comprehensive in this context means that when the PPG derived pulse waveform data (entered Figure 4) are projected down to a 2-dimensional plane (414) created by t-SNE or PCA, and when the user either moves within the 2D plane toward the outer edge of the healthy region (416) or crosses outside of the healthy region (416), the system triggers an alarm and interpretable features of the pulse waveform (523) (e.g., heart muscle strength via rise time, coronary perfusion via SEVR, aortic health via augmentation index) are shared with the care provider for further investigation.

[0118] In an exemplary embodiment of the present invention, consider obtaining a PPG signal of a user with an unknown cardiovascular health state. The PPG signal is fed into a CNN device to assess the user's health state. Then, consider that the results obtained from the CNN analysis of the PPG signal indicate that the user's PPG signal is not associated with a healthy state, but the user's PPG signal cannot be placed in a known unhealthy state when mapped on the CNN manifold. Through conventional analysis of the user's PPG signal, the large artery stiffness index, PWV, and augmentation index have been shown to be higher than normal. Given the results obtained through conventional analysis of the pulse waveform, a literature search can be performed or a medical professional can be consulted to infer that these trends are consistent with patients with hypertrophic cardiomyopathy (HCM), which is a thickening of the heart muscle with a genetic origin. Feedback on health issues can then be provided to the user and interested third parties such as medical professionals, and the manifold of the CNN can be extended to include this health state. In addition, the user's health state related to the progression of the disease can be monitored over time and reported to the user or their medical practitioner so that preventive intervention measures can be performed.

[0119] In one aspect, consider obtaining a PPG signal for a user with unknown cardiovascular health status. The PPG signal is fed into a CNN device to assess the user's health status, and the resulting feature set corresponds to the unknown region in the CNN manifold. By computing interpretable and physiologically relevant features of the pulse waveform (the features characterize healthy individual anatomical aspects such as large vessel health, small vessel health, blood pressure, myocardial health (discussed previously)), it is possible to classify the abnormalities generated for this individual as relevant underlying diseases, because the reduced myocardial health pulse waveform features are associated with abnormalities.

[0120] The user's health status may be notified to the user or an interested third party (such as a medical practitioner). Considering that the medical practitioner may recommend lifestyle changes to the user, the medical practitioner may be interested in continuously monitoring the user's cardiac health status (as derived by the pulse waveform abnormality detection system) to evaluate the efficacy of the recommended lifestyle changes. The results from the pulse waveform abnormality detection system may be made available to the medical practitioner through a web-based dashboard that reports a summary of the user's health status over time, thereby allowing the user to be remotely monitored. Similarly, the medical practitioner may be interested in using such a web-based dashboard to remotely monitor multiple patients. This will enable the medical practitioner to frequently survey the health status of multiple patients and make changes to their medical care when necessary without having to interact personally with each patient.

[0121] Although several aspects have been disclosed in the foregoing description, it will be appreciated by those skilled in the art that many modifications and other aspects related to the present disclosure will be contemplated with the benefit of the foregoing description and the teachings presented in the associated drawings. Therefore, it will be understood that the present disclosure is not limited to the particular aspects disclosed above, and that many modifications and other aspects are intended to be included within the scope of any claim capable of reciting the disclosed subject matter.

[0122] It should be emphasized that the aspects described above are only possible embodiments of the embodiments, which are only stated for a clear understanding of the principles of the present disclosure. Any process description or process block in the flow chart should be understood as a module, section or part representing a code, and the code includes one or more executable instructions for implementing a specific logical function or step in the process, and alternative embodiments are included, wherein the function is not included or not executed at all, and it can depend on the functions involved and not be executed (including substantially simultaneously or in reverse order) in the order shown or discussed, as can be reasonably understood by those skilled in the art of the present disclosure. Many changes and modifications can be made to the aspects described above, without substantially departing from the essence and principles of the present disclosure. In addition, the scope of the present disclosure is intended to cover any and all combinations and sub-combinations of all elements, features and aspects discussed above. All such modifications and changes are intended to be included in the scope of the present disclosure, and all possible claims for the combination of various aspects or elements or steps are intended to be supported by the present disclosure.

Claims

1. A method for identifying a user's health status using an IoT device with interconnected electronic devices and sensors, the method comprising: (a) Using wearable devices to collect users’ PPG signals; (b) preprocessing the PPG signal for conventional analysis of the PPG signal and for extracting critical points and interpretable engineering one-dimensional (1D) features, such as but not limited to rise time, SEVR, ejection duration index, large artery stiffness index, small artery resistance, and features related to hypertension; (c) using an anomaly detection system exemplified by a convolutional neural network trained on pulse waveform data to generate feature vectors in a latent space; (d) using a dimensionality reduction method to construct a low-dimensional representation (two-dimensional or three-dimensional) of the feature vector; (e) labeling partitions of the two-dimensional / three-dimensional space as corresponding to a healthy condition, a specific condition, or unknown based on the category assigned to the PPG signal by the anomaly detection system; (f) creating interpretable engineered 1D features that refer to specific physiological processes associated with health risks; (g) using the explainable engineering features along with the healthy / disease / unknown output of the anomaly detection system to interpret an “unknown” anomaly as healthy or non-healthy based on whether the explainable engineering features have a 1D value associated with a healthy condition or have a 1D value associated with a specific condition; as well as (h) providing feedback to the user or a third party regarding the user's health status in the event of an unknown label output.

2. The method of claim 1, wherein conventional analysis of the PPG signal can be used to: (a) determining the critical points associated with the PPG pulse waveform and its derivative; (b) calculating a complete set of features from the critical points, wherein the critical points associated with the PPG, VPG and APG signals are determined using a quadratic moving average method; (c) determining health-related and interpretability engineering one-dimensional (1D) features from the complete set of features, the complete set of features being calculated from the difference between any two critical points in the form of amplitude, time span, sub-region, and slope features; and (d) Derive a small subset of interpretable features related to independent aspects of health and anatomy.

3. The method of claim 2, wherein the interpretable engineered one-dimensional (1D) features include rise time, normalized rise time, SEVR, ejection duration, ejection duration index, large artery stiffness index, arteriolar resistance, and features associated with hypertension, wherein the features associated with hypertension are derived from a complete set of features calculated from the x-coordinates and y-coordinates of the critical points.

4. A method according to claim 2, wherein the technique for establishing a covariance matrix or principal component analysis is used to remove interpretable engineering one-dimensional (1D) features with covariance, or is used to select interpretable engineering one-dimensional (1D) features that preserve unique health or anatomical related information.

5. The method of claim 1, wherein the convolutional neural network: (a) is constructed from a one-dimensional or two-dimensional representation (via frequency domain methods such as Fourier spectrum) of the PPG signal; (b) configured to extract a feature set from the PPG signal; (c) is constructed to be used to classify these feature sets into health state categories; as well as (d) is constructed for placing the health states in a low-dimensional representation manifold.

6. A method according to claim 5, wherein input data derived from segments of the PPG signal are pre-processed and converted into a one-dimensional representation of the PPG signal, or are used to derive a two-dimensional frequency domain representation of the PPG, or are used through cardiopulmonary coupling, the two-dimensional frequency domain representation of the PPG comprising a Fourier spectrum, a Lomb-Scargle periodogram.

7. The method according to claim 5, wherein the one-dimensional representation or the two-dimensional representation of the PPG signal is used as input data to train the CNN and thereby obtain an output feature set, which is mapped on a low-dimensional representation framework when using a method including t-SNE or PCA.

8. The method of claim 5, wherein a loss function is used to organize the manifold and create continuity among the feature set categories by identifying healthy state regions in the low-dimensional representation manifold, wherein the loss function comprises a triplet loss, a mean squared error loss, or a cross entropy loss.

9. The method of claim 1, wherein a set of user CNN features corresponding to unknown regions in the low-dimensional representation manifold are compared with interpretable engineered one-dimensional (1D) features to assess the user's health status, wherein the interpretable engineered one-dimensional (1D) features contain unique health and anatomical information derived through conventional analysis of the PPG signals.

10. The method of claim 9, wherein the unknown regions in the low-dimensional representation manifold are assigned based on interpretable engineered one-dimensional (1D) features that contain unique health and anatomical information derived by conventional analysis of PPG signals corresponding to those unknown regions.

11. The method of claim 1 , wherein feedback is provided to the user and interested third parties using a display including a desktop computer display, a laptop computer display, a smartphone display, a wearable device display, a phone call, a text message, an email, or a web-based dashboard.

12. The method of claim 1, wherein the computing aspects of the invention are performed remotely on a device such as but not limited to a smart wearable device, a smartphone, a desktop computer, a laptop computer, or can be performed using a cloud computing infrastructure.

13. A method for identifying a user's health status using an IoT device with interconnected electronic devices and sensors, the method comprising: (a) Using wearable devices to collect users’ PPG signals; (b) pre-processing the PPG signal for conventional analysis of the PPG signal, thereby: (i) extracting at least one interpretable engineered one-dimensional (1D) feature from the PPG signal; (c) using an anomaly detection system on the PPG signal to produce a feature vector in a latent space and thereby produce a classification of the PPG signal corresponding to a healthy state or a state associated with another condition; (d) constructing a low-dimensional representation of the feature vector; (e) labeling low-dimensional representations of the PPG signal corresponding to states classified by the anomaly detection system; (f) associating the interpretability engineered 1D features with the classification in the low-dimensional representation space; (g) interpreting anomalies marked as unknown as belonging to a healthy state or associated with another condition; and (h) providing feedback to the user or a third party regarding the user's health status.

14. The method of claim 15, wherein the wearable device comprises one of a fingertip pulse oximeter, an earlobe pulse oximeter, a wrist-worn wearable device, or a smart ring.

15. The method of claim 15, wherein the interpretable engineered 1D features include rise time, SEVR, ejection duration index, large artery stiffness index, small artery resistance, and features related to hypertension.

16. The method of claim 15, wherein the anomaly detection system comprises a convolutional neural network trained on PPG signal data from other users.

17. The method of claim 15, wherein constructing a low-dimensional representation comprises applying PCA or t-SNE to the feature vector.

18. The method of claim 1, wherein the wearable device comprises a fingertip pulse oximeter, an earlobe pulse oximeter, a wrist-worn wearable device, or a smart ring.

Images