System and method for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence

Abstract

A system and method for contactless determination of pulse-wave velocity or pulse-transit time from a received image sequence. The method including determining one or more tracked points on the body-part; tracking displacement of the one or more tracked points over a measurement period; determining a ballistocardiogram signal by determining a derivative of the displacement of the one or more tracked points; determining a pulse wave signal of the subject captured in the image sequence based on bit values from a set of bitplanes that represent hemoglobin concentration changes; determining the pulse-transit time or the pulse-wave velocity by determining a phase shift between the pulse wave signal and the ballistocardiogram signal; and outputting the pulse-transit time or the pulse-wave velocity.

Description

SYSTEM AND METHOD FOR CONTACTLESS DETERMINATION OF PULSE-WAVEVELOCITY OR PULSE TRANSIT TIME FROM A RECEIVED IMAGE SEQUENCETECHNICAL FIELD

[0001] The following relates generally to determination of human vital signs and more specifically to a system and method for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence.BACKGROUND

[0002] The human heartbeat, or cardiac cycle, represents one of the primary vital signs monitored by health care providers and members of the general public alike. Understanding how the pulsed blood from the heartbeat travels around the body is useful for a number of useable health metrics. For example, pulse transit time (PTT) can be determined which is a measure of the time taken for an arterial pulse pressure wave to travel between two sites or regions. In another example, pulse-wave velocity (PWV) can be determined as a measurement of arterial stiffness, which can be useful as a predictor of cardiovascular risk.SUMMARY

[0003] In an aspect of the present invention, there is provided a method for contactless determination of pulse-wave velocity or pulse-transit time from a received image sequence, the method executed on one or more processors, the method comprising: receiving an image sequence capturing a body-part of a subject; determining one or more tracked points on the body-part; tracking displacement of the one or more tracked points over a measurement period; determining a ballistocardiogram signal by determining a derivative of the displacement of the one or more tracked points; determining a pulse wave signal of the subject captured in the image sequence based on bit values from a set of bitplanes that represent hemoglobin concentration changes; determining the pulse-transit time or the pulse-wave velocity by determining a phase shift between the pulse wave signal and the ballistocardiogram signal; and outputting the pulse-transit time or the pulse-wave velocity.

[0004] In a particular case of the method, each tracked point comprising edges or boundaries of the body-part located in the image sequence.

[0005] In another case of the method, tracking displacement comprises tracking vertical displacement.

[0006] In yet another case of the method, tracking displacement comprises performing an optical flow technique.

[0007] In yet another case of the method, determining the ballistocardiogram signal comprises determining both first derivative of the displacement of the one or more tracked points.

[0008] In yet another case of the method, determining the ballistocardiogram signal comprises a determining a standard deviation of the first derivative and comparing the standard deviation to a predetermined threshold.

[0009] In yet another case of the method, determining the pulse wave signal comprises passing the bit values from the set of bitplanes into a trained deep learning machine learning model.

[0010] In yet another case of the method, determining the pulse-transit time or the pulse-wave velocity comprises determining a frequency of interest within a heart band range of the pulse wave signal.

[0011] In yet another case of the method, the method further comprising converting between the pulse-transit time and the pulse-wave velocity using a distance factor.

[0012] In yet another case of the method, the distance factor is determined by measuring a shortest distance between a boundary of the body-part and a location of a region of interest.

[0013] In another aspect, there is provided a system for contactless determination of pulsewave velocity or pulse-transit time from a received image sequence, the system comprising one or more processors and a data storage, the data storage comprising instructions for the one or more processors to execute: an imaging module to receive an image sequence capturing a body-part of a subject; a ballistocardiogram module to determine one or more tracked points on the body-part, to track displacement of the one or more tracked points over a measurement period, and to determine a ballistocardiogram signal by determining a derivative of the displacement of the one or more tracked points; a blood flow module to determine a pulse wave signal of the subject captured in the image sequence based on bit values from a set of bitplanes that represent hemoglobin concentration changes; a pulse metric module to determine the pulse-transit time or the pulse-wave velocity by determining a phase shift between the pulse wave signal and the ballistocardiogram signal; and an output module to output the pulse-transit time or the pulse-wave velocity.

[0014] In a particular case of the system, each tracked point comprising edges or boundaries of the body-part located in the image sequence.

[0015] In another case of the system, tracking displacement comprises tracking vertical displacement.

[0016] In yet another case of the system, tracking displacement comprises performing an optical flow technique.

[0017] In yet another case of the system, determining the ballistocardiogram signal comprises determining both first derivative of the displacement of the one or more tracked points.

[0018] In yet another case of the system, determining the ballistocardiogram signal comprises a determining a standard deviation of the first derivative and comparing the standard deviation to a predetermined threshold.

[0019] In yet another case of the system, determining the pulse wave signal comprises passing the bit values from the set of bitplanes into a trained deep learning machine learning model.

[0020] In yet another case of the system, determining the pulse-transit time or the pulse-wave velocity comprises determining a frequency of interest within a heart band range of the pulse wave signal.

[0021] In yet another case of the system, the pulse metric module converts between the pulsetransit time and the pulse-wave velocity using a distance factor.

[0022] In yet another case of the system, the distance factor is determined by measuring a shortest distance between a boundary of the body-part and a location of a region of interest.

[0023] These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The features of the invention will become more apparent in the following detailed description in which reference is made to the appended drawings wherein:

[0025] FIG. 1 is a block diagram of a system for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence, according to an embodiment;

[0026] FIG. 2 is a flowchart for a method for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence, according to an embodiment;

[0027] FIG. 3 illustrates re-emission of light from skin epidermal and subdermal layers;

[0028] FIG. 4 is a set of surface and corresponding transdermal images illustrating change in hemoglobin concentration for a particular human subject at a particular point in time;

[0029] FIG. 5 is an illustration of bitplanes for a three channel image; and

[0030] FIG. 6 illustrates an example of a face captured to determine pulse wave signals and ballistocardiogram signals.DETAILED DESCRIPTION

[0031] Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0032] Various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: “or” as used throughout is inclusive, as though written “and / or”; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender; “exemplary” should be understood as “illustrative” or “exemplifying” and not necessarily as “preferred” over other embodiments. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.

[0033] Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable / executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.

[0034] The following relates generally to determination of human vital signs and more specifically to a system and method for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence using ballistocardiography signals.

[0035] In embodiments of the present disclosure, technical approaches are provided to determine pulse-transit-time (PTT), which can be used to determine pulse-wave velocity (PWV), without having to contact a subject with measurement equipment. The PTT and PWV determinations are performed using image processing techniques. For example, the PTT determination can be performed on a subject using a general purpose imaging device, such as by a video camera, or using a previously recorded video.

[0036] In some cases of the embodiments of the present disclosure, a received video used to determine the PTT can comprise a video, or series of images, of a subject’s facial area. In other cases, a received video can comprise a series of images of an extremity of the subject’s body that has exposed vascular surface area; for example, the subject’s palm. Generally, each captured portion of the subject may require a separately trained model, as described herein. For the purposes of the following disclosure, reference will be made to capturing the subject’s face; however, it is understood by a person skilled in the art that other areas of the subject can be used with the approach described herein.

[0037] Generally, PWV is a measure of the speed at which pressure waves move along a blood vessel. PWV can be used for a number of purposes, such as used as indicator of arterial stiffness and cardiovascular health. PWV generally quantifies the velocity of the pressure wave generated by the heartbeat as it travels along the arteries.

[0038] PTT, also known as pulse-wave travel time, is the time taken for a pulse wave to travel between two different arterial sites. These two arterial sites are usually separated by a distance on the same or related artery.

[0039] PTT and / or PWV can provide valuable information about a person’s vascular physiology. For example, PTT and PWV can be directly used to determine other health metrics, such as blood pressure. For example, generally when blood pressure increases, the arteries become stiffer, and the pulse wave travels faster, resulting in a shorter PTT. Conversely, when blood pressure decreases, the arteries are less stiff, and the pulse wave travels slower, resulting in a longer PTT.

[0040] PTT can be measured from the time it takes a pulse wave to travel between two regions, such as between the heart and the ROI. PTT is inversely related to PWV and can be calculated as:where d is the distance between the two points determined for PTT.

[0041] Any suitable source for determining the distance can be used; for example, using the height of the person to approximate the distance between the heart and the ROI, a different factor of height to approximate the distance of the ROI from the heart, or a combination of weight and height, such as BMI, to approximate the distance. In this way, a measure that contains information about pulse wave velocity is used with the distance approximation that can be person-specific.

[0042] Other types of PTT measurement involve determining the time between specific fiducial points of two different signals; e.g., R-wave of an ECG and peak of a pulse wave from an arterial site.

[0043] Embodiments of the present disclosure also advantageously use ballistocardiography (BCG). BCG is a method for measuring the mechanical activity of the heart by recording the subtle movements of the body that result from ballistic forces of the heart due to sudden ejection of blood with each heartbeat. In each cardiac cycle, and more specifically, following each heartbeat, the sudden ejection of blood into vessels generates a ballistic force, a small amount of force that moves along and causes a recoil in the body. These movements are very subtle, and yet detectable. The recoil movements are generally waveform-like. Advantageously,embodiments of the present disclosure measure PTT with BCG using a time difference between a blood flow signal at an arterial site and a BCG waveform at the arterial site.

[0044] Referring now to FIG. 1, a system for contactless determination of pulse metrics from videos using machine learning models 100 is shown in accordance with an embodiment. The system 100 includes a processing unit 108, one or more video-cameras 103, a storage device 101 , and an output device 102. The processing unit 108 may be communicatively linked to the storage device 101, which may be preloaded, periodically loaded, and / or continuously loaded with video imaging data obtained from one or more video-cameras 103. The processing unit 108 includes various interconnected elements and modules, including an imaging module 110, a ballistocardiogram module 112, a blood flow module 114, a pulse metric module 116, and an output module 118. In further embodiments, one or more of the modules can be executed on separate processing units or devices, including the video-camera 103 or output device 102. In further embodiments, some of the features of the modules may be combined or run on other modules as required.

[0045] In some cases, the processing unit 108 can be located on a computing device that is remote from the one or more video-cameras 103 and / or the output device 102, and linked over an appropriate networking architecture; for example, a local-area network (LAN), a wide-area network (WAN), the Internet, or the like. In some cases, the processing unit 108 can be executed on a centralized computer server, such as in off-line batch processing.

[0046] The term “video”, as used herein, can include sets of still images. Thus, “video camera” can include a camera that captures a sequence of still images and “imaging camera” can include a camera that captures a series of images representing a video stream.

[0047] In a particular case, using transdermal optical imaging (TOI), the imaging module 110 can isolate hemoglobin concentration (HC) from raw images taken from the digital camera 103. Referring now to FIG. 3, a diagram illustrating an example of re-emission of light from skin is shown. Light 301 travels beneath the skin 302, and re-emits 303 after travelling through different skin tissues. The re-emitted light 303 may then be captured by the optical cameras 103. The dominant chromophores affecting the re-emitted light are melanin and hemoglobin. Since melanin and hemoglobin have different color signatures, it has been found that it is possible to obtain images mainly reflecting HC under the epidermis as shown in FIG. 4.

[0048] Using transdermal optical imaging (TOI), the imaging module 110 obtains each captured image in a video stream, from the camera 103, and performs operations upon theimage to generate a corresponding optimized hemoglobin concentration (HC) image of the subject. From the HC data, the facial blood flow localized volume concentrations can be determined. The image processing unit 104 isolates HC in the captured video sequence. In an exemplary embodiment, the images of the subject’s faces are taken at 30 frames per second using a digital camera 103. It will be appreciated that this process may be performed with alternative digital cameras, lighting conditions, and frame rates.

[0049] In a particular case, isolating HC can be accomplished by analyzing bitplanes in the sequence of video images to determine and isolate a set of the bitplanes that approximately maximize signal to noise ratio (SNR) for the blood flow and pulse in a region of interest. Bitplanes are a fundamental aspect of digital images. Typically, a digital image consists of certain number of pixels (for example, a width X height of 1920X1080 pixels). Each pixel of the digital image having one or more channels (for example, color channels red, green, and blue (RGB)). Each channel having a dynamic range, typically 8 bits per pixel per channel, but occasionally 10 bits per pixel per channel for high dynamic range images. Whereby, an array of such bits makes up what is known as the bitplane. In an example, for each image of color videos, there can be three channels (for example, red, green, and blue (RGB)) with 8 bits per channel. Thus, for each pixel of a color image, there are typically 24 layers with 1 bit per layer. A bitplane in such a case is a view of a single 1 -bit map of a particular layer of the image across all pixels. For this type of color image, there are therefore typically 24 bitplanes (i.e. , a 1 -bit image per plane). Hence, for a 1 -second color video with 30 frames per second, there are at least 720 (30X24) bitplanes. FIG. 5 is an exemplary illustration of bitplanes for a three-channel image (an image having red, green and blue (RGB) channels). Each stack of layers is multiplied for each channel of the image; for example, as illustrated, there is a stack of bitplanes for each channel in an RGB image. Using bitplanes provides a greater level of accuracy for making predictions of blood flow, and thus PTT and PWV determinations disclosed herein. Particularly, a greater accuracy is possible because employing bitplanes provides a greater data basis for training the machine learning model.

[0050] TOI signals can be taken from regions of interest (ROIs) of the video capturing the subject, for example, region 302 illustrated in FIG. 6; and can be defined manually or automatically for the video images. The ROI is preferably selected on the basis of areas that are particularly indicative of PTT / PWV measurement. Using the native images that consist of all bitplanes of all three R, G, B channels, signals that change over a particular time period (for example, 10 seconds) on each of the ROIs are extracted.

[0051] The raw signals can be pre-processed using one or more filters depending on the signal characteristics. Such filters may include, for example, a Butterworth filter, a Chebyshev filter, or the like. Using the filtered signals from the ROI, machine learning can be employed to systematically identify bitplanes that will significantly increase the signal differentiation (for example, where the SNR improvement is greater than 0.1 db) and bitplanes that will contribute nothing or decrease the signal differentiation. After discarding the latter, the remaining bitplane images can optimally determine blood flow generally associated with a determination of PTT / PWV.

[0052] Machine learning approaches (such as a Long Short Term Memory (LSTM) neural network, or a suitable alternative such as non-linear Support Vector Machine) and deep learning may be used to assess the existence of common spatial-temporal patterns of hemoglobin changes that are indicative of blood flow. The machine learning process involves manipulating the bitplane vectors (for example, 24 bitplanes X 30 fps) using the bit value in each pixel of each bitplane along the temporal dimension. In one embodiment, this process requires subtraction and addition of each bitplane to maximize the signal differences in the ROI over a time period. In some cases, to obtain reliable and robust computational models, the entire dataset can be divided into three sets: the training set (for example, 80% of the whole subject data), the test set (for example, 10% of the whole subject data), and the external validation set (for example, 10% of the whole subject data). The time period can vary depending on the length of the raw data (for example, 15 seconds, 60 seconds, or 120 seconds). The addition or subtraction can be performed in a pixel-wise manner. An existing machine learning algorithm, the Long Short Term Memory (LSTM) neural network, or a suitable alternative thereto is used to efficiently and obtain information about the improvement of differentiation in terms of accuracy, which bitplane(s) contributes the best information, and which does not in terms of feature selection. The Long Short Term Memory (LSTM) neural network allows the system to perform group feature selections and classifications. The set of bitplanes to be isolated from image sequences to reflect temporal changes in HC is obtained for determination of blood flow in the ROI for determining PTT / PWV.

[0053] To extract facial blood flow data, facial HC change data on each pixel of the image can be extracted as a function of time when the subject is being viewed by the camera 103. The ROI can selected according to, for example, the subject’s underlying physiology, such as by the autonomic nervous system (ANS) regulatory mechanisms. The ROIs can be manually selected or automatically detected with the use of a face tracking. As an example, the system 100 canmonitor stationary HC changes contained by a selected ROI over time by observing (or graphing) the resulting temporal profile (for example, shape) of the selected ROI HC intensity values over time. In some cases, the system 100 can monitor more complex migrating HC changes across multiple ROIs by observing (or graphing) the spatial dispersion (HC distribution between ROIs) as it evolves over time.

[0054] In this way, an ROI is selected that provides a suitable waveform signal. For example, an ROI with a good signal-to-noise ratio (SNR), such as but not limited to on a person’s cheek or forehead.

[0055] Thus, it is possible to obtain a video sequence of a subject and apply the HC extracted from selected bitplanes to models to determine blood flow, which can then be used to determine PTT / PWV, as described herein. For long running video streams with changes in blood flow and intensity fluctuations, changes of the estimation and intensity scores over time relying on HC data based on a moving time window (e.g., 10 seconds) may be used.

[0056] In this way, the TOI signal can make use of the optical sensors / cameras to capture subtle changes in the light reflected from the skin surface; such as sensors / cameras of smartphones and digital cameras. TOI identifies slight variations in the light absorption and reflection, which correspond to different blood flow patterns. TOI, as used in the embodiments of the present disclosure, is particularly advantageous because it allows for non-contact, remote monitoring, making it highly useful for telemedicine and health monitoring applications. It provides a convenient and non-invasive way to gather health data, which can be particularly beneficial for continuous monitoring in clinical and consumer health scenarios.

[0057] The present embodiments advantageously provide contactless determination of PTT and PWV by using a determined time difference of a TOI signal in the ROI against a corresponding BCG signal.

[0058] The TOI signal is identified at the ROI, as described herein, such as by capturing a remote photoplethysmography (rPPG) signal in the face of the subject. The BCG signal can be determined using a remote visual BCG of movement of the subject determined by identifying sudden recoils of the visual peripherals of the subject’s captured body part (e.g., head).

[0059] Turning to FIG. 2, a flowchart for a method 200 for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence or pulse transit time from a received image sequence is shown, in accordance with an embodiment.Advantageously, the method 200 does not require any expert-driven manual signal processing or feature engineering.

[0060] At block 202, the imaging module 110 receives an input image sequence capturing a human subject from the camera 103 and / or the storage device 101. The image sequence can be in the form of a raw video.

[0061] At block 204, in some cases, the imaging module 110 preprocesses the images in the image sequence. For example, compressing the images / videos to lower resolution to decrease size whole ensuring no significant loss of information. As another example, anonymizing the identity of the subjects for legal or regulatory compliance, or other privacy concerns.

[0062] The ballistocardiogram module 112 detects ballistocardiogram signals from the body part (e.g., face) of the captured subject using the image sequence. At block 206, the ballistocardiogram module 112 identifies specific points on the body part, such as edges and boundaries, using any suitable image processing approach; such as using a suitable trained machine learning model, gradient-based approaches, Laplacian approaches, Canny edge detection, or a combination thereof. At block 208, the ballistocardiogram module 112 tracks displacement (e.g., vertical displacement) in the image sequence, frame-by-frame, over a duration of a measurement period (for example, 30 seconds). For example, using an optical flow technique such as the Kanade-Lewis-Tomase (KLT) algorithm.

[0063] At block 210, the ballistocardiogram module 112 determines the ballistocardiogram signal by determining a velocity of the displacement of the captured body part. To obtain the BCG waves (which is generally a periodic signal), first and second derivatives of the displacement can be determined. In many cases, only the first derivative representing velocity is sufficient. In most cases, rather than observing a velocity threshold, the ballistocardiogram module 112 can examine a standard deviation (SD) of the first derivative (velocity) and compare the SD to preset appropriate thresholds. If the SD is too high, it would indicate that the determined velocity is not BCG-related, and may be a movement artefact.

[0064] In some cases, the BCG signal can be detrended to remove noise due to other signals using, for example, moving average subtraction, polynomial fitting, or the like. For example, higher harmonics from breathing, or other movement artifacts, could be in the frequency band of the heart rate and interfere with the BCG signal.

[0065] In some cases, at block 212, the ballistocardiogram module 112 applies signal processing approaches to the BCG signal, such as fast-Fourier-transform (FFT) and a Hamming window.

[0066] At block 214, the blood flow module 114 uses a machine learning (ML) model to detect pulse wave signals at the ROI (also referred to as determining the TOI signal) taking the received images as input.

[0067] The ML models used by the blood flow module 114 can be any suitable approach. For example, deep learning (DL) models such as convolutional neural networks (CNNs). In other cases, a trained ensemble of deep DL models can be used, including CNNs and deep neural networks (DNNs), such as multi-layer perceptrons (MLPs). Generally, the choice of ML model used by the blood flow module 114, also the non-linear function used in the selected ML model, and the levels of complexity in the model’s architecture need to be determined during training of the model (e.g. more layers in the CNN and / or DNN, additional skipped connections in the CNN, or the like). The models can be trained using supervised learning, where each input video has a labeled set of ground truths corresponding to the vitals that are to be determined. Models can be trained on numerous training videos; for example, thousands of videos. After training, models can be validated for their accuracy and generalizability using a combination of approaches that include k-fold cross validation, performance tuning on separated validation sets, and final performance checks on pristine test sets that represent field data.

[0068] At block 216, the blood flow module 114 applies signal processing approaches to the TOI signal, such as fast-Fourier-transform (FFT) and a Hamming window.

[0069] At block 218, in some cases, the pulse metric module 116 determines a frequency of interest within a heart band range of the TOI signal determined by the blood flow module 114. In an example, the heart rate band can be between 0.7 to 3 Hz. In some cases, the pulse metric module 116 determines a signal-to-noise (SNR) ratio of the BCG signal determined by the ballistocardiogram module 112 and / or the TOI signal determined by blood flow module 114 within the frequency band of interest. At block 220, the pulse metric module 116 determines the pulse-transit time as a phase shift between the TOI signal determined by the by the blood flow module 114 and the BCG determined by the ballistocardiogram module 112. In some cases, the pulse metric module 116 can determine the pulse-wave velocity in addition to or instead of the pulse-transit time metric. While a phase-offset is preferably used, any suitable approach for determining the time offset between the BCG signal and the TOI signal can be used.

[0070] In some cases, the pulse metric module 116 identifies a distance factor d to convert between PTT and PWV. The distance factor d can be determine by, for example, measuring the shortest distance between the visible peripheral boundary of the body of the subject, and the location of the ROI. The raw measured distance may be adjusted using one or more correction factors depending on the raw measurement and additional factors such as physiological factors of the subject, such as age, sex, height, weight, or ethnic background. In some cases, the distance factor d is determined by non-visual approaches, such as, for example, by relying on profile factors or physiological factors of the subject.

[0071] At block 220, in some cases, the pulse metric module 116 applies quality criteria to ensure validity of the determined pulse-wave velocity / pulse-transit time. In some cases, the quality criteria can be based on identifying if the SNR of either the BCG signal and TOI signal, individually, are above a predetermined threshold. In some cases, together, the quality criteria can be based on identifying if the SNR of a correlation of both the BCG signal and TOI signal is above a predetermined threshold.

[0072] At block 222, the output module 118 outputs the pulse-wave velocity, the pulsetransit time metric, or both, to the output device 102 and / or the storage device 101.

[0073] In this way, the method 200 allows for the determination of PTT by detecting the BCG signal, which takes the form of the head vibrating from the shock wave of the heartbeat, and measure the time (e.g., in millseconds) it takes for the TOI signal (i.e. the pulsatile or AC waveform) to arrive at the ROI. FIG. 6 illustrates an example of a face 300 captured by the camera 103. The BCG signal is captured by the face moving between a first position 304 and a second position 306. The region of interest 302 is used to capture the TOI signal. Having both the BCG signal and the TOI signal allows for the determination of PTT, as described herein.

[0074] In an example, the system 100 can identify signal patterns such as R-waves, S- waves, or any other pulse peak or wave form indicators, for the BCG signal and the TOI signal. By comparing the appearance time of similar pattern features, such as R-waves, in each one of the signals (BCG signal and TOI-based pulse wave signal) the RR-interval (the time between each pulse) can be measured, which inversely corresponds to the heartbeat rate.

[0075] In an example, the system 100 identifies a first peak time as the time of an BCG R- wave peak in the BCG signal timeline. The model then identifies a second peak time as the time of the next TOI signal R-wave peak after the identified first peak time. If the time difference between the two identified peak times are less than RR-interval, the PTT is calculated as thetime difference between the two peak time. Otherwise, the system 100 can select the next BCG R-wave peak after the previously selected first peak time and repeat the determination.

[0076] In other cases, the first peak time is identified using TOI R-wave signals instead of BCG signals and the second peak time is identified using BCG signals.

[0077] Referring back to FIG.2, at block 214, the blood flow module 114 feeds the preprocessed videos and the identified PTT to the ML models to determine PWV. In some cases, the blood flow module 114 outputs the identified BCG signal to the output device 102 and / or the storage device 101.

[0078] The determined PWV and / or PTT can be used for any suitable downstream application, such as for the determination of blood pressure. A comparison of a group of hypertensive people against a group of normotensive people, it would generally be expected that the TOI signal to arrive earlier for the hypotensive group than for the normotensive group, taking into account physiological differences between each person. In both cases, the shock wave from BCG would arrive at the face / head earlier than the TOI signal. As such, this time difference is likely to be lower in the hypertensive group, taking into account other physiological differences between each person.

[0079] The ML models, described herein, can use training examples that comprise inputs comprising images from videos captured of human body parts and known outputs (ground truths) of described signals at described body points. The known ground truth values can be captured using any suitable device / approach; for example, a pulse oximeter, an electrocardiogram (ECG), a ballistocardiogram, a plethysmography sensor, a sphygmomanometer, or the like. The relationship being approximated by the machine learning model is pixel data from the video images to pulse wave determination; whereby this relationship is generally complex and multi-dimensional. Through machine learning training, such a relationship can be outputted as vectors of weights and / or coefficients. The trained machine learning model being capable of using such vectors for approximating the input and output relationship between the video images input data and the determined vital sign information. In this way, advantageously, the ML models take the multitude of training sample videos, and corresponding ground truth of measured values, and learn which features of input videos are correlated with which vital signs. Thus, an ML model can be generated that can determine and identify pulse waves given a raw video of a person, such as a video of their face, as input.

[0080] In some cases, optical sensors pointing, or directly attached to the skin of any body parts such as for example the wrist or forehead, in the form of a wrist watch, wrist band, hand band, clothing, footwear, glasses, helmet, or appliances, or components of vehicles such as steering wheel may be used. From these body areas, the system 100 may also determine PWV. In some cases, the system may be installed in robots and their variables (e.g., androids, humanoids) that interact with humans to enable the robots to detect PWV on the face or other- body parts of humans whom the robots are interacting with.

[0081] The embodiments described herein may be applied to a plurality of fields. For example, they may be executed on a smartphone device to allow a user of the smartphone to measure their PWV. In another example, the system may be located in a hospital room in association with a video camera to allow the hospital staff to monitor the PWV of a patient without causing the patient discomfort by having to attach a device to the patient.

[0082] Other applications may become apparent.

[0083] Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

CLAIMS1. A method for contactless determination of pulse-wave velocity or pulse-transit time from a received image sequence, the method executed on one or more processors, the method comprising: receiving an image sequence capturing a body-part of a subject; determining one or more tracked points on the body-part; tracking displacement of the one or more tracked points over a measurement period; determining a ballistocardiogram signal by determining a derivative of the displacement of the one or more tracked points; determining a pulse wave signal of the subject captured in the image sequence based on bit values from a set of bitplanes that represent hemoglobin concentration changes; determining the pulse-transit time or the pulse-wave velocity by determining a phase shift between the pulse wave signal and the ballistocardiogram signal; and outputting the pulse-transit time or the pulse-wave velocity.

2. The method of claim 1, wherein each tracked point comprising edges or boundaries of the body-part located in the image sequence.

3. The method of claim 1, wherein tracking displacement comprises tracking vertical displacement.

4. The method of claim 1, wherein tracking displacement comprises performing an optical flow technique.

5. The method of claim 1, wherein determining the ballistocardiogram signal comprises determining both first derivative of the displacement of the one or more tracked points.

6. The method of claim 5, wherein determining the ballistocardiogram signal comprises a determining a standard deviation of the first derivative and comparing the standard deviation to a predetermined threshold.

7. The method of claim 1, wherein determining the pulse wave signal comprises passing the bit values from the set of bitplanes into a trained deep learning machine learning model.

8. The method of claim 1, wherein determining the pulse-transit time or the pulse-wave velocity comprises determining a frequency of interest within a heart band range of the pulse wave signal.

9. The method of claim 1, further comprising converting between the pulse-transit time and the pulse-wave velocity using a distance factor.

10. The method of claim 9, wherein the distance factor is determined by measuring a shortest distance between a boundary of the body-part and a location of a region of interest.

11. A system for contactless determination of pulse-wave velocity or pulse-transit time from a received image sequence, the system comprising one or more processors and a data storage, the data storage comprising instructions for the one or more processors to execute: an imaging module to receive an image sequence capturing a body-part of a subject; a ballistocardiogram module to determine one or more tracked points on the bodypart, to track displacement of the one or more tracked points over a measurement period, and to determine a ballistocardiogram signal by determining a derivative of the displacement of the one or more tracked points; a blood flow module to determine a pulse wave signal of the subject captured in the image sequence based on bit values from a set of bitplanes that represent hemoglobin concentration changes; a pulse metric module to determine the pulse-transit time or the pulse-wave velocity by determining a phase shift between the pulse wave signal and the ballistocardiogram signal; and an output module to output the pulse-transit time or the pulse-wave velocity.

12. The system of claim 11, wherein each tracked point comprising edges or boundaries of the body-part located in the image sequence.

13. The system of claim 11, wherein tracking displacement comprises tracking vertical displacement.

14. The system of claim 11, wherein tracking displacement comprises performing an optical flow technique.

15. The system of claim 11, wherein determining the ballistocardiogram signal comprises determining both first derivative of the displacement of the one or more tracked points.

16. The system of claim 15, wherein determining the ballistocardiogram signal comprises a determining a standard deviation of the first derivative and comparing the standard deviation to a predetermined threshold.

17. The system of claim 11, wherein determining the pulse wave signal comprises passing the bit values from the set of bitplanes into a trained deep learning machine learning model.

18. The system of claim 11, wherein determining the pulse-transit time or the pulse-wave velocity comprises determining a frequency of interest within a heart band range of the pulse wave signal.

19. The system of claim 11, wherein the pulse metric module converts between the pulsetransit time and the pulse-wave velocity using a distance factor.

20. The system of claim 19, wherein the distance factor is determined by measuring a shortest distance between a boundary of the body-part and a location of a region of interest.

Images