System and method for determination of pulse-transit time of a subject from a received image sequence

Abstract

A system and method for contactless determination of pulse-transit time from a received image sequence. The method including: determining hemoglobin concentration (HC) signals over time at a first region of interest (ROI) and a second ROI, the first ROI and the second ROI forming a pair of ROIs, the second ROI located downstream from the first ROI along a common pulse wave path; determining a pulse transit time based on a time difference between peaks in the HC signals; and outputting the pulse transit time. The method can further include determining pulse wave velocity (PWV) using the determined PTT and a distance between the pair of ROIs, and outputting the determined PWV. The method can further include determining an estimate of blood pressure using the determined PWV with a trained machine learning model, and outputting the determined estimate of blood pressure.

Description

SYSTEM AND METHOD FOR DETERMINATION OF PULSE-TRANSIT TIME OF A SUBJECTFROM A RECEIVED IMAGE SEQUENCETECHNICAL FIELD

[0001] The following relates generally to estimation of human blood pressure and more specifically to a system and method for contactless determination of pulse-transit time of a subject from a received image sequence.BACKGROUND

[0002] The human heartbeat, or cardiac cycle, represents one of the primary vital signs monitored by health care providers and members of the general public alike. Understanding how the pulsed blood from the heartbeat travels around the body is useful for a number of useable health metrics. For example, pulse transit time (PTT) can be determined which is a measure of the time taken for an arterial pulse pressure wave to travel between two sites or regions. In another example, pulse-wave velocity (PWV) can be determined as a measurement of arterial stiffness, which can be useful as a predictor of cardiovascular risk.SUMMARY

[0003] According to an aspect of the invention, there is provided A method for contactless determination of pulse-transit time from a received image sequence, the method executed on one or more processors, the method comprising: receiving an image sequence capturing a body-part of a subject; determining hemoglobin concentration (HC) signals over time at a first region of interest (ROI) and a second ROI, the first ROI and the second ROI forming a pair of ROIs, the second ROI located downstream from the first ROI along a common pulse wave path; determining a pulse transit time based on a time difference between peaks in the HC signals; and outputting the pulse transit time.

[0004] In a particular case of the method, the method further comprising determining pulse wave velocity (PWV) using the determined PTT and a distance between the pair of ROIs, and outputting the determined PWV.

[0005] In another case of the method, the method further comprising determining an estimate of blood pressure using the determined PWV with a trained machine learning model, and outputting the determined estimate of blood pressure.

[0006] In yet another case of the method, the machine learning model further takes, as input, selected features derived from the respective HC signals of the pair of ROIs and demographic information about the human subject.

[0007] In yet another case of the method, the trained machine learning model is a linear regression model.

[0008] In yet another case of the method, the method further comprising determining an estimate of mean arterial blood pressure, based on an assumed range for the ratio of arterial vessel diameter over arterial wall thickness, using a Moens-Korteweg equation.

[0009] In another aspect, there is provided a system for contactless determination of pulsetransit time from a received image sequence, the system comprising one or more processing units and a data storage memory, the data storage memory comprising instructions for the one or more processing units to execute: a transdermal optical imaging module to receive an image sequence capturing a body-part of a subject and determine hemoglobin concentration (HC) signals overtime at a first region of interest (ROI) and a second ROI, the first ROI and the second ROI forming a pair of ROIs, the second ROI located downstream from the first ROI along a common pulse wave path; a pulse module to determine a pulse transit time based on a time difference between peaks in the HC signals; and an output module to output the pulse transit time.

[0010] In a particular case of the system, the pulse module further determines pulse wave velocity (PWV) using the determined PTT and a distance between the pair of ROIs, and the output module further outputs the determined PWV.

[0011] In another case of the system, the one or more processing units further execute a blood pressure module to determine an estimate of blood pressure using the determined PWV with a trained machine learning model, and the output module further outputs the determined estimate of blood pressure.

[0012] In yet another case of the system, the machine learning model further takes, as input, selected features derived from the respective HC signals of the pair of ROIs and demographic information about the human subject.

[0013] In yet another case of the system, the trained machine learning model is a linear regression model.

[0014] In yet another case of the system, the output module further outputs an estimate of mean arterial blood pressure, based on an assumed range for the ratio of arterial vessel diameter over arterial wall thickness, using a Moens-Korteweg equation.

[0015] These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features of the invention will become more apparent in the following detailed description in which reference is made to the appended drawings wherein:

[0017] FIG. 1 is a block diagram of a system for contactless determination of pulse-wave velocity or pulse transit time from a received image sequence, according to an embodiment;

[0018] FIG. 2 illustrates re-emission of light from skin epidermal and subdermal layers;

[0019] FIG. 3 is a set of surface and corresponding transdermal images illustrating change in hemoglobin concentration for a particular human subject at a particular point in time;

[0020] FIG. 4 is a set of surface and corresponding transdermal images illustrating change in hemoglobin concentration for a particular human subject at a particular point in time;

[0021] FIG. 6 shows an example of the system of FIG. 1 as applied to two regions of interest on a subject’s face;

[0022] FIG. 7 shows a chart of hemoglobin concentration (HC) signals for respective regions of interest (ROIs) of a corresponding pair lying along a common artery in the subject’s face; and

[0023] FIG. 8 is a flowchart of a method for estimating blood pressure of a human subject, according to an embodiment.DETAILED DESCRIPTION

[0024] Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In otherinstances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0025] Various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: “or” as used throughout is inclusive, as though written “and / or”; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender; “exemplary” should be understood as “illustrative” or “exemplifying” and not necessarily as “preferred” over other embodiments. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.

[0026] Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD- ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable / executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.

[0027] In embodiments of the system and method described herein, technical approaches are provided to solve the technological problem of determining pulse-transit time and pulsewave velocity, such as for the determination of human blood pressure, without having to contact the subject with a blood pressure measurement instrument. Blood pressure is determined using image processing techniques performed over a plurality of images captured by one or more digital imaging cameras, such as a digital video camera.

[0028] The technical approaches described herein offer the substantial advantages of not requiring direct physical contact between a subject and a blood pressure measurement instrument. As an example of a substantial advantage using the technical approaches described herein, measurement can be performed on a subject using a suitable imaging device, such as by a video camera communicating over a communications channel. As another example of a substantial advantage using the technical approaches described herein, measurements can be determined from previously recorded video material.

[0029] Referring now to FIG. 1 , a system for contactless determination of pulse-transit time of a subject from a received image sequence 100 is shown. The system 100 includes a processing unit 108, one or more video-cameras 103, a storage device 101, and an output device 102. The processing unit 108 may be communicatively linked to the storage device 101 , which may be preloaded, periodically loaded, and / or continuously loaded with video imaging data obtained from one or more video-cameras 103. The processing unit 108 includes various interconnected elements and modules, including a TOI module 110, a pulse module 112, a blood pressure module 114, a first filter module 116, a profile module 120, and an output module 126. The TOI module 110 includes an image processing unit 104. The video images captured by the video-camera 103 can be processed by the image processing unit 104 and stored on the storage device 101. In further embodiments, one or more of the modules can be executed on separate processing units or devices, including the video-camera 103 or output device 102. In further embodiments, some of the features of the modules may be combined or run on other modules as required.

[0030] In some cases, the processing unit 108 can be located on a computing device that is remote from the one or more video-cameras 103 and / or the output device 102, and linked over an appropriate networking architecture; for example, a local-area network (LAN), a wide-area network (WAN), the Internet, or the like. In some cases, the processing unit 108 can be executed on a centralized computer server, such as in off-line batch processing.

[0031] The term “video”, as used herein, can include sets of still images. Thus, “video camera” can include a camera that captures a sequence of still images and “imaging camera" can include a camera that captures a series of images representing a video stream.

[0032] Using transdermal optical imaging (TOI), the TOI module 110 can isolate hemoglobin concentration (HC) from raw images taken from the digital camera 103. Referring now to FIG. 2, a diagram illustrating the re-emission of light from skin is shown. Light 201 travels beneath the skin 202, and re-emits 203 after travelling through different skin tissues. The re-emitted light 203 may then be captured by optical cameras 103. The dominant chromophores affecting the reemitted light are melanin and hemoglobin. Since melanin and hemoglobin have different color signatures, it has been found that it is possible to obtain images mainly reflecting HC under the epidermis as shown in FIG. 3.

[0033] Using transdermal optical imaging (TOI), the TOI module 110, via the image processing unit 104, obtains each captured image in a video stream, from the camera 103, and performs operations upon the image to generate a corresponding optimized hemoglobin concentration (HC) image of the subject. From the HC data, the facial blood flow localized volume concentrations can be determined. The image processing unit 104 isolates HC in the captured video sequence. In an exemplary embodiment, the images of the subject’s faces are taken at 30 frames per second using a digital camera 103. It will be appreciated that this process may be performed with alternative digital cameras, lighting conditions, and frame rates.

[0034] In a particular case, isolating HC can be accomplished by analyzing bitplanes in the sequence of video images to determine and isolate a set of the bitplanes that approximately maximize signal to noise ratio (SNR). The determination of high SNR bitplanes is made with reference to a first training set of images constituting the captured video sequence, in conjunction with blood pressure data gathered from the human subjects. The determination of high SNR bitplanes is made with reference to an HC training set constituting the captured video sequence. In some cases, this data is supplied along with other devices, for example, EKG, pneumatic respiration, blood pressure, laser Doppler data, or the like, collected from the human subjects, and received by the profile module 120, in order to provide ground truth to train the training set for HC change determination. A blood pressure training data set can consist of blood pressure data obtained from human subjects by using one or more blood pressure measurement devices as ground truth data; for example, an intra-arterial blood pressure measurement approach, an auscultatory approach, or an oscillometric approach. The selection of the training data set based on one of these three exemplary approaches depends on a setting in which the contactless blood pressure measurement system is used; as an example, if the human subject is in a hospital intensive care setting, the training data can be received from an intra-arterial blood pressure measurement approach.

[0035] Typically, a digital image consists of certain number of pixels (for example, a width X height of 1920X1080 pixels). Each pixel of the digital image having one or more channels (for example, color channels red, green, and blue (RGB)). Each channel having a dynamic range, typically 8 bits per pixel per channel, but occasionally 10 bits per pixel per channel for high dynamic range images. Whereby, an array of such bits makes up what is known as the bitplane. In an example, for each image of color videos, there can be three channels (for example, red, green, and blue (RGB)) with 8 bits per channel. Thus, for each pixel of a color image, there are typically 24 layers with 1 bit per layer. A bitplane in such a case is a view of a single 1 -bit map of a particular layer of the image across all pixels. For this type of color image, there are therefore typically 24 bitplanes (i.e., a 1-bit image per plane). Hence, for a 1-second color video with 30 frames per second, there are at least 720 (30X24) bitplanes. FIG. 4 is an exemplary illustration of bitplanes for a three-channel image (an image having red, green and blue (RGB) channels). Each stack of layers is multiplied for each channel of the image; for example, as illustrated, there is a stack of bitplanes for each channel in an RGB image. In the embodiments described herein, Applicant recognized the advantages of using bit values for the bitplanes rather than using, for example, merely the averaged values for each channel. Thus, a greater level of accuracy can be achieved for making predictions of HC changes, and thus blood pressure measurements as disclosed herein, and as described for making predictions. Particularly, a greater accuracy is possible because employing bitplanes provides a greater data basis for training the machine learning model.

[0036] TOI signals can be taken from regions of interest (ROIs) of the human subject, for example forehead, nose, and cheeks, and can be defined manually or automatically for the video images. The ROIs are preferably non-overlapping and are generally spaced with enough distance in between to have a reasonably quantifiable difference in pulse arrival time. These ROIs are preferably selected on the basis of which HC is particularly indicative of pulse transit time measurements. Using the native images that consist of all bitplanes of all three R, G, B channels, signals that change over a particular time period (for example, 10 seconds) on each of the ROIs are extracted.

[0037] The raw signals can be pre-processed using one or more filters by the filter module 116, depending on the signal characteristics. Such filters may include, for example, a Butterworth filter, a Chebyshev filter, or the like. Using the filtered signals from two or more ROIs, machine learning is employed to systematically identify bitplanes that will significantly increase the signal differentiation (for example, where the SNR improvement is greater than 0.1db) and bitplanes that will contribute nothing or decrease the signal differentiation. After discarding the latter, the remaining bitplane images can optimally determine blood flow generally associated with a determination of systolic and diastolic blood pressure.

[0038] Machine learning approaches (such as a Long Short Term Memory (LSTM) neural network, or a suitable alternative such as non-linear Support Vector Machine) and deep learning may be used to assess the existence of common spatial-temporal patterns of hemoglobin changes across subjects. The machine learning process involves manipulating the bitplane vectors (for example, 24 bitplanes X 30 fps) using the bit value in each pixel of each bitplane along the temporal dimension. In one embodiment, this process requires subtraction and addition of each bitplane to maximize the signal differences in all ROIs over the time period. In some cases, to obtain reliable and robust computational models, the entire dataset can be divided into three sets: the training set (for example, 80% of the whole subject data), the test set (for example, 10% of the whole subject data), and the external validation set (for example, 10% of the whole subject data). The time period can vary depending on the length of the raw data (for example, 15 seconds, 60 seconds, or 120 seconds). The addition or subtraction can be performed in a pixel-wise manner. An existing machine learning algorithm, the Long Short Term Memory (LSTM) neural network, or a suitable alternative thereto is used to efficiently and obtain information about the improvement of differentiation in terms of accuracy, which bitplane(s) contributes the best information, and which does not in terms of feature selection. The Long Short Term Memory (LSTM) neural network allow us to perform group feature selections and classifications. The LSTM machine learning algorithm are discussed in more detail below. From this process, the set of bitplanes to be isolated from image sequences to reflect temporal changes in HC is obtained for determination of blood pressure.

[0039] To extract facial blood flow data, facial HC change data on each pixel of each subject’s face image is extracted as a function of time when the subject is being viewed by the camera 103. In some cases, to increase signal-to-noise ratio (SNR), the subject’s face can be divided into a plurality of regions of interest (ROIs). The division can be according to, for example, the subject’s differential underlying physiology, such as by the autonomic nervous system (ANS) regulatory mechanisms. In this way, data in each ROI can be averaged. The ROIs can be manually selected or automatically detected with the use of a face tracking. This information can then form the basis for the training set. As an example, the system 100 can monitor stationary HC changes contained by a selected ROI over time, by observing (or graphing) the resulting temporal profile (for example, shape) of the selected ROI HC intensityvalues over time. In some cases, the system 100 can monitor more complex migrating HC changes across multiple ROIs by observing (or graphing) the spatial dispersion (HC distribution between ROIs) as it evolves over time.

[0040] Thus, it is possible to obtain a video sequence of any subject and apply the HC extracted from selected bitplanes to the computational models to determine blood flow. For long running video streams with changes in blood flow and intensity fluctuations, changes of the estimation and intensity scores over time relying on HC data based on a moving time window (e.g., 10 seconds) may be used.

[0041] FIG. 6 illustrates a diagrammatic implementation of the system 100 as applied to the face of a human subject 1.

[0042] FIG. 8 illustrates a flowchart of a method 800 for determination of pulse-transit time of a subject from a received image sequence, in accordance with an embodiment.

[0043] At block 802, the image processing unit, via the TOI module 110, receives an image sequence that captures the subject from the camera 103 or from the storage device 101 ; for example, capturing the face 2 of the human subject 1. At block 804, the TOI module 110 determines hemoglobin concentration (HC) changes over time at a pair of regions of interest (ROIs), for example, ROI-A and ROI-B in FIG. 6. In some cases, the TOI module 110 can also determine HC changes over time at another pair of ROIs, for example ROI-C and ROI-D in FIG.6. Each pair of ROIs are located along a common blood flow path, where one of the ROIs in the pair is located further downstream on the blood flow path than the other ROI in the pair, such that the second ROI is located downstream from the first ROI along a common pulse wave path. For example, having both ROIs located along a common artery as illustrated as 4A and 4B, respectively, in the example of FIG. 6. The HC changes overtime are indicative of a blood flow signal, as illustrated in the example chart of FIG. 7 where the signal from ROI-A is indicated as A and the signal from ROI-B is indicated as B.

[0044] At block 806, the pulse module 112 determines a pulse-transit time (PTT) based on a time difference between consecutive peaks in the HC signals, such as those indicated at pA and pB in the example of FIG. 7.

[0045] At block 808, the pulse module 112 determines a pulse wave velocity (PWV) based on the determined PTT and a distance between the ROIs. PWV is inversely related to PTT and can be calculated as:where d is the distance between the two ROIs where pulse waves have been detected.

[0046] At block 810, in some cases, the blood pressure module 114 estimates blood pressure using a machine learning model, for example, a linear regression model. The machine learning model takes the PWV as input. In some cases, the model can also take selected features derived from the respective HC signals of the pair of ROIs and / or demographic information about the human subject.

[0047] The machine learning model can be trained using any suitable methodology that generally maximizes performance and generalizability by splitting data into balanced training, validation and test sets. In some cases, during training, measures to mitigate overfitting can be taken by either early stopping, regularization, dropout, or a combination of thereof.

[0048] In further cases, the blood pressure module 114 can estimate the blood pressure by determining an estimate of mean arterial blood pressure, based on an assumed range for the ratio of arterial vessel diameter over arterial wall thickness, using a Moens-Korteweg equation.

[0049] To improve accuracy of the estimate, the second pair of ROIs lying along a different artery can be used to arrive at a more accurate measurement of PTT and PWV. For example, an average of the PTT and PWV values from both sets of ROIs can be used. Generally, the other pair of ROIs registered with the different artery is captured in the same video as that used for the first pair of ROIs. For example, in the example of FIG. 6, the first pair of ROIs is located on one side of the nose and the other pair of ROIs is located on an opposite side thereof.

[0050] At block 812, the output module 126 outputs the determined PTT, and in some cases, the estimated blood pressure, to the output device 102 or the storage device 101 .

[0051] The ROIs on the subject's face can be predetermined, such as predetermined locations on the face that are spaced apart. The ROIs are relatively small in size compared to the subject’s face, as to be substantially localized to the artery with which they are located. In other words, each one of the pair of ROIs delimits an area on the face the majority of which is coincident with the common artery or, in other words, on which the artery were to project onto the surface of the skin. While the example of FIG. 6 illustrates the ROIs located on the face of the subject, it is understood that the ROIs can be located on any suitable body part so long as the ROIs are located on the same blood flow path.

[0052] The peaks in the HC signals which are used to determine PTT represent the same pulse travelling through the artery. The peaks are consecutive, or in other words chronological, in that a selected peak in a first one of the HC signals, such as peak PA in HC signal A, and a subsequent peak in a second one of the HC signals, such as pBin HC signal B, is an earliest peak in the other HC signal occurring after the selected peak in the first one of the signals, are considered for the PTT calculation.

[0053] It will be appreciated that the step of determining the PWV comprises determining a distance between the ROIs. In the illustrated embodiment, the distance is based on a height of the human subject. Height information may be provided to the system for calculation of distance as input received from a user. When the system is executed on a smartphone, the input may be provided by, as to be received by the system from, the human subject.

[0054] Typically, the PTT, PVW and estimates of blood pressure is output on a visual display as the output device 102. When the system is a smartphone, the blood pressure estimate is preferably displayed on a smartphone display. However, any suitable output device 102 can be used to communicate the information.

[0055] In the illustrated embodiment, the method further comprises a step 811 of capturing the video of the face of the human subject. Preferably, the video is captured by a single camera. That is, the single camera is positioned relative to the subject’s face in such a manner as to capture, in a common frame of the video, facial regions associated with the pair of ROIs to be used to derive an estimate of blood pressure.

[0056] This provides an arrangement for contactless detection of human blood pressure which can be performed by a system without calibration thereof.

[0057] Other applications may become apparent.

[0058] Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

CLAIMS1. A method for contactless determination of pulse-transit time from a received image sequence, the method executed on one or more processors, the method comprising: receiving an image sequence capturing a body-part of a subject; determining hemoglobin concentration (HC) signals over time at a first region of interest (ROI) and a second ROI, the first ROI and the second ROI forming a pair of ROIs, the second ROI located downstream from the first ROI along a common pulse wave path; determining a pulse transit time based on a time difference between peaks in the HC signals; and outputting the pulse transit time.

2. The method of claim 1, further comprising determining pulse wave velocity (PWV) using the determined PTT and a distance between the pair of ROIs, and outputting the determined PWV.

3. The method of claim 1, further comprising determining an estimate of blood pressure using the determined PWV with a trained machine learning model, and outputting the determined estimate of blood pressure.

4. The method of claim 3, wherein the machine learning model further takes, as input, selected features derived from the respective HC signals of the pair of ROIs and demographic information about the human subject.

5. The method of claim 3, wherein the trained machine learning model is a linear regression model.

6. The method of claim 1, further comprising determining an estimate of mean arterial blood pressure, based on an assumed range for the ratio of arterial vessel diameter over arterial wall thickness, using a Moens-Korteweg equation.

7. A system for contactless determination of pulse-transit time from a received image sequence, the system comprising one or more processing units and a data storage memory, the data storage memory comprising instructions for the one or more processing units to execute:a transdermal optical imaging module to receive an image sequence capturing a body-part of a subject and determine hemoglobin concentration (HC) signals over time at a first region of interest (ROI) and a second ROI, the first ROI and the second ROI forming a pair of ROIs, the second ROI located downstream from the first ROI along a common pulse wave path; a pulse module to determine a pulse transit time based on a time difference between peaks in the HC signals; and an output module to output the pulse transit time.

8. The system of claim 7, wherein the pulse module further determines pulse wave velocity (PWV) using the determined PTT and a distance between the pair of ROIs, and the output module further outputs the determined PWV.

9. The system of claim 7, wherein the one or more processing units further execute a blood pressure module to determine an estimate of blood pressure using the determined PWV with a trained machine learning model, and the output module further outputs the determined estimate of blood pressure.

10. The system of claim 9, wherein the machine learning model further takes, as input, selected features derived from the respective HC signals of the pair of ROIs and demographic information about the human subject.

11. The system of claim 9, wherein the trained machine learning model is a linear regression model.

12. The system of claim 7, the output module further outputs an estimate of mean arterial blood pressure, based on an assumed range for the ratio of arterial vessel diameter over arterial wall thickness, using a Moens-Korteweg equation.

Images