Digital healthcare device for measuring heart rate using remote PPG method

Abstract

The digital healthcare device according to the present invention includes a processor that determines whether to capture a subject's face as a color image or an infrared (IR) image based on a predetermined time period or whether the surroundings are below a predetermined brightness level, and a camera that captures the subject's face to capture a facial image based on the determination of the processor. The processor detects a region of interest (ROI) corresponding to the face from the facial image, captures a first image for a first region from the detected ROI, captures a second image for a second region from the detected ROI, applies the captured first and second images to a predetermined learned algorithm model, and outputs a remote Photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject.

Description

[Technical field]

[0001] The present invention relates to a digital healthcare device, and more particularly to a digital healthcare device for measuring the heart rate, respiratory rate, sleep state, etc. of a subject such as a baby in a non-contact manner using a remote PPG method. [Background technology]

[0002] There are studies that use PPG sensors to non-invasively sense biosignals and understand the relationship to human diseases. However, due to the current COVID-19 pandemic, technologies that can monitor health remotely in a non-invasive manner have become significantly more important. Countries are advising the use of remote health strategies as much as possible to reduce the risk of COVID-19 in medical environments. Therefore, new methods are required to replace the existing biometric monitoring methods using sensors that require contact with the body.

[0003] Remote health monitoring technology is based on communication systems such as mobile phones and online health portals. Such remote health monitoring technology may be highly required for continuous patient monitoring even after epidemics such as COVID-19 are over. It is used to measure the physiological signals of users based on facial video streams using cameras. In addition to infectious diseases, such technology can also be used for vital signs monitoring of infants, elderly people, and mental health monitoring.

[0004] However, currently there is no research or healthcare product that estimates human biosignals and predicts heart rate, stress index, respiratory rate, etc. as a remote health monitoring technology. This invention was made with the support of the Smart Cradle Development Project that can be linked to a wheelchair (Project unique number (1465036599), Project number (HJ21C0014030022), Research period (2022.01.01~2022.12.31)), which is a research and development project for assistive devices for the elderly and disabled supported by the Ministry of Health and Welfare of Korea and the Korea Health Industry Development Institute. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem to be solved by the present invention is to provide a digital healthcare device for measuring heart rate using a remote PPG method.

[0006] Another technical problem to be solved by the present invention is to provide a monitoring method for subject health care using a remote PPG method.

[0007] Yet another technical problem to be solved by the present invention is to provide a computer-readable recording medium having recorded thereon a computer program for executing a method for monitoring a subject's health care using a remote PPG method.

[0008] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0009] In order to achieve the above technical objectives, a digital healthcare device according to an embodiment of the present invention includes a processor that determines whether to capture a face of a subject as a color image or an infrared (IR) image based on whether a predetermined time period or surroundings are below a predetermined brightness, and a camera that captures the face of the subject to capture a facial image based on the determination of the processor, and the processor detects a region of interest (ROI) corresponding to the face from the facial image, acquires a first image for a first region from the detected ROI, acquires a second image for a second region from the detected ROI, applies the acquired first image and second image to a predetermined learned algorithm model, and outputs a remote photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject. The first region may correspond to a forehead region, and the second region may correspond to a cheek region.

[0010] The digital healthcare device further includes a BCG sensor for sensing a ballistocardiogram (BCG) signal of the subject, and the processor may calculate a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and output the heart rate of the subject based on the first heart rate and the second heart rate. The output heart rate corresponds to an average heart rate of the first heart rate and the second heart rate. The predetermined learned algorithm model uses a Siamese Neural Networks (SNN) model of multi-task learning. The processor may detect two eye region images from the region of interest, detect two pupil images from the two detected eye region images, and detect two irises from the two detected pupil images, and determine that the subject is in a wake state when all of the two irises are recognized. The processor may determine that the subject is in a sleep state when a state in which none of the two irises are recognized from the two detected pupil images has elapsed for a predetermined time. The digital healthcare device further includes a bed on which the subject, an infant, or toddler, lies, and the processor may control the bed to maintain an active bounce function when the subject is determined to be in a wake state. The digital healthcare device includes a bed on which the subject lies, and the processor may control vertical and horizontal movements of the bed to gradually stop the bounce function when the subject is determined to be in a sleep state. The digital healthcare device further includes a body frame, and the camera is movable on the body frame so that the subject's face is photographed, taking into account the subject's lying posture and direction. The digital healthcare device may further include a communication unit for transmitting the output heart rate to a linked terminal. The digital healthcare device further includes a bed on which the subject lies, and the BCG sensor is provided to be attached to an inner surface of a cover covering the bed.The processor may calculate a respiration rate based on the output respiration rate signal waveform, and determine whether or not the subject has sleep apnea based on the calculated respiration rate. The digital healthcare device may further include a communication unit for transmitting the calculated respiration rate to a linked terminal.

[0011] In order to achieve the above technical objective, a monitoring method for subject health care using a remote PPG scheme according to another embodiment of the present invention may include: determining whether to acquire a face of a subject as a color image or an infrared (IR) image based on whether a predetermined time period or surroundings are below a predetermined brightness; acquiring a facial image by photographing the face of the subject according to the determination of the processor; detecting a region of interest (ROI) corresponding to the face from the facial image; acquiring a first image for a first region and a second image for a second region from the detected ROI; and applying the acquired first and second images to a predetermined learned algorithm model to output a remote Photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject. The first region may correspond to a forehead region, and the second region may correspond to a cheek region.

[0012] The method may further include sensing a ballistocardiogram (BCG) signal of the subject, calculating a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and outputting the heart rate of the subject based on the first heart rate and the second heart rate. The method may further include detecting two eye site images from the region of interest, detecting two pupil images from the two detected eye site regions, and determining that the subject is in a wake state when two irises are both recognized from the detected two pupil images. The method may further include determining that the subject is in a sleep state when a state in which neither of the two irises is recognized from the detected two pupil images has elapsed for a predetermined time. The method may further include calculating a respiration rate based on the output respiration rate signal waveform, and determining whether the subject has sleep apnea based on the calculated respiration rate. Effect of the Invention

[0013] A healthcare device according to one embodiment of the present invention measures information on a subject's heart rate, respiratory rate, stress index, sleep condition (sleep apnea), etc. in a non-contact manner, and can greatly improve accuracy.

[0014] A healthcare device according to an embodiment of the present invention can greatly improve the accuracy of determining a wake / sleep state of a subject by recognizing the subject's iris.

[0015] A healthcare device according to one embodiment of the present invention can perform continuous health monitoring by estimating vital signs of infants, the elderly, and sick people based on facial images in a manner that predicts PPG and respiratory rate (RR).

[0016] A healthcare device according to an embodiment of the present invention has the effect of estimating vital signs based on images in a non-contact manner and monitoring the health of infants, patients, etc. in a situation where an infectious disease such as COVID-19 continues.

[0017] The effects obtained from the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Brief description of the drawings]

[0018] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide examples of the present invention and, together with the detailed description, explain the technical concepts of the present invention. [Figure 1] FIG. 1 is a diagram illustrating a layer structure of an artificial neural network. [Diagram 2] FIG. 1 is a diagram illustrating a ballistocardiogram (signal) waveform. [Diagram 3] FIG. 1 is an exemplary diagram for explaining rPPG (remote PPG). [Figure 4] FIG. 4 is a block diagram illustrating the configuration of a health care device (400) according to an embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram for explaining the output of a PPG signal and a respiratory rate (RR) signal in an MTL algorithm learning model using the rPPG technique. [Figure 6] FIG. 13 illustrates an analysis of PPG signal frequency and distribution of LF score / HF score for calculating a subject's stress index. [Figure 7] FIG. 4 illustrates an exemplary healthcare device (400) according to the present invention. [Figure 8] FIG. 4 illustrates the steps of a healthcare device (400) according to the present invention detecting the iris of a subject. [Figure 9] FIG. 4 is a diagram illustrating in detail how the healthcare device (400) of the present invention detects the pupil (iris) of a subject. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below together with the accompanying drawings is for describing exemplary embodiments of the present invention, and does not represent the only embodiments in which the present invention can be implemented. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will understand that the present invention can be implemented without such specific details.

[0020] In some cases, known structures and devices may be omitted or shown in block diagram form focusing on the core functions of each structure and device in order to avoid obscuring the concept of the present invention. Also, the same components are described with the same reference numerals throughout the specification.

[0021] Before explaining the present invention, we will explain artificial intelligence (AI), machine learning, and deep learning. The relationship between these three concepts can be easily understood by imagining three concentric circles. Artificial intelligence is the largest circle, followed by machine learning, and deep learning, which is leading the current artificial intelligence boom, is the smallest circle.

[0022] Deep learning will be explained in more detail below.

[0023] Deep learning is a type of artificial neural network (ANN) that uses human neural network theory, and is a set of machine learning models or algorithms that refer to a deep neural network (DNN) that is composed of a layer structure and has one or more hidden layers (hereinafter referred to as intermediate layers) between the input layer and the output layer. Simply put, deep learning can be said to be an artificial neural network with a deep hierarchy.

[0024] Deep neural networks can be considered the successor to artificial neural networks, and are the latest version of artificial neural networks that have overcome existing limitations and achieved success in areas where many artificial intelligence technologies have failed in the past. To explain how artificial neural networks are modeled by imitating biological neural networks, they are modeled as shown in Table 1 below, with biological neurons as nodes in terms of processing units and synapses as weights in terms of connections.

[0025] [Table 1]

[0026] FIG. 1 is a diagram illustrating a layer structure of an artificial neural network. Just as human biological nerve cells are connected not one but several to perform a meaningful task, in an artificial neural network, individual neurons are connected to each other through synapses to form multiple layers, and the connection strength between each layer can be updated with a weight value. In this way, the multi-layer structure and connection strength are utilized in the fields of learning and cognition. Each node is connected by a link with a weight value, and the overall model learns by repeatedly adjusting the weight value. The weight value is the basic means for long-term memory and represents the importance of each node. The artificial neural network initializes these weight values, updates and adjusts the weight value in the training data set, and trains the overall model. When a new input value is entered after training is completed, the appropriate output value is inferred. The learning principle of an artificial neural network can be said to be a process in which intelligence is formed from the generalization of experience, and is performed in a bottom-up manner. In Figure 1, when there are two or more intermediate layers (i.e., 5 to 10), the layers are considered to be deep and are called a deep neural network, and the learning and inference model performed through such a deep neural network can be called deep learning. Although an artificial neural network can function to some extent with only one intermediate layer (usually called a hidden layer) excluding input and output, the number of nodes or layers must be increased as the complexity of the problem increases. Among these, it is effective to increase the number of layers to create a multi-layered model, but the scope of use is limited due to the limitations of inefficient learning and the large amount of calculation required to learn the network.

[0027] While previous deep neural networks have generally been designed as forward propagation neural networks, recent research has successfully applied deep learning architectures to recurrent neural networks (RNNs). For example, deep neural network architectures have been applied to the field of language modeling. Convolutional neural networks (CNNs) have been widely applied in computer vision, with successful applications well documented. More recently, convolutional neural networks have been applied to acoustic modeling for automatic speech recognition (ASR), and have been considered more successful than existing models. Deep neural networks may be trained using a standard error backpropagation algorithm. Here, weights may be updated using stochastic gradient descent using the following equation:

[0028] The present invention proposes a healthcare device that predicts information related to a living body (heart rate, sleep state such as sleep apnea, wake / sleep state, etc.) and stress index based only on face image information in a non-contact state using a predetermined learned algorithm model. A health monitoring method that predicts PPG and respiration rate (RR) based on face images in a non-contact manner is not only suitable for subjects who require continuous health monitoring such as infants and adults with diseases such as diabetes, but can also be a safe health monitoring method in an environment where COVID-19, which is transmitted by contact, is prevalent. The present invention proposes a method for predicting a subject's heart rate, stress index, presence or absence of sleep apnea, and wake / sleep state using a remote PPG (rPPG) signal and a ballistocardiogram (BCG) signal. Next, a ballistocardiogram and PPG obtained by sensing the human body as biological information used in the present invention will be briefly described.

[0029] <Heart rate (Photoplethysmograph, PPG) sensor>

[0030] As an example of a heart rate sensor, a PPG (Photoplethysmograph) sensor will be described. A PPG sensor is a photoelectric pulse wave sensor. Photoplethysmograph (PPG) is a pulse wave measurement method that uses the optical properties of biological tissue to measure the blood flow rate in blood vessels, thereby determining the heart activity state or heart rate. A pulse wave is a pulsating waveform that is generated as blood is pumped out of the heart, and can be measured from the change in blood flow rate, i.e., the change in blood vessel volume, generated by the relaxation and contraction of the heart. Photoplethysmograph is a method of measuring pulse waves using light, and uses an optical sensor to detect and measure changes in optical properties such as reflection and absorption / transmittance ratio of biological tissue that occur when the volume changes, thereby enabling pulse measurement. This method is widely used due to its advantages of non-invasive pulse measurement, compact size, and ease of use, and can be used as a biosignal sensor in wearable devices.

[0031] Ballistocardiogram (BCG) Sensor

[0032] FIG. 2 is a diagram illustrating a ballistocardiogram (signal) waveform.

[0033] During the cardiac cycle, the moment blood is ejected from the ventricle and passes through the aorta, it sends a recoil force to the body. The signal that measures the vibration (ballistic motion) caused by the change in blood flow in the heart and blood vessels is called a ballistocardiogram (BCG). A ballistocardiogram is a signal that measures the ballistic motion caused by the change in blood flow in the heart and blood vessels as the heart contracts and relaxes, and is an indicator of the heart's activity similar to an electrocardiogram.

[0034] Similar to an electrocardiogram, a ballistocardiogram is known to be an index of cardiac activity, containing information on cardiac output, regurgitation due to myocardial dysfunction, and abnormal blood flow. Therefore, this biosignal has potential for clinical use, such as evaluation of cardiac function, diagnosis of cardiac disease (myocardial dysfunction, etc.), confirmation of treatment effects, and observation of recovery. Ballistocardiogram signals can be measured using acceleration sensors, load cell sensors, PVDF film sensors, EMFi sensors, etc. When using such sensors, there is no need to attach electrodes to the body, so signals can be measured in an unrestrained / unaware state, making them useful for health monitoring over long periods of time or during daily life.

[0035] As shown in Fig. 2, the heartbeat pattern in a ballistocardiogram signal is expressed by H, I, J, and K peaks, and the part recognized as the actual heartbeat is the J peak. In a ballistocardiogram signal, the heartbeat pattern appears in various forms depending on noise, environment, and personal influences, but usually, the I peak clearly shows a large difference depending on the environment, measurement conditions, and individual differences, and the magnitudes of the H peak and J peak are not constant. Also, when the I peak is small, there are cases where only one of the H peak or J peak appears in a large form.

[0036] The remote measurement method will now be briefly described.

[0037] FIG. 3 is an exemplary diagram for explaining rPPG (remote PPG).

[0038] rPPG can be used to measure heart rate and heart rate variability remotely. As shown in Figure 3, a high-resolution camera can be used to record video to measure heart rate remotely. It can be useful for various physical, health, and emotional monitoring, such as monitoring drivers, the elderly, and infants. rPPG uses the same principle as PPG, but is a non-contact measurement. It measures the changes in the reflection of red, green, and red light that change on the skin as a comparison between specular reflection and diffuse reflection. Specular reflection is pure light reflection from the skin. Diffuse reflection is reflection that remains due to absorption and scattering of skin tissue that changes with changes in blood volume. It is based on the principle that hemoglobin reflects red light and absorbs green light. Figure 3 shows examples of red, green, and blue light waveforms detected after signal processing using the rPPG method, and it can be seen that there are differences in the rPPG waveforms detected for each of these lights. Remote measurement technology for remotely monitoring vital signs is highly applicable in environments that require non-contact monitoring, such as infant monitoring and infectious diseases such as COVID-19, because it can predict vital signs only from facial information without contact. Facial images are captured using ambient light around the subject as a light source.

[0039] 《MTL(Multi-Task Learning)》

[0040] There have been many attempts to predict biological information using deep neural networks and machine learning. In the implementation of mobile medical systems, Multi-Task Learning (MTL) is an important approach to perform multiple tasks with limited resources. PPG and respiratory rate signals are simultaneously extracted from a facial video stream using MTL. In this invention, we propose a complex value-based Multi-Task Learning (MTL) algorithm model that processes video streams simultaneously. Two facial regions consist of complex number data that are simultaneously processed by a complex-valued neural network architecture. Using this complex process, PPG and respiratory rate signals can be extracted more efficiently and accurately compared to real-value single-task learning algorithms.

[0041] Multi-Task Learning (MTL) is a model learning method that learns and predicts two or more tasks simultaneously through a shared layer. By learning related tasks simultaneously, the learned representation is shared, and thus each task can help the model learning with good representation. Useful information obtained during learning can have a positive effect on other tasks and contribute to a better model. In addition, by predicting multiple tasks simultaneously, it is learned as a generalized model that is more resistant to overfitting. Conventionally, two tasks had to be created individually, but since it is created as one model, it greatly helps to reduce the model weight and is more advantageous for application to mobile devices such as smartphones.

[0042] FIG. 4 is a block diagram illustrating the configuration of a health care device 400 according to an embodiment of the present invention, and FIG. 5 is a diagram illustrating the output of a PPG signal and a respiratory rate (RR) signal in an MTL algorithm learning model using the rPPG method.

[0043] Referring to FIG. 4 , a healthcare device 400 according to the present invention may include a processor 410 , a camera 420 , a bed 430 , a BCG sensor 440 , a display unit 450 , a communication unit 460 , a body frame 470 , and a bed motion control unit 480 .

[0044] The camera 420 captures the face of a subject (e.g., an infant lying on a bed 430). The camera 420 can transmit the captured facial image of the subject to the processor 410 via the communication unit 460. The processor 410 can acquire the facial image of the subject. Here, the captured facial image of the subject is a color image, which may be an RGB image.

[0045] Referring to FIG. 5, the processor 410 can obtain original data 510 obtained by photographing the face of a subject, and detect a facial image 520 of the subject from the original data 510. The processor 410 can obtain region of interest frames 530 and 540 from the detected facial image 520, and detect a first region (e.g., forehead) region 430 and a second region (e.g., cheek region) 560 from each region of interest frame 530 and 540. The processor 410 can learn the facial image using a deep learning network used in image processing, such as a convolutional neural network (CNN) or a siamese neural network (SNN). In the present invention, it is preferable that the learning model of the multiple learning algorithm uses an SNN (siamese neural network). The processor 410 repeatedly applies a filter (kernel) to all regions of the forehead and cheek images extracted from the facial image 520 to find and learn a pattern. The reason for extracting the forehead and cheeks is to use the difference between the two body parts due to the time difference in blood rising from the heart as significant information for learning the model.

[0046] The detected image for the forehead region and the image for the cheek region are color images. For convenience of explanation, the image 550 for the forehead region is referred to as a first color image, and the image 560 for the cheek region is referred to as a second color image. At this time, the processor 410 converts the first color image and the second color image into black and white images, and obtains a first black and white image from the first color image and a second black and white image from the second color image. After converting to black and white, the processor 410 learns using MTL (e.g., Multi-task Siamese network) to predict the PPG signal and the respiratory rate signal.

[0047] As shown in FIG. 5, the processor 410 converts the first color image for the forehead area and the second color image for the cheek area into a first black-and-white image and a second black-and-white image, respectively, and inputs them to a predetermined trained multi-task learning algorithm model using a Siamese neural network (SNN). The processor 410 applies the first black-and-white image and the second black-and-white image to a predetermined trained multi-task learning algorithm model, and outputs a remote PPG signal waveform (also called a PPG signal waveform output by an rPPG method) and a respiration rate (RR) waveform of the subject. As described above, the processor 410 converts the color images for the forehead area and the cheek area into black-and-white images, respectively, and inputs them to the predetermined trained algorithm model, preferably when it is a predetermined set daytime or the surroundings are brighter than a predetermined brightness. The processor 410 can control the camera 420 to capture the subject's face and obtain a color image when it is the predetermined set daytime or the surroundings are brighter than a predetermined brightness.

[0048] On the other hand, if it is a predetermined night time or the surroundings are less than a predetermined brightness, the processor 410 may control the camera 420 or a separately provided camera for capturing infrared images to capture the subject's face and acquire an infrared (IR) image. If it is a predetermined night time or the surroundings are less than a predetermined brightness, the camera 420 captures the subject's face and acquires an infrared (IR) image, and the processor 410 may detect a region of interest (ROI) corresponding to the face from the infrared image, and acquire a first image for the forehead area and a second image for the cheek area from the detected ROI. The processor 410 may then apply the first image and the second image respectively acquired from the infrared image to a predetermined learned multi-task learning algorithm model, and output a remote photoplethysmography (rPPG) signal waveform of the subject. The processor 410 can then calculate a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and output the subject's heart rate based on the first heart rate and the second heart rate.

[0049] In this way, the processor 410 can select whether to acquire the subject's face as a color image or an infrared image depending on the time of day or the brightness of the surroundings. The processor 410 can convert the color image into a black and white image based on the time of day or the brightness of the surroundings and input it into a predetermined learned algorithm (MTL algorithm) model and apply it, or input the image acquired from the infrared image into a predetermined learned algorithm model and apply it.

[0050] The healthcare device 400 according to the present invention may include a bed 430 for a subject (e.g., an infant) to lie on. A BCG sensor 440 may be attached to the inside of a cover of the bed 430. Since the BCG sensor 440 is provided on the inner surface of the bed cover, it is possible to sense a ballistocardiogram signal from the side of the subject when the subject is lying down.

[0051] The processor 410 can calculate the heart rate from the ballistocardiogram signal waveform acquired from the BCG sensor 440, acquire the PPG signal waveform from a predetermined learned MTL algorithm model, and calculate the heart rate from the acquired PPG signal waveform. Here, the heart rate calculated from the ballistocardiogram signal waveform is referred to as the first heart rate, and the heart rate calculated from the PPG signal waveform is referred to as the second heart rate. The first heart rate may be calculated by the number of detections of J-peaks in the BCG signal waveform per unit time (e.g., 1 minute), and the second heart rate may be calculated by the number of peaks in the rPPG signal waveform per unit time (e.g., 1 minute). The processor 410 can output the subject's heart rate based on the first heart rate calculated from the ballistocardiogram signal waveform and the second heart rate calculated from the PPG signal waveform acquired using the rPPG technique. For example, the processor 410 can output a heart rate obtained by averaging the first heart rate and the second heart rate.

[0052] Furthermore, the processor 410 can calculate heart rate variability (HRV) from the ballistocardiogram signal waveform. The heart rate variability can be calculated, for example, by the following formula 1. Here, the heart rate variability calculated from the ballistocardiogram signal waveform is referred to as a first heart rate variability.

[0053]

number

[0054] The processor 410 can calculate the heart rate variability from the PPG waveform calculated by the rPPG technique according to the following Equation 2. Here, the heart rate variability calculated from the PPG signal waveform is referred to as a second heart rate variability.

[0055]

number

[0056] FIG. 6 is a diagram illustrating an example of a PPG signal frequency analysis and a distribution of LF score / HF score for calculating a stress index of a subject.

[0057] The processor 410 needs to perform frequency analysis of the PPG signal waveform and frequency analysis of the BCG signal waveform in order to calculate the stress index. FIG. 6 shows frequency analysis of the PPG signal waveform as an example. The processor 410 can calculate a first stress index (score) of the subject based on the first heart rate variability, and calculate a second stress index of the subject based on the second heart rate variability. Here, the method of calculating the first stress index and the second stress index is as shown in the following Equation 3.

[0058]

number

[0059] Equation 3 calculates the sympathetic nervous system signal (LF) and the parasympathetic nervous system signal (HF) and converts them into a stress index.

[0060] The following description will be given with reference to the above formula 3 and FIG. 6(a). LF (sympathetic nerves) is the integral value of the power spectral density of the heart rate variability in the low frequency range of 0.04 to 0.15 Hz. HF (parasympathetic nerves) is the integral value of the power spectral density graph of the heart rate variability in the high frequency range of 0.15 to 0.40 Hz. The stress index (score) is calculated by taking the natural logarithm of LF power and HF power, and then calculating the LF score and HF score according to the range to convert it into a stress score. The score indicates whether the 5th region is normal and how far it is from the 5th region.

[0061] The processor 410 may calculate a first stress index (score) of the subject based on the above Equations 1 and 3, and may calculate a second stress index based on the above Equations 2 and 3. The processor 410 may output a stress index of the subject based on the first stress index and the second stress index. For example, the processor 410 may output an average value of the first and second stress indices as the stress index of the subject.

[0062] The processor 410 outputs a respiration rate (RR) signal waveform in addition to a PPG signal waveform based on a predetermined trained multi-task learning algorithm model using a Siamese neural network (SNN). The processor 410 also performs a subject's health monitoring function, such as judging whether or not the subject (e.g., an infant) has sleep apnea based on the output respiration rate signal waveform. The processor 410 can control the output respiration rate signal waveform, respiration rate, etc. to be displayed on the display unit 450 so that a person monitoring the subject can check them. The processor 410 also controls the output heart rate, the output stress index, and the output (or calculated) respiration rate to be displayed on the display unit 450, thereby enabling the subject's health condition to be monitored from the outside.

[0063] The communication unit 460 may periodically or non-periodically transmit information such as the output heart rate, the output stress index, and the output respiration rate to a user's terminal or a server that performs health monitoring of the subject via WiFi (registered trademark), Bluetooth (registered trademark), etc. The non-periodical transmission refers to transmitting only when at least one of the output heart rate, the output stress index, and the output respiration rate exceeds a predetermined threshold. For example, the communication unit 460 may transmit the output respiration rate to the linked terminal only when it is determined that the subject is in a sleep apnea state. The user may periodically or non-periodically receive the health monitoring information of the subject using his / her linked terminal to check the subject's health condition.

[0064] FIG. 7 is a diagram illustrating a healthcare device 400 according to the present invention.

[0065] 7, the body frame 470 of the healthcare device 400 may include a lower support part that supports the bed 430 from below, and a side support part 475 that surrounds the bed 430 and acts as a guard to prevent the subject from falling off the bed 430. The camera 420 is located on the side support part 475 and can capture the subject's face. The side support part 475 may be provided with a moving member that can move the camera 420 in order to track the subject's face along with the subject's movement while the subject is lying on the bed 430. The camera 420 can capture the subject's face while moving on the side support part 475 of the body frame 470, taking into account the subject's lying posture and direction.

[0066] FIG. 8 is a diagram illustrating the steps in which the healthcare device 400 according to the present invention detects the iris of a subject, and FIG. 9 is a diagram for explaining in detail how the healthcare device 400 according to the present invention detects the pupil (iris) of a subject.

[0067] Referring to Fig. 8, the camera 420 captures the face of the subject. The processor 410 detects an image 810 of the subject's face region, which is a region of interest, and extracts two eye region images 820 of the subject from the detected face image 810. The processor 410 then detects two irises from the detected two eye region images 820. In this case, the processor 410 determines that the subject is in a wake state when both irises are detected or recognized from the detected two eye region images 820. In contrast, the processor 410 may determine that the subject is in a sleep state when neither of the two irises are recognized from the detected two eye region images 820 for a predetermined time.

[0068] 9, the processor 410 can detect eye region areas from a face image of a subject to obtain an eye region image 910. The processor 410 can detect pupils from the obtained eye region image 910 to obtain a pupil region image 920, and can remove eyebrows and noise from the pupil region image 920 to obtain a pupil image 930. The processor 410 can detect an iris 950 based on the pupil image 930 and display it in an eye region image 940.

[0069] For convenience of explanation, only one eye is shown in FIG. 9, but the camera 420 and the processor 410 perform photographing, image processing, etc. for both eyes of the subject.

[0070] When the subject (e.g., an infant) is determined to be in a wake state, the processor 410 can control the bed motion control unit 480 to maintain the bounce function currently operating in the bed 430. The bed motion control unit 480 is provided under the bed 430, and performs the bounce function of the bed 430 by controlling the horizontal and vertical movements of the floor board of the bed 430 under the control of the processor 410. In contrast, when the subject is determined to be in a sleep state, the processor 410 can control the bed motion control unit 480 to control the horizontal and vertical movements of the floor board of the bed 410 to gradually stop the bounce function.

[0071] Conventionally, the position of the eyes was detected using the brightness around the eyes and the position of the eyebrows, which resulted in problems such as erroneously recognizing other parts of the face as the eyes. However, the processor 410 according to the present invention as described above does not search for the position of the subject's eyes, but detects the irises in the two eye part images 820, and after determining whether the irises are recognized, estimates whether the subject is in a awake state or a sleep state, thereby greatly improving the accuracy of determining the wake / sleep state of the subject.

[0072] The above-described embodiments are combinations of the components and features of the present invention in a predetermined form. Each component or feature should be considered as optional unless otherwise expressly stated. Each component or feature may be implemented in a form not combined with other components or features. It is also possible to combine some components and / or features to configure an embodiment of the present invention. The order of operations described in the embodiments of the present invention may be changed. Some configurations or features of one embodiment may be included in another embodiment, or may be replaced by corresponding configurations or features of another embodiment. It is clear that claims that do not have an explicit reference relationship in the claims may be combined to configure an embodiment, or may be included as a new claim by amendment after filing.

[0073] In the present invention, the processor 410 may be embodied by hardware, firmware, software, or a combination thereof. When an embodiment of the present invention is embodied using hardware, the processor 410 may include an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like configured to execute the present invention. The method for monitoring health care according to the present invention may be embodied as a computer-readable recording medium having a program recorded thereon for executing the method on a computer.

[0074] It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential features of the present invention. Therefore, the above detailed description should not be interpreted as limiting in any respect, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. [Industrial Applicability]

[0075] A digital healthcare device for measuring heart rate using remote PPG techniques is available for industrial use.

Claims

1. A processor that determines whether to capture a color image or an infrared (IR) image of the subject's face based on a predetermined time of day or whether the surroundings are below a predetermined brightness level; a camera for capturing a facial image of the subject as determined by the processor; The processor detects a region of interest (ROI) corresponding to a face from the facial image; obtaining a first image for a first region and a second image for a second region from the detected region of interest; A digital healthcare device that applies the acquired first and second images to a predetermined trained algorithm model and outputs a remote Photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject.

2. The method further includes a ballistocardiogram (BCG) sensor for sensing a BCG signal of the subject; The digital healthcare device of claim 1 , wherein the processor calculates a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and outputs the subject's heart rate based on the first heart rate and the second heart rate.

3. The digital healthcare device of claim 2 , wherein the output heart rate corresponds to an average heart rate of the first heart rate and the second heart rate.

4. The digital healthcare device of claim 1 , wherein the predetermined learned algorithmic model utilizes a Multi-task Learning Siamese Neural Networks (SNN) model.

5. The processor, detecting two eye region images from the region of interest, and detecting two pupil images from the two detected eye region images; The digital healthcare device of claim 1 , wherein the subject is determined to be in a awake state when two irises are detected and all of the two detected pupil images are recognized.

6. The digital healthcare device of claim 5 , wherein the processor determines that the subject is in a sleep state when a state in which neither of the two irises is recognized from the two detected pupil images has elapsed for a predetermined time.

7. The apparatus further includes a bed for an infant corresponding to the subject to lie on, The digital healthcare device of claim 5 , wherein the processor controls the bed to maintain an active bounce function when the subject is determined to be in a wake state.

8. a bed for the subject to lie on, The digital healthcare device of claim 6 , wherein the processor controls the vertical and horizontal movements of the bed to gradually stop a bounce function when the subject is determined to be in a sleep state.

9. Further comprising a body frame; The digital healthcare device according to claim 1 , wherein the camera is movable on the body frame so that the face of the subject is captured taking into account the subject's lying posture and direction.

10. The digital healthcare device according to claim 1 , wherein the first region corresponds to a forehead and the second region corresponds to a cheek.

11. further comprising a bed for the subject to lie on; The digital healthcare device of claim 1 , wherein the BCG sensor is configured to be attached to an inner surface of a cover covering the bed.

12. The digital healthcare device according to claim 1 , wherein the processor calculates a respiration rate based on the output respiration rate signal waveform, and determines whether or not the subject has sleep apnea based on the calculated respiration rate.

13. The digital healthcare device of claim 12, further comprising a communication unit for transmitting the calculated respiratory rate to a linked terminal.

14. 1. A method for remote PPG monitoring of a subject for health care purposes, comprising: determining whether to capture the subject's face as a color image or an infrared (IR) image based on a predetermined time period or whether the surroundings are below a predetermined brightness level; acquiring a facial image by photographing a face of the subject as determined by the processor; detecting a region of interest (ROI) corresponding to a face from the face image; acquiring a first image for a first region and a second image for a second region from the detected region of interest; applying the acquired first and second images to a predetermined learned algorithm model, and outputting a remote Photoplethysmography (rPPG) signal waveform and a respiration rate signal waveform of the subject.

15. sensing a ballistocardiogram (BCG) signal of the subject; 15. The method for monitoring health care according to claim 14, further comprising: calculating a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform; and outputting a heart rate of the subject based on the first heart rate and the second heart rate.

16. detecting two eye region images from the region of interest, and detecting two pupil images from the two detected eye region areas; The method for monitoring healthcare according to claim 15, further comprising determining that the subject is in a awake state when both irises are recognized from the two detected pupil images.

17. The method for monitoring healthcare according to claim 15, further comprising determining that the subject is in a sleep state when a state in which neither of the two irises is recognized from the two detected pupil images has elapsed for a predetermined time.

18. calculating a respiratory rate based on the output respiratory rate signal waveform; The method for monitoring health care according to claim 14, further comprising determining whether or not the subject has sleep apnea based on the calculated respiratory rate.

19. The method of claim 18, wherein the first region corresponds to a forehead region and the second region corresponds to a cheek region.

20. A computer-readable recording medium having recorded thereon a computer program for executing the healthcare monitoring method according to any one of claims 14 to 19.

Images