Deriving information about a person's sleeping and waking states from a sequence of video frames

Abstract

In order to obtain information about a person's sleep state and wake state, a device (100) is used, which includes a camera (10) and a processing unit (20). The camera (10) is used to capture a sequence of video frames during a time period, and the processing unit (20) is configured to process the video frames provided by the camera (10) and provide an output, which represents the sleep state and wake state of the person during the time period. In particular, the processing unit (20) is configured to execute an algorithm, according to which the following items are determined: (i) a motion value-time relationship; (ii) a set of features related to the corresponding period in the motion value-time relationship; and (iii) a classifier for the corresponding period, wherein the algorithm is further configured to apply an adaptive prior probability determined for a specific person based on the motion value of the corresponding period to the classifier.

Description

Technical Field

[0001] The present invention relates to a device designed to derive information about a person's sleeping state and waking state from a sequence of video frames, the device comprising a camera for capturing the sequence of video frames during a time period; and a processing unit configured to process the video frames provided by the camera by executing an algorithm and provide an output representing the sleeping state and waking state of the person during the time period.

[0002] The invention also relates to a computer program product, comprising a program code of a computer program, the program code being used to make the computer execute the algorithm of the mentioned device when the computer program is loaded on the computer. Background Art

[0003] Sleep plays an important role in the development of newborns (both full-term and premature infants). Monitoring the sleep of infants is essential for explaining the physical and mental development of infants and the quality of sleep. Such monitoring is usually based on the assessment of wakefulness-sleep patterns during representative time periods. In order to obtain reliable information about wakefulness-sleep patterns, extended monitoring with multiple days is recommended.

[0004] Sensors attached to the fragile skin of infants may cause irritation and damage the skin. Therefore, it is desirable to use inconspicuous (non-contact) monitoring technology. Many inconspicuous devices or systems or the most inconspicuous devices or systems have been developed and used to perform objective infant sleep monitoring. The operation of such devices or systems is based on the use of various sensing technologies and various tools, such as actigraphy, ballistocardiography, capacitive electrocardiography, inductive sensors, photoplethysmography, and near-infrared cameras or thermal cameras. In particular, camera-based baby monitors have been well commercialized in the market because their advantage is that monitoring is completely inconspicuous.

[0005] With regard to the use of video cameras, it should be noted that advanced image processing algorithms for human tracking that attempt to recognize the infant's face and / or body and corresponding movements are expected to work well in identifying the infant's sleeping state. However, these algorithms are computationally expensive because they need to process a large number of video frames, and thus may require the use of high-performance computing units that are difficult and expensive to integrate into cameras.

[0006] US 2016 / 364617 A1 relates to a remote biometric monitoring system that may include a digital camera having a digital sensor, a processor, and a memory. The camera's processor may locally execute one or more algorithms to perform computer vision analysis on captured images of a sleeping subject to determine the subject's activity state. The activity state may include a sleeping state.

[0007] For example, an actigraphy analysis algorithm may be applied that includes steps or operations configured to determine whether a subject is moving based on a series of images captured by a digital camera. Such an algorithm may include the following steps: (i) applying an optical flow algorithm to the images; (ii) determining the average motion level for selected time events; (iii) determining whether the subject is moving by comparing the average motion to a selected threshold; (iv) accumulating the results for each event into epochs; (v) smoothing and filtering the epoch-based motion determinations as needed; and (vi) determining the subject's activity state based on a threshold analysis.

[0008] Gunnarsdottir Kristin M et al., "A Novel Sleep Stage Scoring System: Combining Expert-Based Rules with a Decision Tree Classifier" (2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC), IEEE, July 18, 2018, pp. 3240-3243) discloses an algorithm that utilizes a likelihood ratio decision tree classifier and extracts features from EEG, EMG, and EOG signals based on predefined rules of the American Academy of Sleep Medicine Manual. Features are calculated in 30-second epochs in the time and frequency domains of the signal and used as input to a classifier that assigns each epoch to one of five possible stages.

[0009] JP 2012000375 A relates to a sleep state determination device which is capable of determining a person's sleep state and includes (i) a body movement amount calculation device for calculating time series waveform data of the body movement amount based on data obtained by detecting the body movement of a subject during sleep by a camera; (ii) a body movement extraction device for extracting large body movements that are continuously performed for a predetermined time or longer from the calculated time series waveform data of the body movement, and eliminating the body movements in a body movement constraint segment in which the large body movements are constrained; and (iii) a determination device for calculating the stillness duration of the large body movements in each body movement constraint segment to extract the time interval of the large body movements, and determining into which of a plurality of sleep depth stages the subject's sleep state is classified based on the time interval of the large body movements and the body movement amount of the large body movements.

[0010] WO 2016 / 193030 A1 relates to a method and system for determining a sleep state of a subject based on sensor data obtained from a microphone and a camera monitoring the subject and from a body movement sensor sensing the body movement of the subject. The audio data, the video data and the body movement data are analyzed to determine the sleep state of the subject as one of the following: an awake vocal state, an awake non-vocal state, a REM sleep state, and a deep sleep state or a light sleep state.

[0011] WO 2017 / 196695 A2 relates to a video monitoring system comprising a camera head including an infrared illumination source and an image sensor. The camera head is maintained in a fixed position and orientation above a crib so that the image sensor captures images of the crib and an intervention area adjacent to the crib from a fixed perspective. The system may include a server, wherein the camera head is configured to transmit a streaming video signal to the server, and the server is configured to analyze the video signal to extract and provide behavioral information about the sleep pattern of an infant in the crib. The server may also be configured to analyze the video signal to detect actions taken by a caregiver in the intervention area. Summary of the invention

[0012] The object of the present invention is to provide a reliable way of monitoring the sleeping and waking states of a person without the use of expensive equipment. In this regard, it should be noted that the fact that the present invention is presented and explained in the context of sleep analysis of infants should not be understood as implying that the invention is limited to this context. The fact is that the present invention is equally applicable to monitoring the sleeping and waking states of older children and adults.

[0013] According to the present invention, a device is provided, which is designed to derive information about a person's sleep state and wake state from a sequence of video frames. The device includes a camera for capturing a sequence of video frames during a time period; and a processing unit, which is configured to process the video frames provided by the camera and provide an output, which represents the sleep state and wake state of the person during the time period, wherein the processing unit is configured to execute an algorithm, according to which (i) a motion value-time relationship is determined from the video frames; (ii) a feature set related to a corresponding period in the motion value-time relationship is determined by extracting a number of different features from the motion values ​​in each period in the corresponding period; and (iii) a classifier for the corresponding period is determined by classifying the corresponding feature set related to the corresponding period as representing a sleep state or a wake state of the person. The algorithm is also configured to apply an adaptive prior probability determined for a specific person based on the motion value of the corresponding period to the classifier.

[0014] When the present invention is put into practice, in the following further description of various aspects of the invention, a non-contact device in the form of a video camera is used to capture a video stream of a person assumed to be an infant. A processing unit is provided for executing an algorithm configured to quantify body motion from the video stream as a step in a reliable sleep / wake identification process. Any suitable technique for determining motion value-time relationships from video frames can be applied. For example, the algorithm can be configured to apply a technique known as 3D recursive search motion estimation to identify video body motion records over time.

[0015] The camera may be of any suitable type, and an RGB camera or an infrared camera may be used, for example. When it comes to the position of the camera relative to an area where the baby may be present (such as a bed or incubator), all that is required is that the camera is positioned so that it can see at least a portion of the baby's body. Beyond this, there are no specific requirements regarding camera placement.

[0016] A set of features representing body movement and possibly the relative likelihood of being asleep is extracted from the motion value-time relationship within the corresponding period in the total time period (recording). In order to reduce the variability between infants and / or between records, it may be advantageous to normalize the features of each record. The corresponding set of features associated with the corresponding period is classified as representing a sleeping state or a waking state of a person so that a classifier is determined for each period in the corresponding period. For example, the algorithm can be configured to apply a Bayesian-based linear discriminant analysis to determine a classifier for the corresponding period, which does not change the fact that other types of analysis can also be applied within the framework of the present invention.

[0017] Putting the invention into practice also includes performing a personalized approach to adjust the prior probabilities of the classifier for each record so as to find the prior probabilities for each specific person in each specific record. Information about the way to further adjust the classifier can be used for machine learning.

[0018] Any information that can be derived from the distribution of motion values ​​in the corresponding period and / or the total motion percentage over the entire record can be used to determine the prior probability (priors). For example, the algorithm can be configured to determine the adaptive prior probability based on the number of periods with non-zero motion values ​​or motion values ​​greater than a certain minimum value (i.e., the number of periods with motion values ​​greater than a reference motion value, which is equal to or higher than zero). In this regard, the algorithm can be particularly configured to assign a high prior probability of wakefulness when the number of periods with motion values ​​greater than the reference motion value is equal to or higher than a threshold, which may be an experimentally selected threshold; and assign a low prior probability of wakefulness when the number of periods with motion values ​​greater than the reference motion value is lower than a threshold. Alternatively, the algorithm can be configured to determine the best relationship between the number of periods with motion values ​​greater than the reference motion value and the prior probability of wakefulness. Examples of the number of periods with motion values ​​greater than the reference motion value include percentages or other statistics such as mean, median, standard deviation, interquartile range, and entropy. Practical examples of the best relationship include the best linear relationship such as can be determined by applying linear regression. Other examples include predefined correlation coefficients, non-linear relationships, and piecewise linear / non-linear relationships.

[0019] Further, filters such as median filters can be used for the classifier to smooth the sleep classification results and wake classification results by removing short period deviations from the overall classifier pattern with respect to consecutive periods and generate a final classification result. In order to have a filter as mentioned, the algorithm can be configured to evaluate whether two epoch sequences are interrupted by one epoch or a limited number of epochs, both of which represent a minimum time length and involve a feature set to be classified as representing only one of the person's sleep state or wake state, and one epoch or a limited number of epochs involve one or more feature sets to be classified as representing the other of the person's sleep state or wake state; and if so, one or more classifiers of one epoch or a limited number of epochs are set to be the same as the classifiers of the two epoch sequences. In this way, the result of a smoothing effect is obtained, in which classifiers suggesting short wake states are removed from long sleep periods, and in which classifiers suggesting short sleep states are removed from long wake periods. For example, a number of epochs not greater than 2 or 3 can be considered to be a limited number of epochs.

[0020] The features to be extracted from the motion values ​​in each of the corresponding periods may be any suitable type of features, such as features representing aspects of motion. For example, the features include at least one of the following: (i) the mean of the motion values ​​in each of the corresponding periods; and (ii) the number of non-zero motion values ​​or motion values ​​greater than a certain minimum value (i.e., motion values ​​greater than a reference motion value, which is equal to or higher than zero) in each of the corresponding periods. Additionally, the features to be extracted from the motion values ​​in each of the corresponding periods may include relative sleep likelihood, wherein the algorithm may be configured to include, in the process of determining the relative sleep likelihood, a step of determining the temporal distance of each of the corresponding periods from the most recent period with a high activity level. In this regard, it should be noted that, depending on at least one of the mean value of the motion values ​​in each of the corresponding periods and the number of motion values ​​greater than the reference motion value in each of the corresponding periods, the features to be extracted from the motion values ​​in each of the corresponding periods may also include at least one of the following: (i) relative sleep likelihood, wherein the algorithm is configured to include, in the process of determining the relative sleep likelihood, a step of determining the temporal distance between each of the corresponding periods and the nearest period with a high activity level, and wherein the algorithm is configured to identify the period with a high activity level by obtaining the period with the highest mean value of the motion values, the period with the highest mean value reaching a predetermined maximum percentage of the total number of periods; and (ii) relative sleep likelihood, wherein the algorithm is configured to include, in the process of determining the relative sleep likelihood, a step of determining the temporal distance between each of the corresponding periods and the nearest period with a high activity level, and wherein the algorithm is configured to identify the period with a high activity level by obtaining the period with the highest number of motion values ​​that are greater than the reference motion value, the period with the highest number of periods reaching a predetermined maximum percentage of the total number of periods. It will be appreciated that including relative sleep likelihood in the process of identifying sleep set / wake state contributes to the accuracy of the final output of the process.

[0021] As mentioned previously, the algorithm can be configured to normalize the features so that the effects of variations such as between-babies variations and / or between-record variations can be reduced. Any normalization process can be used, including Z-scores, maximum-minimum, quantiles, percentiles, and unit standard deviation.

[0022] Advantageously, the processing unit is designed for machine learning, thereby even further improving the analysis results, i.e. even better detection of actual situations can be provided during the lifetime of the device. For example, the algorithm can be configured to: determine a machine learning classifier based on the difference between (i) an initial set of classifiers determined based on the features and (ii) a final set of classifiers determined by at least applying an adaptive prior probability; and use the machine learning classifier for adjusting the algorithm in terms of determining the classifier for the corresponding period.

[0023] In particular, in the context of infant care, it may be desirable to take into account the fact that movement may be caused by the action of caring for the infant. In view of this, an apparatus of the present invention that is specifically designed for infant care is feasible and involves a special algorithm that is configured to classify a corresponding set of features associated with a corresponding period as representing a care state of the infant other than a sleeping state or a wakeful state of the infant. In that case, the algorithm may be configured to set the classifier for the period to a care state classifier when the period is a period of activity level above a threshold, the threshold being selected to distinguish between a wakeful state and a care state.

[0024] In a practical embodiment of the present invention, a computer program product may be provided, which includes a program code of a computer program, so that when the computer program is loaded on a computer, the computer is caused to execute the algorithm.

[0025] The above and other aspects of the invention will become apparent and elucidated with reference to the following detailed description of various aspects of the theoretical background of the invention and practical ways of putting the invention into practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will now be explained in more detail with reference to the accompanying drawings, in which the same or similar parts are indicated by the same reference numerals, and in which:

[0027] Figure 1 Diagrammatically shows a camera and a processing unit of a device according to the invention for monitoring the sleeping and waking state of an infant in an incubator;

[0028] Figure 2 is a schematic diagram of various steps of an algorithm to be executed by a processing unit for providing an output representing the sleeping state and the awake state of an infant during a recording period;

[0029] Figure 3 A first example of a graph showing a motion value-time relationship and monitoring results associated therewith;

[0030] Figure 4a second example of a graph showing movement value-time relationships and monitoring results associated therewith; and

[0031] Figure 5 Shown are scatter plots and linear fits between the percentage of epochs with non-zero motion values ​​and the percentage of wakefulness for several recordings. DETAILED DESCRIPTION

[0032] As explained previously, the present invention is about obtaining reliable information about a person's sleep in an unobtrusive manner, in particular by capturing a sequence of video frames and using a processing unit programmed to follow a certain algorithm when analyzing the video frames. In many cases, when considering a certain period of time, it is desirable to know during which periods of this period the person was asleep and during which periods the person was awake. According to the present invention, video detection of the person's movements is the basis of the analysis to be performed in order to obtain the desired knowledge.

[0033] Figure 1 Diagrammatically shown is a camera 100 and a processing unit 20 of a device 100 according to the invention for monitoring the sleeping state and the awake state of an infant 30 in an incubator 40. The positioning of the camera 10 relative to the incubator 40 is not very critical, since in fact the only requirement is to arrange the camera 10 at a position where the camera 10 can record the movements of at least a part of the body of the infant. In order to keep the cost of the device 100 as low as possible, it is advantageous to use only one camera 10, but this does not change the fact that the invention also covers the use of two or even more cameras 10.

[0034] The processing unit 20 may be provided in various ways. Practical examples include the arrangement of the processing unit 20 as an integral part of the camera 10 and the arrangement of the processing unit 20 in a computer system located separately from the camera 10. In any case, the processing unit 20 is arranged and configured so as to be able to receive information from the camera 10. Further, any type of display device such as a screen (not shown) may be used to display information output from the processing unit 20 to the user. For the purpose of conveying information, communication between one or more components of the device 100 may be performed through a wired system or in a wireless manner, wherein the Internet may be used if necessary.

[0035] Figure 2 1 is a diagram of the various steps of an algorithm to be executed by the processing unit 20 in order to provide an output representing the sleeping state and the awake state of the infant during the recording period. The following description refers to the various steps and provides additional information about each of those steps, wherein it is assumed that the present invention is applied to Figure 1The illustrated scenario is one in which it is desired to obtain information about the wake-sleep pattern of an infant 30 in an incubator 40. However, it should be understood that the present invention is also applicable to other scenarios, including home scenarios in which an infant (or an elderly person) can be monitored while lying in bed.

[0036] The motion or video motion record (i.e., motion estimated from the video record) is mainly caused by body movement, parental activity / care, or other disturbances (e.g., from other moving objects). This is very important information that can distinguish whether the baby 30 is present in the incubator 40, thereby enabling further automatic analysis of the baby's sleep. The main concept is as follows: Assuming that the baby 30 usually has a higher degree of body movement during the awake state than during the sleeping state, the body movement of the baby 30 is highly correlated with the baby's sleeping state and awake state.

[0037] In a first step of the analysis of the video frames, a motion estimation technique is employed to quantify the motion into a video actigraphy (VA). In the example shown, the motion estimation technique is assumed to be a technique known as 3D Recursive Search (3DRS). This particular technique has been shown to be robust to scene changes, particularly lighting changes, meaning that the effects of daylight can be eliminated. The video recording type may be RGB in grayscale. Another possible example of a video recording type is NIR, wherein it should be noted that the application of RGB may be most applicable in the context of premature infants, while the application of NIR may be most applicable in the context of full-term infants. Depending on the camera 10 employed, the raw 3DRS motion estimate corresponding to the video recording may have a frame rate of approximately 15 Hz, and may also have another frame rate such as 8 Hz.

[0038] Figure 3 An example of VA values ​​associated with a premature infant 30 is shown, the VA values ​​being obtained by running the 3DRS algorithm for processing video frames having approximately 2 hours of recording taken at a frequency of approximately 15 Hz. Figure 4 An example of VA values ​​associated with a healthy full-term infant 30, obtained by running the 3DRS algorithm for processing video frames with about 24 hours of recording taken at the same frequency of about 15 Hz, is shown. Larger VA estimates generally correspond to greater body motion or more body motion.

[0039] In the second step 2 of the analysis of the video frames, features are extracted from the raw estimated motion data. According to the guidelines of the American Academy of Sleep Medicine (AASM), sleep states should be classified for consecutive non-overlapping periods of 30 seconds. In view of this, it is assumed that features are extracted on a 30-second basis. In the following, it is explained how to extract four features for each period, two of those features are calculated based on the mean of the VA value in the corresponding period, and the other two of those features are calculated by counting the non-zero VA values ​​in the corresponding period. The set of four features for each period obtained in this way is summarized as follows:

[0040] (i) Video actigraphy mean (VAM) = the mean of VA values ​​(motion estimation) within 30 seconds, which is calculated based on the raw VA data after 3DRS motion estimation;

[0041] (ii) Video body motion recording count (VAC) = the (average) count of non-zero VA values ​​(motion estimation) within 30 seconds, which is calculated based on the raw VA data after 3DRS motion estimation;

[0042] (iii) Relative sleep probability based on VAM (PS VAM ) = relative sleep likelihood, measured by the temporal distance from the most recent period with high activity level calculated based on VAM; and

[0043] (iv) Relative sleep probability based on VAC (PS VAC ) = relative sleep likelihood, measured by the temporal distance from the most recent period with high activity level calculated based on VAC.

[0044] In the following, more details of the various features and the manner in which they are determined are provided.

[0045] VAM aims to capture the average amplitude of body movements, while VAC characterizes the movement frequency, i.e., the number of movements, in each epoch. Assume that the raw VA data for a 30-second epoch is u = {u1, u2, ..., u k}, where when the frequency of the video frame is 15 Hz, k is 450, and the mean of the VA data in this period is calculated by the following equation:

[0046]

[0047] And the (average) count of non-zero VA data in that period is calculated by the following equation:

[0048]

[0049] in

[0050] For all i (i=1, 2, ..., k)

[0051] In situations where body movement is reduced, such as during quiet wakefulness, it is challenging to identify wakefulness using only VAM or VAC. VAM and PS VAC Further accuracy can be achieved by quantifying the logarithm of the time difference between each epoch and the closest epoch with significant body movement, which corresponds to a large VAM value or a large VAC value. The results can be smoothed by a moving average operation using an experimentally optimized window of 10 minutes. For each recording, for the calculation of PS VAM Value or PS VAC For the purpose of averaging the values, epochs with VAM values ​​or VAC values ​​above the 95th percentile in the entire recording are considered to have high activity levels. It is assumed that epochs that are closer to epochs with higher activity levels and therefore have smaller time differences are more likely to correspond to wakefulness, although they may have less body movement. Assume that there is a series of n epoch-based VAM feature values ​​or VAC feature values ​​a={a1, a2, ..., a n}, then aT is the subset of a whose feature value is greater than the threshold T, where the associated epoch index is eT = {e1, e2, ..., e m}. Accordingly, we have b={b1,b2,...,b n} is from the same series of PS VAM Eigenvalue or PS VAC Then, the value b at period x (x = 1, 2, ..., n) can be calculated x , so that:

[0052] b x =ln(min{|x-e1|, |x-e2|,...,|xe m |})

[0053] T can be experimentally chosen to be, for example, the 95th percentile of feature values ​​in the entire recording. In any case, it follows from the foregoing that the second step 2 of analyzing the video frame may involve: after calculating the two primary features VAM and VAC, obtaining two new, secondary features PS VAM and PS VAC .

[0054] In the third step 3 of analyzing the video frames, features were normalized for each recording in order to reduce the global variability between days and subjects conveyed by VA. VAM and PS VAC , the feature values ​​can be normalized to zero mean and unit standard deviation for the entire record (Z-score normalization), while for VAM and VAC, it may be practical to normalize the feature values ​​between 0 and 1 (max-min normalization). Within the framework of the present invention, any suitable type of normalization method can be applied, wherein it should be noted that the normalization method may be different for different features.

[0055] In a fourth step 4 of analyzing the video frames, a classification is performed based on the feature set of the corresponding epoch. In particular, for each epoch, the feature set of that particular epoch is classified as representing a wakeful state or a sleeping state, and possibly also an out-of-bed state. Figure 3 and Figure 4 In the above diagram, the result of the classification process is shown above the corresponding graph. The classifier for the corresponding period can be determined in any suitable way. For example, it may be practical to use Bayesian linear discriminant analysis in this process.

[0056] The processing unit 20 is configured to take into account the prior probabilities of the classifiers. This step of analyzing the video frames is a parallel step consisting of Figure 2 Indicated by reference numeral 4a in the figure. In general, when making a decision, the prior probability of the classifier is usually set to prefer one category. Based on the assumption that body movement indicates a wake state, intuitively, records with more periods of body movement should have more wake periods, resulting in a higher probability of wake periods and a lower probability of sleep periods. This is due to Figure 5 As confirmed by the significant correlation shown Figure 5 In , the relationship between the percentage of non-zero motion periods and the percentage of wakeful periods is illustrated. Therefore, the present invention proposes a personalized adaptive prior probability (adaptive priors) that depends on the percentage of non-zero motion, which can be calculated based on VAM or VAC, for example. When the prior probability is assumed to be PriW for wakefulness and 1-PriW for sleep, and the percentage of periods with non-zero motion is assumed to be PE, PE is compared with a threshold value S. If it appears that PE is greater than S, a higher PriW (PriW_high) should be assigned, otherwise a lower PriW (PriW_low) should be assigned, such that:

[0057]

[0058] In one example involving experimentally selected values, S is 0.14, the upper PriW is 0.5, and the lower PriW is 0.1. The personalized prior probability can also be determined using a linear regression method, in which the best linear relationship between PE and PriW is established.

[0059] Figure 5 A scatter plot is shown and illustrates a linear fit between the percentage of non-zero movement epochs and the percentage of awake epochs. The scatter plot was derived from 45 recordings for healthy term infants. At p<0.0001, Pearson's correlation coefficient was 0.66.

[0060] In the fifth step 5 of analyzing the video frames, suspicious segmented wakefulness or sleep states of "very short" duration are "filtered out", while relatively longer wakefulness and sleep periods are retained for annotation. Therefore, it is advantageous to apply a "low-pass filter" to smooth the detection results, because in doing so, it is possible to correct for a single or very short misclassified wakefulness and sleep period. The window size can be chosen experimentally to optimize the classification performance.

[0061] Finally, processing unit 20 outputs identification output that can be communicated to and interpreted by a user in order to provide the user with knowledge regarding the recorded wake-sleep behavior of infant 30 during the recorded time.

[0062] In the context of the present invention, experiments were performed to verify the algorithm described above and to check whether the present invention is suitable for obtaining reliable information about the wakefulness-sleep behavior of both healthy full-term infants and premature infants. For premature infants, 45 video records (738 × 480 pixels or 768 × 576 pixels) from 7 infants with an average gestational age of 29.9 weeks were included. For healthy full-term infants, 29 hours (1280 × 720 pixels) of video data from 8 infants with an average age of 6 months were included. The inclusion criteria are as follows: full-term infants (most of the time) need to sleep in their own beds and bedrooms. Artificial annotators score sleep states and wakefulness states in non-overlapping periods lasting 30 seconds. These annotations are used as the gold standard for automatic classification. For premature infants, care will also be annotated and considered to be awake, but the awake period is much less.

[0063] In order to demonstrate the effectiveness of the proposed classification algorithm, a leave-one-subject-out cross-validation (LOOCV) was applied (independent of the subject). The classification performance was evaluated using the overall accuracy and the chance compensation metric Cohen's Kappa coefficient. The following table compares and presents the sleep classification results and the wake classification results for premature infants and healthy full-term infants using different feature sets and settings / methods. It can be seen that using adaptive prior probabilities for each infant and filtering the results can improve the classification performance of both premature infants and full-term infants.

[0064]

[0065]

[0066]

[0067] The sleep classification results and wake classification results can be used for higher level interpretations, such as the total time the infant is lying in bed, the total sleeping time, the total awake time, the number of awakenings, and other infant sleep / wake statistics. As previously mentioned, the present invention also covers the application of the proposed algorithm for the purpose of monitoring the wake-sleep behavior of people older than infants. The device 100 according to the present invention can be used in homes, hospitals, and other environments. The reasons for applying the present invention may be varied, including the desire to obtain information about the sleep quality of a person and the desire to enact as many care actions as possible during the wake state.

[0068] It will be clear to those skilled in the art that the scope of the present invention is not limited to the examples discussed above, but that several modifications and variations thereof are possible without departing from the scope of the present invention as defined in the appended claims. It is intended that the present invention be interpreted as including all such modifications and variations as long as they fall within the scope of the claims or their equivalents. Although the present invention has been described and illustrated in detail in the drawings and the specification, such description and explanation should be considered to be merely illustrative or exemplary and not restrictive. The present invention is not limited to the disclosed embodiments. The accompanying drawings are schematic, and details not necessary for understanding the present invention may have been omitted, and they are not necessarily drawn to scale.

[0069] By studying the drawings, the description and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other steps or elements, and the indefinite article "a" or "an" does not exclude a plurality. Any reference signs in the claims should not be construed as limiting the scope of the invention.

[0070] Unless expressly indicated otherwise, elements and aspects discussed for or in connection with a particular embodiment may be suitably combined with elements and aspects of other embodiments. Thus, the mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0071] The term "comprising" as used herein will be understood by those skilled in the art to encompass the term "consisting of." Thus, with respect to one embodiment, the term "comprising" may mean "consisting of," but in another embodiment, may mean "comprising at least the defined species and optionally one or more other species."

[0072] A possible overview of the invention is as follows. In order to derive information about a person's sleep state and wake state from a sequence of video frames, a device is used, which includes a camera 10 and a processing unit 20. The camera 10 is used to capture a sequence of video frames during a time period, and the processing unit 20 is configured to process the video frames provided by the camera 10 and provide an output representing the sleep state and wake state of the person during the time period. In particular, the processing unit 20 is configured to execute an algorithm, according to which the following items are determined: (i) a motion value-time relationship; (ii) a set of features related to the corresponding period in the motion value-time relationship; and (iii) a classifier for the corresponding period, wherein the algorithm is also configured to apply a personalized adaptive prior probability (i.e., an adaptive prior probability determined for a specific person based on the motion value of the corresponding period) to the classifier.

Claims

1. A device (100) designed to derive information about a person's sleeping state and awake state from a sequence of video frames, the device (100) comprising: - a camera (10) for capturing a sequence of video frames during a time period, and a processing unit (20) configured to process the video frames provided by the camera (10) and to provide an output representing the sleeping state and the awake state of the person during the time period, wherein the processing unit (20) is configured to execute an algorithm according to which, - determining a motion value-time relationship from said video frames, - determining a set of features associated with the respective epoch in the motion value-time relationship by extracting a number of different features from the motion values ​​in each of the respective epochs, wherein the set of features represents a relative likelihood of body movement and / or being asleep; - normalizing said features, - determining a classifier for the corresponding epoch by classifying a corresponding set of features associated with the corresponding epoch as representing a sleeping state or a waking state of the person, and - applying to said classifier an adaptive prior probability, said adaptive prior probability being determined for a specific person as a function of said motion values ​​for said corresponding period; The algorithm is configured to determine the adaptive prior probability as a function of the value of the number of epochs out of all epochs having motion values ​​greater than a reference motion value, the reference motion value being equal to or higher than zero.

2. The apparatus (100) according to claim 1, wherein the algorithm is configured to determine the adaptive prior probability based on a distribution of the motion values ​​within the corresponding period.

3. The apparatus (100) according to claim 1, wherein the algorithm is configured to: assign a high prior wakefulness probability (PriW_high) when the number of periods with motion values ​​greater than the reference motion value is equal to or higher than a threshold (S); and assign a low prior wakefulness probability (PriW_low) when the number of periods with motion values ​​greater than the reference motion value is lower than the threshold (S).

4. The apparatus (100) according to claim 1, wherein the algorithm is configured to determine an optimal relationship between the value of the number of periods having motion values ​​greater than the reference motion value and the a priori probability of wakefulness.

5. The apparatus (100) according to any one of claims 1 to 4, wherein the algorithm is further configured to apply a smoothing filter to the classifier, the smoothing filter removing short-term deviations from the overall classifier pattern with respect to consecutive periods.

6. The apparatus (100) according to claim 5, wherein in order to have the smoothing filter, the algorithm is configured to: evaluate whether two epoch sequences are interrupted by one epoch or a finite number of epochs, both of which represent a minimum time length and involve a set of features to be classified as representing only one of the sleeping state or the waking state of the person, and the one epoch or the finite number of epochs involve one or more sets of features to be classified as representing the other of the sleeping state or the waking state of the person; and if this is the case, set one or more of the classifiers of the one epoch or the finite number of epochs to be the same as the classifiers of the two epoch sequences.

7. An apparatus (100) according to any one of claims 1, 2, 3, 4 and 6, wherein the features to be extracted from the motion values ​​in each of the corresponding periods include at least one of the following: (i) the mean of the motion values ​​in each of the corresponding periods; and (ii) the number of motion values ​​in each of the corresponding periods that are greater than a reference motion value, wherein the reference motion value is equal to or greater than zero.

8. An apparatus (100) according to claim 7, wherein the features to be extracted from the motion values ​​in each of the corresponding periods include relative sleep likelihood, and wherein the algorithm is configured to include, in determining the relative sleep likelihood, a step of determining the temporal distance of each of the corresponding periods from a nearest period with a high activity level.

9. The apparatus (100) according to claim 7, wherein the features to be extracted from the motion values ​​in each of the corresponding periods include at least one of the following: (i) a relative sleep likelihood, wherein the algorithm is configured to include, in the process of determining the relative sleep likelihood, a step of determining the temporal distance of each of the corresponding periods from the most recent period with a high activity level, and wherein the algorithm is configured to identify the period with a high activity level by obtaining the periods with the highest mean value of the motion values ​​reaching a predetermined maximum percentage of the total number of periods; and (ii) a relative sleep likelihood, wherein the algorithm is configured to include, in the process of determining the relative sleep likelihood, a step of determining the temporal distance of each of the corresponding periods from the most recent period with a high activity level, and wherein the algorithm is configured to identify the period with a high activity level by obtaining the periods with the highest number of motion values ​​greater than the reference motion value reaching a predetermined maximum percentage of the total number of periods.

10. An apparatus (100) according to any one of claims 1, 2, 3, 4, 6, 8 and 9, wherein the algorithm is configured to: determine a machine learning classifier based on the difference between (i) an initial set of classifiers determined based on the features and (ii) a final set of classifiers determined by at least applying the adaptive prior probability; and use the machine learning classifier to adjust the algorithm with respect to the classifier determined at the corresponding period of time.

11. An apparatus (100) according to any one of claims 1, 2, 3, 4, 6, 8 and 9, wherein the algorithm is configured to apply 3D recursive search motion estimation to determine the motion value-time relationship from the video frames, and / or apply Bayesian-based linear discriminant analysis to determine the classifier for the corresponding period.

12. The device (100) according to any one of claims 1, 2, 3, 4, 6, 8 and 9 is designed for infant care, wherein the algorithm is configured to classify the corresponding set of features related to the corresponding period as representing a care state of the infant (30) other than a sleeping state or a waking state of the infant, and wherein the algorithm is configured to set the classifier of the period to a care state classifier when the period is a period of activity level above a selected threshold, and the selected threshold is used to distinguish the waking state from the care state.

13. A computer program product comprising a program code of a computer program, so as to cause a computer to execute the algorithm of the device (100) according to any one of claims 1 to 12 when the computer program is loaded on the computer.

Images