Devices, systems, and methods for supporting improved respiratory monitoring

Abstract

An apparatus for supporting improved respiratory monitoring comprises: an input unit configured to obtain a set of image frames of a region of interest of a subject; a processing unit configured to analyze the set of image frames to estimate an amount of motion of the object in a first direction and an amount of motion in a second direction, where the second direction is substantially perpendicular to the first direction; an evaluation unit configured to: compare the amount of exercise in the first direction to the amount of exercise in the second direction, and determine a respiratory monitoring quality based on the comparison; and an output unit configured to provide a quality signal indicative of the respiratory monitoring quality.

Description

TECHNICAL FIELD

[0001] The present invention relates to a device for supporting improved respiration monitoring. The invention also relates to a corresponding system and method. BACKGROUND

[0002] Vital signs such as heart rate (HR), respiratory rate (RR) or blood oxygen saturation are important indicators of a person’s health and well-being and predictors of acute medical conditions and chronic disease states. For this reason, vital signs are widely monitored in inpatient and outpatient care settings, at home or in other medical, leisure and fitness settings.

[0003] In a clinical setting, observing respiratory activity (respiratory rate) is highly relevant. RR is one of the most important basic vital signs to assess the health status of a subject (patient). In an intensive care unit (ICU) setting, respiration is routinely measured from an electrocardiogram via ECG electrodes and the measured thoracic impedance changes during respiratory activity.

[0004] Camera-based measurement of vital signs, in particular RR, allows for non-contact measurement, improving subject comfort and reducing infection risk, since cleaning of contact probes or use of disposable electrodes is no longer required. Furthermore, non-contact vital sign measurement can reduce staff time and consumable costs and reduce environmental impact.

[0005] In general, non-contact measurement also allows for vital sign measurement in use cases where vital sign data can provide additional safety or early detection, but is not currently performed due to cost / benefit factors of contact-based methods.

[0006] With respect to RR, a (video) camera can capture respiratory movements of a subject’s chest or abdomen in an image stream. Respiratory movements cause a temporal modulation of certain image features, where the frequency of the modulation corresponds to the respiratory rate of the monitored subject. Examples of such image features are the average amplitude in a spatial region of interest located around the subject’s chest, or the position of the maximum of the spatial cross-correlation of the region of interest in subsequent images.

[0007] However, camera-based respiration monitoring is based on detecting subtle respiratory motion in a selected region of interest (ROI) in the subject's chest / abdominal region. Thus, the quality and reliability of the obtained vital sign information is largely influenced by the quality of the input image data, which is influenced by the proper selection of image contrast and selected region of interest. Moreover, the non-contact nature of these measurements has an increased sensitivity to subject motion. In particular, while respiration rate is important and a large number of monitoring techniques are available, respiration rate is obviously difficult to measure when the subject is moving. For example, radar signal based remote monitoring techniques are typically suppressed by artifacts and thus ineffective during a period of motion.

[0008] For non-contact vital sign monitoring, in particular respiration monitoring, metrics can be used to determine when disturbances such as motion will prevent accurate measurements. These metrics can then be used to suppress measurements to prevent giving false outputs. In fact, there are different methods for filtering disturbances from measurement signals. However, it is preferable to increase the likelihood of correct measurements rather than suppressing false outputs. Thus, instructions can be given to the subject to remain "still", but this is ambiguous and can easily be forgotten, in particular if the subject is physically weak, e.g. as in an emergency department environment. SUMMARY

[0009] It is an object of the present invention to provide a device for supporting improved respiration monitoring, in particular for supporting a higher quality respiration monitoring of a subject. Thus, it is an object of the present invention to support accurate respiration monitoring.

[0010] In a first aspect of the present invention, a device for supporting improved respiration monitoring is presented, the device comprising: an input unit configured to obtain a set of image frames of a region of interest of a subject; a processing unit configured to analyze the set of image frames to estimate an amount of motion of the subject in a first direction and an amount of motion in a second direction, wherein the second direction is substantially perpendicular to the first direction, an evaluation unit configured to compare the amount of motion in the first direction to the amount of motion in the second direction and to determine a respiration monitoring quality based on the comparison; and an output unit configured to provide a quality signal indicative of the respiration monitoring quality.

[0011] In a second aspect of the present invention, a system for supporting improved respiration monitoring is presented, the system comprising: an imaging device configured to capture a set of image frames of a region of interest of a subject; An apparatus for supporting improved respiratory monitoring; and a user interface configured to provide feedback to a user on the quality of the respiratory monitoring based on the quality signal.

[0012] In yet further aspects of the application, corresponding methods, computer programs, and non-transitory computer-readable recording media are provided. The computer program comprises program code means for causing a computer to perform the steps of the method disclosed herein when said computer program is executed on a computer, and the non-transitory computer-readable recording medium has stored thereon a computer program product which, when executed by a processor, causes the execution of the method disclosed herein.

[0013] Preferred embodiments of the application are defined by the dependent claims. It is to be understood that the claimed system, method and computer program have similar and / or identical preferred embodiments as the claimed apparatus, as defined by the dependent claims and as disclosed herein.

[0014] The present application is based on the idea of providing support in respiratory monitoring for a subject (patient) and clinical staff. In particular, feedback on the quality of the respiratory monitoring should be provided to allow for a better monitoring quality. In fact, the idea of the present application is to provide guidance to a subject (or other user) to enter an optimal position, orientation, posture and / or motion (stillness) such that respiratory monitoring can be performed in an optimal way.

[0015] To this end, for example, an object is monitored by an imaging device such as a video camera. The imaging device is directed in particular to a region of interest of the object, preferably a region that allows to detect / measure motion due to respiration. The recorded image frames are then provided to a device for supporting improved respiration monitoring. An input unit of the device is configured to obtain at least two image frames. The image frames do not have to be (directly) consecutive image frames, but can be image frames of (arbitrary) different points in time. A processing unit then analyzes the image frames. In particular, the processing unit is configured to use the image frames to estimate motion (of the object depicted by the image frames), in particular motion in the region of interest. For example, the processing unit can be configured to detect motion or motion in a plurality of pixels (voxels) and / or a group of pixels (voxels) of at least the region of interest of the at least two image frames. Standard motion estimation algorithms can be applied to detect the above-mentioned motion. The processing unit can further set a first (reference) direction and a second (reference) direction (in the image plane of the group of images), the second (reference) direction being substantially perpendicular to the first (reference) direction, and can project the detected motion onto the first direction and the second direction to estimate an amount of motion in the first direction and an amount of motion in the second direction (of the object). Alternatively, the detected motion can be filtered with respect to the first direction and the second direction. The amount of motion in the first direction and / or the second direction can be given by a motion signal (translation signal) corresponding to the first direction and / or the second direction, respectively.

[0016] In the evaluation unit, the amounts of motion in the two directions are compared (back and forth) with each other. In other words, the two quantities are related to each other, wherein the relationship (comparison) of the quantities comprises for example a difference of the quantities or a ratio of the quantities. Based on the result of the comparison (relationship), the evaluation unit determines a respiration monitoring quality, in particular a value of a respiration monitoring quality measure. In other words, the processing unit is configured to convert the result of the comparison into a feedback measure. For example, the result of the comparison can be linked to predetermined quality (measure values), for example stored in a look-up table. Based on the quality (in particular the quality measure value), an output unit provides a quality signal indicating the respiration monitoring quality (measure value).

[0017] The user interface can use the quality signal to provide feedback on the quality of the respiration monitoring. The feedback can be given (close to) real-time, i.e. during the respiration monitoring. The user interface can comprise a graphical user interface, e.g. a display or a touch screen, which can show the feedback as a (target) value and / or range. For example, the feedback can be provided in percentage, wherein 100% indicates the highest possible quality measure value. In other words, the feedback can be provided in the form of a quality index. Similarly, the user interface can comprise one or more visual indicators in the form of light sources, wherein the light sources can emit light of a predetermined color and / or a predetermined (temporal) pattern based on the quality signal. In other words, the feedback can have the form of another non-numerical indication, e.g. a “traffic light”, a (a’s) cow eye or an audible cue. However, the user interface is not limited to these examples.

[0018] The feedback can be based on the position, orientation, posture and / or motion of the subject (relative to the imaging device). Information on the position, orientation, posture and / or motion of the subject can be derived from the set of images provided by the imaging device itself or from other information sources.

[0019] By providing feedback to the subject during the respiration monitoring, the subject is provided with information on the quality of the monitoring and possibly suggestions to directly improve said monitoring. This is due to the fact that for non-contact respiration monitoring, the measurement accuracy is particularly high when the subject remains still. Moreover, by providing feedback, the subject is also given something to focus on, which helps to prevent distraction from remaining still (i.e. a distraction that can lead to unwanted motion, i.e. motion that interferes with the respiration monitoring).

[0020] It should be noted that the imaging device of the system can or can not be the same device that is used for non-contact respiration (frequency) monitoring. The imaging device of the system according to the present application is mainly used to capture the motion of the subject, in particular the motion associated with the respiration (inspiration and / or expiration) of the subject. However, this does not necessarily mean that the images captured by said device are further used to derive information on the respiration frequency, respiration rate or other vital signs associated with respiration. In fact, the imaging device of the system can be used to improve the respiration monitoring, while a (further) radar device or camera can be used to perform the measurements associated with respiration.

[0021] It should be understood that in the context of the present application as disclosed herein, the term “patient” is not limited to a user suffering from a disease, but can in general be used for any user or subject.

[0022] In an embodiment of the device for supporting improved respiratory monitoring, the processing unit is configured to estimate the amount of motion of the subject by setting a motion vector reference point in one of the image frames and by determining a motion vector based on a displacement of the vector reference point in another image frame. In particular, the processing unit is configured to estimate the amount of motion of the subject in a first direction by setting a first motion vector reference point in one of the image frames and by determining a first motion vector based on a displacement (i.e. a translation) of the first vector reference point in another image frame in the first direction. Similarly, the processing unit can estimate the amount of motion of the subject in a second direction by setting a second motion vector reference point in one of the image frames and by determining a second motion vector based on a displacement of the second vector reference point in another image frame in the second direction. Preferably, the first motion vector reference point and the second motion vector reference point are identical. Further, preferably, the image in which the first motion reference point and the second motion reference point are set is identical, and also the image from which the displacement of the points is determined is identical. This way it is guaranteed that the amount of motion in both directions is detected at the same time and at the same location. The motion vectors can also be referred to as translation vectors or displacement vectors. In general, the processing unit can be configured to estimate the amount of motion of the subject by determining a series of motion vectors based on a series of displacements of the vector reference points in a series of image frames. This way, a typical back-and-forth motion, in particular for respiration, can be detected / captured.

[0023] In general, any available algorithm can be used to determine the first and second motion vectors in the image frames. For example, respiration effort can be tracked by using motion detection from optical flow. Other examples of such algorithms include ProCor, kernel tracking and face tracking. In fact, some algorithms do not determine the displacement of the reference vector points (directly). However, such a displacement can be determined with any of these methods, for example, by determining the difference of the detected positions in two separate images. In the present application, however, these positions are considered as motion vector reference points.

[0024] In another embodiment, the amount of motion is the absolute value of the motion integrated over time. Thus, the amount of motion can be given by the modulus (or a series of moduli) of the motion vectors, which is integrated over time (in particular, considering the time between the first and last image frame (i.e. between the image frames to which the motion vectors correspond)). More precisely, the amount of motion (or motion change) in the first (second) direction can be given by the modulus of the first (second) motion vector, which is integrated over the time between the image frames to which the first (second) motion vector corresponds.

[0025] However, the amount of motion can also be given by the absolute value of the motion only (i.e. without any integration over time) (i.e. the modulus of the motion vector).

[0026] In general, for the evaluation of the quality of the respiration measurement, the integration over time can be omitted. The respiration signal is typically a proxy for position rather than velocity. In other words, the measurement signal related to respiration monitoring is typically given in units of position. However, for camera-based measurements, the velocity (or the displacement of the reference point of the motion vector between multiple frames) is typically easier to measure accurately than the absolute position. Thus, for camera-based respiration measurements, there is typically an integration step from velocity to position, wherein the (position) signal can then be used for respiration peak detection and / or rate calculation. Thus, by including the integration step, the quality of the determined final (position) signal can represent the true quality in a more reliable way.

[0027] In another embodiment, the evaluation unit is configured to calculate a ratio of the amount of motion in the first direction and the amount of motion in the second direction and to determine the respiration monitoring quality based on the ratio. For example, the evaluation unit can be configured to calculate a ratio of the time integral of the absolute value of the motion in the first direction and the time integral of the absolute value of the motion in the second direction. More specifically, the evaluation unit can calculate a ratio of the time integral of the modulus of the motion vector in the first direction and the time integral of the modulus of the motion vector in the second direction. In other words, the evaluation unit can calculate a ratio of the time integral of the modulus of the translation signal corresponding to the first direction and the time integral of the modulus of the translation signal corresponding to the second direction.

[0028] If the value of the ratio is higher than or equal to a threshold value (e.g. a value of 1), the respiration signal based on the amount of motion in the first direction, or more generally, the respiration signal linked to the first direction (in particular the movement of the object in the first direction) can be considered as reliable. The threshold value can depend on the type of respiration monitoring selected and can thus be set according to the type of respiration monitoring.

[0029] In general, the direction amplitudes or any other indicator of the amount of motion of the object (e.g. generated in an intermediate step of determining the motion vector) can be considered as motion vectors. Thus, the motion of the object can also be represented by a series of direction amplitudes. Thus, the evaluation unit can be configured to compare the amount of motion in the first direction and the amount of motion in the second direction by determining a ratio of the direction amplitudes in the first direction and the direction amplitudes in the second direction (in particular by using a ratio of the modulus of the amplitudes), which are integrated over time (similar to the motion vectors).

[0030] The motion vector / direction amplitude is not limited to images of 2D cameras, but can also be applied to image frames captured by 3D and / or thermal cameras or radio frequency sensors (determined from image frames captured by 3D and / or thermal cameras or radio frequency sensors).

[0031] In embodiments of the device, the first direction is substantially parallel to a height direction of the object and the second direction is substantially parallel to a width direction of the object (or vice versa). In particular, the processing unit can be configured to detect the height direction and the width direction of the object (e.g. from the image frames) and to compare the directions to the first direction and the second direction, respectively. Thus, the processing unit can be configured to detect a divergence of the first direction from the height direction and a divergence of the second direction from the width direction, respectively. Furthermore, the processing unit can be configured to determine how the first direction and / or the second direction need to be changed to match the height direction and / or the width direction, respectively, and to change / adjust the direction(s) accordingly.

[0032] In case the first direction is (substantially) identical to the height direction and the second direction is (substantially) identical to the width direction, the expected respiratory movements (in particular of the chest and the abdomen) are (substantially) limited to the second direction. Thus, if the evaluation unit determines (upon comparison) that the amount of motion in the second direction is much higher than the amount of motion in the first direction, the quality of the respiratory monitoring can be determined to be high. In fact, depending on the comparison of the two amounts of motion, the processing unit can determine a quality index, wherein the higher the quality index, the higher the amount of motion in the second direction relative to the amount of motion in the first direction.

[0033] In yet another embodiment, the input unit, the processing unit, the evaluation unit and the output unit are configured to work in real-time. Thus, both obtaining the image frames and providing the quality signal are quasi-instantaneous. In other words, the device is configured to perform real-time processing. In case the device is part of a system comprising a user interface, this allows feeding back information to the user about the quality of the respiratory monitoring in real-time.

[0034] In further embodiments, the processing unit is configured to generate a respiratory signal from the set of image frames, wherein the processing unit is configured to filter the respiratory signal. The respiratory signal can be indicative of the number of breaths of the object within a predetermined time. In fact, the respiratory signal can be a signal indicative of the amount of motion in the first direction and / or the second direction (and thus indicative of the number of breaths within a predetermined time). Thus, the respiratory signal can also be referred to as a motion signal and the motion signal can be subjected to filtering, in particular frequency filtering.

[0035] In particular, the processing unit can be configured to estimate the number of breaths of the object in the time span between two image frames of the set of image frames by using (in the time span) the estimated amount of motion of the object in the first direction and / or the second direction (e.g. by using a series of motion vectors). However, the respiration signal can be determined from the set of image frames in other ways. The respiration signal can then be further filtered. For example, the processing unit can be configured to filter the respiration signal with respect to a predetermined frequency band (i.e. to filter out (and / or cut off) predetermined frequencies). To this end, a band-pass filter can be used which allows frequencies within an expected range of vital signs (i.e. 5 to 60 breaths per minute (for an adult)). Thus, the amount of motion in the first direction and / or the second direction (a signal corresponding to the amount of motion) can be filtered.

[0036] Thus, in order to more accurately determine the quality of respiration monitoring, the evaluation unit can be configured to compare the filtered amount of motion in the first direction with the (filtered) amount of motion in the second direction and to determine the quality of respiration monitoring based on the comparison.

[0037] In another embodiment, the evaluation unit is configured to compare the amount of motion in the first direction with a first threshold value and / or to compare the amount of motion in the second direction with a second threshold value, wherein the evaluation unit is configured to determine a gain factor based on any of the comparisons, respectively. In particular, the evaluation unit can be configured to compare the amount of motion in the first direction with a first threshold value and to determine a first gain factor based on the comparison. Similarly, the evaluation unit can be configured to compare the amount of motion in the second direction with a second threshold value and to determine a second gain factor based on the comparison. In particular, it can be determined whether and to what extent the amount of motion meets the threshold value, respectively. The corresponding gain factor can then be determined. For example, if the amount of motion in the first direction meets (in particular is below) the first threshold value, the gain factor can be set to a high value (low value). On the other hand, if the amount of motion in the first direction does not meet (in particular is the same as or higher than) the first threshold value, the gain factor can be set to a low value (high value). Typically, the gain factor can be any numerical value, preferably normalized to the interval [0, 1].

[0038] In particular, the evaluation unit can be configured to determine the respiratory monitoring quality based on any of the gain factor(s). Indeed, as mentioned above, a gain factor represents a measure of the amount of motion. Generally, it can be expected that the higher the (detected) amount of motion in the direction of the chest movement of the subject during respiration, the better the respiratory monitoring quality. However, very large motions can not be derived from respiratory movements, but can be caused by other motions of the subject. Thus, very large amounts of motion can prevent an accurate measurement. In other words, very large amounts of detected / measured motion are an indicator of low quality of respiratory monitoring (regardless of the comparison of the amount of motion). Thus, it is advantageous to apply a gain factor to the determined quality (and thus to the quality signal and feedback provided by the system).

[0039] In an embodiment of the system for supporting improved respiratory monitoring, the imaging device comprises any of the following: a camera, in particular a video camera; a radar sensor, in particular a Doppler radar sensor; an ultrasound imaging device; a thermal camera, in particular a microbolometer camera; a laser Doppler imaging device; a speckle vibrometry unit; and a depth camera, in particular a time-of-flight camera.

[0040] For example, if the face of the subject is visible, a thermal infrared camera can detect temperature differences and changes in front of the nose / mouth area. A depth camera allows to obtain 3D images. Thus, a time-of-flight (ToF) camera allows depth sensing. However, depth reconstruction can also be achieved by using multiple (video) cameras (stereo principle). Regardless of the kind of the mentioned imaging devices, respiratory measurements using any of these devices benefit from limited body motion, and thus from the support given by the device for supporting improved respiratory monitoring.

[0041] In another embodiment of the system, the feedback provided by the user interface further comprises a suggestion for enhancing the respiratory monitoring quality, the suggestion relating to the position, orientation, posture and / or motion of the subject. The suggestion can comprise a signal to have the subject stand still or to move to a position where he / she has better visibility to the imaging devices of the system. In particular, the suggestion can comprise an instruction for the subject to change his / her position, orientation and / or posture such that a first direction is substantially parallel to the height direction of the subject and a second direction is substantially parallel to the width direction of the subject (or vice versa). Thus, the device can comprise an instruction to turn the subject by 90° or to turn, stand up, lie down, etc.

[0042] To provide said feedback, the device for supporting improved respiratory monitoring, in particular the processing unit of the device, can be configured to detect a height direction and a width direction of the subject and to compare said (height and width) directions to a first direction and a second direction, respectively. Thus, the processing unit can be configured to detect a divergence of the first direction from the height direction and a divergence of the second direction from the width direction, respectively. Further, the processing unit can be configured to determine how the first direction and / or the second direction need to be changed to match the height direction and / or the width direction, respectively, and to adjust said (first and second) directions accordingly. Similarly, the processing unit can be configured to determine how the height direction and / or the width direction of the subject need to be changed to match the first direction and / or the second direction, respectively. Based thereon, the processing unit can be configured to provide a signal comprising instructions for the subject to change his / her position, orientation, posture and / or movement such that the first direction is identical to the height direction and / or the second direction is identical to the width direction. Based on said signal, the user interface of the system can provide the above-mentioned suggestions.

[0043] In yet another embodiment of the system, the user interface is configured to provide said feedback in real-time and / or the user interface is configured to provide said feedback as visual feedback, auditory feedback and / or haptic feedback.

[0044] Since the device for supporting improved respiratory monitoring can be configured to work in real-time, the quality signal can be provided in real-time. Thus, the user interface of the system is able to provide feedback on the quality of the respiratory monitoring in real-time. Therefore, the health professional and the subject can get immediate feedback on the quality of the ongoing respiratory monitoring and can intervene immediately to improve said monitoring. This allows a faster, more reliable and more accurate clinical diagnosis.

[0045] The feedback of the system can be provided in different ways and thus can be adapted to the health status of the selected subject and the monitoring method. For example, if the subject is blind or if the clinical staff can not supervise the monitor due to other tasks to be done, the user interface can comprise a loudspeaker to provide auditory feedback on the quality of the respiratory monitoring. Similarly, the user interface can comprise a screen indicating the optimal monitoring position of the subject and the actual position. Thus, the subject is able to see whether his / her position is optimal and how to move to enter the optimal position for respiratory monitoring. As another example, when the position, orientation, posture and / or movement of the subject does not allow a high quality monitoring, the subject can stand on a platform that is able to vibrate during the respiratory monitoring, wherein the platform vibrates.

[0046] Preferably, the feedback is provided in a continuous manner. However, the feedback can also be provided in (time) intervals, in particular regular (time) intervals. BRIEF DESCRIPTION OF DRAWINGS

[0047] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter. In the following drawings: Figure 1 a schematic diagram of an embodiment of a device according to the application is shown, Figure 2 a subject (patient) 50 and orthogonal reference directions during respiration according to the application are shown, Figure 3 a flow chart of a first embodiment of a method according to the application is shown, Figure 4 a flow chart of an embodiment of the steps of a first embodiment of a method according to the application is shown, Figure 5 a first set of graphs of translation signals indicative of the amount of motion in the horizontal direction and in the vertical direction, respectively, and a comparison thereof are shown, Figure 6 a second set of graphs of translation signals indicative of the amount of motion in the horizontal direction and in the vertical direction, respectively, and a comparison thereof are shown, Figure 7 a third set of graphs of translation signals indicative of the amount of motion in the horizontal direction and in the vertical direction, respectively, and a comparison thereof are shown, Figure 8 a schematic diagram of an embodiment of a system according to the application is shown, Figure 9 various graphs illustrating feedback on the quality of respiratory monitoring provided by a system according to the application are shown, and Figure 10 graphs illustrating suggestions for supporting respiratory monitoring performed by a system 1 of the application are shown. DETAILED DESCRIPTION

[0048] Figure 1 a schematic diagram of an embodiment of a device 10 according to the application is shown. The device 10 comprises an input unit 14, a processing unit 16, an evaluation unit 20 and an output unit 24. The device 10 can comprise circuitry, e.g. a processor, processing circuitry, a computer, dedicated hardware, etc. performing the functions of the device. In another embodiment, separate units or elements representing the circuitry together can be used.

[0049] In this embodiment, the input unit 14 is configured to obtain a set of image frames 12, in particular an image stream, of a region of interest of the subject 50. The set of image frames 12 can be obtained, for example, from an imaging device or from a storage device. To this end, the input unit 14 can be coupled or connected, directly or indirectly (e.g. via a network or a bus), to the imaging device or to the storage device. Thus, the input unit 14 can be, for example, a (wired or wireless) communication interface or a data interface, e.g. a Bluetooth interface, a Wi-Fi interface, a LAN interface, an HDMI interface, a direct cable connection or any other suitable interface allowing data transfer to the device 10. The obtained set of image frames 12 can specifically show a chest region of the subject 50. The set of image frames 12 can be provided to the processing unit 16.

[0050] In the processing unit 16, the motion of the subject between at least two (not necessarily successive) image frames 12 can be determined using parts or all of the region of interest. In particular, the processing unit 16 can analyze the image frames 12 (in particular in the chest region) and can estimate the amount of motion of the chest of the subject in two directions, which are substantially orthogonal to each other. In particular, the processing unit 16 estimates the amount of motion of the chest in a first direction and in a direction perpendicular to the first direction (second direction). The amount of motion can be provided by the distance the chest moves in the first direction and in the second direction, respectively. Then, motion data 18 comprising the estimates of the amount of motion in these two directions (e.g. a translation signal indicative of the motion in the first direction and a translation signal indicative of the motion in the second direction) are provided to the evaluation unit 20, which compares the amount of motion in these two directions to each other. In other words, the evaluation unit 20 compares the distance the chest moves in the first direction to the distance the chest moves in the second direction. Based on the motion comparison, the evaluation unit 20 determines a quality measure 22 of the subject monitoring. Both the processing unit 16 and the evaluation unit 20 can be any kind of device configured to perform the following operations: processing the image frames 12 to estimate the amount of motion of the subject in a first direction and in a second direction by using the frames, and determining the quality of the respiratory monitoring by comparing the amounts of motion. The processing unit 16 and the evaluation unit 20 can be implemented in software and / or hardware, e.g. as programmed processors, computers, laptops, PCs, workstations, etc.

[0051] By using the quality / quality metric 22 of the respiratory monitoring determined by the evaluation unit 20, the output unit 24 provides a quality signal 26. The output unit 24 can typically be any interface that provides the generated signal (e.g., transmits the generated signal to another device or provides the generated signal for retrieval by another device, such as transmitting the generated signal to another computer or directly transmitting the generated signal to the user interface 40 for display). Therefore, the output unit 24 can typically be any (wired or wireless) communication or data interface.

[0052] Figure 2 The figure shows the object 50 and orthogonal reference directions during respiration. Solid lines indicate the object in his / her expiratory state. Dashed lines indicate the object in his / her inspiratory state, i.e., in a state of lung expansion, with the thoracic cavity expanding and the rib cage extending outwards. In fact, when the object is seated or standing, the predominantly observed respiratory movement (due to inspiratory / expiratory) is lateral. This is due to… Figure 2 A double arrow parallel to the x-axis indicates this. On the other hand, when the object inhales or exhales, no movement or at least very little movement in the y-direction is expected. Therefore, a measure of respiratory movement (i.e., movement of the object's body (particularly the chest and abdomen)) can be the integral of an absolute measurement of vertical translation (or "distance traveled") (i.e., translation in the y-direction within a certain movement window) divided by the translation in the horizontal direction (x-direction). A value significantly greater than 1 in this ratio will indicate that vertical movement is stronger than horizontal movement. In this case, the detected movement may not originate from breathing, but from other movements of the object. In other words, if the ratio is greater than 1, respiratory monitoring is unreliable. For example, the ratio is most effective if the x-axis and y-axis are chosen to be (substantially) parallel to the object's width and height directions, where the object's width direction can be defined by a straight line between the tops of the object's left and right shoulders. Similarly, the object's height direction can be defined by a straight line indicating the distance from the head to the toes.

[0053] Typically, another set of directions (preferably orthogonal) can be chosen, for example, as directions that are not parallel to but perpendicular to the height of the object, such as... Figure 2 The directions x′ and y′ are shown.

[0054] Figure 3 A flowchart of a first embodiment of the method 100 according to the present invention is shown. The steps of the method can be performed by the device 10, wherein the main steps of the method are performed by the processing unit 16 and the evaluation unit 20. The method can be implemented, for example, as a computer program running on a computer or processor.

[0055] In a first step 102 of the method 100, a set of image frames 12 of a region of interest of a subject is obtained (received or retrieved). The image frames 12 can be obtained, e.g., directly from a camera or from a storage device (e.g., a buffer or memory or an image repository, e.g., a subject record stored in a hospital) via a wired or wireless network.

[0056] In a second step 104 and 104', the image frames are analyzed with respect to the motion of the subject in the region of interest in two different directions. In particular, in step 104, the amount of motion of the subject in a first direction is estimated, and in step 104', the amount of motion of the subject in a second direction is estimated, wherein the second direction is substantially perpendicular to the first direction. For this purpose, generally any available algorithm can be used. Examples of motion estimation algorithms include ProCor, optical flow, kernel tracking, and face tracking. By using these algorithms, a motion vector in the first direction and a motion vector in the second direction can be determined, respectively. In this embodiment of the method 100, steps 104 and 104' are performed in parallel. However, the steps can likewise be performed sequentially, with step 104 followed by step 104', or vice versa.

[0057] Once the amount of motion in the first direction and the amount of motion in the second direction have been estimated, respectively, these two quantities are compared to each other in step 106. In step 108, a quality of respiration monitoring measure is determined based on the comparison in step 106. In step 110, a quality signal is generated based on the quality measure and provided for further processing.

[0058] Figure 4 A flow chart illustrating an embodiment of step 104 of the first embodiment of the method 100 is shown. In this embodiment, step 104 is split into three sub-steps 1042, 1044 and 1046. As mentioned above, in step 104, the amount of motion of the subject in a first (reference) direction is estimated. In particular, in a first sub-step 1042, a motion vector reference point is set in one of the image frames. Based on the displacement / translation of the vector reference point in another (successive) image frame, a motion vector is determined. In sub-step 1044, the absolute value of the motion vector is determined. In other words, the modulus of the motion vector is determined. Subsequently, the modulus of the motion vector is integrated over time.

[0059] Figure 5 , Figure 6 and Figure 7A set of graphs showing translation signals indicative of the amount of motion in the horizontal direction and the amount of motion in the vertical direction, respectively, and a comparison thereof are shown. In particular, the figures show the amount of motion of a portion of an object, which portion is for example represented by a motion vector reference point. The signals shown, which are indicative of the amount of motion in different directions, are derived from time series of horizontal motion vectors and vertical motion vectors, i.e. from a set comprising a plurality of image frames. In the upper left graph, the horizontal motion of the object (portion) is represented by a horizontal translation signal. In the upper middle graph, the absolute value of the horizontal translation signal is shown. The lower left graph and the lower middle graph show the vertical translation signal and the absolute value thereof, respectively. From the graphs it can be seen that the amplitude of the vertical translation signal is significantly higher than the amplitude of the horizontal translation signal. This means that the detected motion of the object travels in the vertical direction rather than in the horizontal direction. In the given case, a quality measure for determining the quality of the respiratory monitoring of the object is given by the ratio of the time integral over the absolute translation in the vertical direction and the time integral over the absolute translation in the horizontal direction. This ratio, i.e. the quality measure, is shown in the right graph. From this graph it can be seen that the value of the quality measure is significantly higher than 1. In this case, it can be assumed that it is very likely that the respiration can be retrieved from the vertical signal in a reliable manner. Figure 5

[0060] Figure 6 In the upper left graph, the horizontal motion of the object (portion), i.e. the (horizontal) translation signal, is shown. In the upper middle graph, the absolute value of the horizontal translation signal is shown. The lower left graph and the lower middle graph show the vertical translation signal and the absolute value thereof, respectively. It can be seen that the horizontal translation signal is very different from the vertical translation. However, the amplitudes of the signals are similar. On the right, a graph of the ratio of the time integral over the absolute translation in the vertical direction and the time integral over the absolute translation in the horizontal direction is shown, which ratio represents the quality (measure) for the respiratory monitoring. The measure is shown to be higher than the value 1. Thus, there is a reasonable likelihood that a reliable determination of features regarding the respiration can be retrieved from the vertical signal.

[0061] Figure 7 In the upper left graph, the horizontal motion of the object (portion), i.e. the (horizontal) translation signal, is shown. In the upper middle graph, the absolute value of the horizontal translation signal is shown. The lower left graph and the lower middle graph show the vertical translation signal and the absolute value thereof, respectively. According to the measurement signals shown in the graphs, the (absolute) horizontal translation signal dominates over the (absolute) vertical translation signal. The quality measure given by the ratio of the time integral over the absolute translation in the vertical direction and the time integral over the absolute translation in the horizontal direction is lower than 1. Thus, it is less likely that the respiratory monitoring can be retrieved from the vertical translation signal in a reliable manner.

[0062] Figure 8 ​A schematic diagram illustrating a first embodiment of a system 1 according to the present application is shown. In this embodiment, the system 1 comprises a device 10 for supporting improved respiratory monitoring in an imaging device 30 and a user interface 40.

[0063] The imaging device 30 is configured to perform respiratory monitoring of a subject and comprises in this embodiment a video camera. However, in general, the imaging device 30 can be a radar sensor, an ultrasound imaging device, a thermal camera, a laser Doppler imaging device, a speckle vibrometry unit or a time-of-flight camera, which are in conventional cases used to obtain image data of a subject, e.g. image data of a specific body region of a subject. The imaging device 30 can be implemented, e.g. in software and / or hardware, as or in a processor or computer. For example, a programmed processor can be comprised in the imaging device 30, which can execute a computer program which can be stored in a memory accessed by the processor.

[0064] In this embodiment, the video camera is directed at the subject's chest to capture an image stream comprising a set of image frames 12 of the subject over a predetermined time period. The imaging device 30 provides the image frames 12 to the device 10, which analyzes the image frames 12 and provides a quality signal 26 indicative of the quality of the respiratory monitoring by the imaging device 30.

[0065] The user interface 40, which is in this embodiment a touch screen, then provides feedback on the quality of the respiratory monitoring based on the quality signal. In particular, the touch screen is configured to display the quality, e.g. in terms of a quality index. Alternatively or additionally, feedback on the quality of the respiratory signal can be provided in a color-coded manner by the user interface 40.

[0066] Figure 9 Various diagrams illustrating feedback of the quality of the respiratory monitoring provided by the system 1 according to the present application are shown. In particular, these diagrams show a quality index q over the respiratory monitoring time t. The quality index q is indicative of the quality of the respiratory monitoring by the imaging device 30 of the system 1. In the left diagram, the quality index is rather low and lies only in the dark shaded area. This means that the respiratory monitoring quality is low, so the respiration is monitored in an unreliable way. In embodiments, the meaning of the illustrated quality index can be better visualized by shading the dark shaded area in red and the light shaded area in green.

[0067] In the middle and right diagrams, the quality index q can be considered to represent a reliable respiratory monitoring, since the index clearly lies in the light shaded area, i.e. the quality index is above a predetermined threshold (indicated by the transition from the dark shaded area to the light shaded area). However, since the value of q is generally higher in the middle diagram than in the right diagram, the respiratory monitoring quality corresponding to the middle diagram can be considered to be higher than the respiratory monitoring quality corresponding to the right diagram.

[0068] Figure 10 A figure is shown which illustrates a recommendation performed by the system 1 according to the application for supporting the respiratory monitoring. In particular, Figure 10 Screens of the user interface 40 of the system in three different situations are shown. The screen depicted on the left shows a position template (dotted line) corresponding to an optimal position (and orientation) of the head and shoulders of the subject, the indicated position (and orientation) allowing optimal respiratory monitoring. Said screen of the user interface can thus be seen as a guidance or advisory means of where the subject has to position himself / herself. In addition, as depicted on the middle screen, the user interface can also provide feedback (solid line) on the actual position of the subject. In other words, the feedback screen can show a camera view of the subject overlaid with an outline of the optimal subject position (abstracted by the solid line). From the middle screen it can be seen that the actual position of the subject does not correspond to the optimal position indicated by the template. However, from the display of the two positions, the subject is advised how to move to occupy the optimal position. The screen on the right shows a coinciding subject position and position template. The right screen thus displays a position of the subject having an optimal position for respiratory monitoring.

[0069] While the application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the application is not limited to the disclosed embodiments. Other

[0070] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single element or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0071] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for supporting improved respiratory monitoring, the device (10) comprising: an input unit (14) configured to obtain a set of image frames (12) of a region of interest of a subject (50); a processing unit (16) configured to analyze the set of image frames (12) to estimate an amount of motion of the subject (50) in a first direction and an amount of motion in a second direction, wherein the second direction is substantially perpendicular to the first direction, an evaluation unit (20) configured to compare the amount of motion in the first direction to the amount of motion in the second direction and to determine a respiratory monitoring quality based on the comparison; and an output unit (24) configured to provide a quality signal (26) indicative of the respiratory monitoring quality.

2. The device (10) according to claim 1, the processing unit (16) is configured to estimate the amount of motion of the subject by setting a motion vector reference point in one of the image frames and by determining a motion vector based on a displacement of the vector reference point in another image frame. wherein 3. The device (10) according to claim 1 or 2, the amount of motion is an absolute value of motion integrated over time. wherein 4. The device (10) according to any one of the preceding claims, the evaluation unit (20) is configured to calculate a ratio of the amount of motion in the first direction to the amount of motion in the second direction and to determine the respiratory monitoring quality based on the ratio. wherein 5. The device (10) according to any one of the preceding claims, the first direction is substantially parallel to a height direction of the subject (50) and wherein the second direction is substantially parallel to a width direction of the subject (50). wherein 6. The device (10) according to any one of the preceding claims, the input unit (14), the processing unit (16), the evaluation unit (20) and the output unit (24) are configured to work in real-time. wherein the processing unit (16) is configured to generate a respiratory signal from the set of image frames, wherein the processing unit (16) is configured to filter the respiratory signal.

7. The apparatus (10) according to any one of the preceding claims, wherein, 8. The device (10) according to any one of the preceding claims, the evaluation unit (20) is further configured to compare the amount of motion in the first direction to a first threshold value and / or to compare the amount of motion in the second direction to a second threshold value, wherein wherein the evaluation unit (20) is configured to determine a gain factor based on any of the comparisons.

9. The device (10) according to claim 8, the evaluation unit (20) is configured to determine a respiratory monitoring quality based on the gain factor. wherein, 10. A system (1) for supporting improved respiratory monitoring, the system comprising: an imaging device (30) configured to capture a set of image frames of a region of interest of a subject; a device (10) according to any one of claims 1-9; and 11. The system (1) according to claim 10, the imaging device (30) is configured to capture the set of image frames in real-time. a user interface (40) configured to provide feedback on the respiratory monitoring quality to a user based on the quality signal.

11. The system (1) according to claim 10, wherein the imaging device (30) comprises any of: a camera, in particular a video camera; a radar sensor, in particular a Doppler radar sensor; an ultrasound imaging device; a thermal camera, in particular a microbolometer camera; a laser Doppler imaging device; a speckle vibrometry unit; and a depth camera, in particular a time-of-flight camera.

12. The system (1) according to claim 10 or 11, wherein the feedback further comprises a suggestion for enhancing the respiratory monitoring quality, the suggestion relating to a position, orientation, posture and / or motion of the subject.

13. The system (1) according to any of claims 10-12, wherein, the user interface (40) is configured to provide the feedback in real-time, and / or wherein the user interface (40) is configured to provide the feedback as visual feedback, auditory feedback and / or haptic feedback.

14. A method for supporting improved respiratory monitoring, the method comprising: obtaining a set of image frames of a region of interest of a subject; analyzing the set of image frames to estimate an amount of motion of the subject in a first direction and an amount of motion in a second direction, wherein the second direction is substantially perpendicular to the first direction, comparing the amount of motion in the first direction to the amount of motion in the second direction and determining a respiratory monitoring quality based on the comparison; and providing a quality signal indicative of the respiratory monitoring quality.

15. A computer program comprising program code means for causing a computer to perform the steps of the method according to claim 14 when said computer program is carried out on the computer.

Images