Devices, systems, and methods to support improved respiratory monitoring

Abstract

A device supporting improved respiratory monitoring includes an input unit configured to acquire a set of image frames of a target region of interest; a processing unit configured to analyze the set of image frames and estimate the momentum of the target in a first direction and a second direction, wherein the second direction is substantially perpendicular to the first direction; an evaluation unit configured to compare the momentum in the first direction with the momentum in the second direction and determine the quality of respiratory monitoring based on the comparison; and an output unit that provides a quality signal indicating the quality of respiratory monitoring.

Description

Technical Field

[0001] The present invention relates to an apparatus for assisting in improved respiratory monitoring. The present invention further relates to corresponding systems and methods.

Background Art

[0002] Vital signs such as heart rate (HR), respiratory rate (RR), or blood oxygen saturation are important indicators of a person's health and well-being and are also predictors of acute and chronic disease states. For this reason, vital signs are widely monitored in inpatient and outpatient care settings, homes, or other medical, leisure, and fitness settings.

[0003] In a clinical environment, observing respiratory activity (respiratory rate) is very important. RR is one of the most important basic vital signs for evaluating the health status of a subject (patient). In an intensive care unit (ICU), respiration is routinely measured from an electrocardiogram via ECG electrodes, and the measured thoracic impedance changes during each respiratory activity.

[0004] Camera-based measurement of vital signs, particularly RR, enables non-contact measurement, thereby improving the comfort of the subject and reducing the risk of infection. This is because cleaning of contact probes or use of disposable electrodes is no longer required. Furthermore, non-contact vital sign measurement can reduce staff time and the cost of consumables and mitigate the impact on the environment.

[0005] Generally, non-contact measurement enables vital sign measurement in use cases where vital sign data may offer additional safety or early detection but is not currently implemented due to the cost / benefit factors of contact-based methods.

[0006] Regarding respiration (RR), a (video) camera can capture the breathing motion of the subject's chest or abdomen in a stream of images. The breathing motion results in temporal modulation of certain image features, where the modulation frequency corresponds to the number of breaths of the subject being monitored. Examples of such image features include the mean amplitude in a spatial region of interest located around the subject's chest, or the location of the maximum value of the spatial cross-correlation of the region of interest in subsequent images. [Overview of the project] [Problems that the invention aims to solve]

[0007] However, camera-based respiratory monitoring relies on detecting subtle respiratory movements in a selected region of interest (ROI) within the subject's chest / abdominal region. Therefore, the quality and reliability of the resulting vital sign information are heavily influenced by the quality of the input image data, which depends on image contrast and the appropriate selection of the region of interest. Furthermore, the non-contact nature of these measurements results in increased sensitivity to subject movement. Respiratory rate, in particular, is notoriously difficult to measure when the subject is moving, despite its importance and the abundance of available monitoring techniques. During movement, remote monitoring techniques, such as those based on radar signals, are typically overwhelmed by artifacts and rendered ineffective.

[0008] In non-contact vital sign monitoring, particularly respiratory monitoring, metrics can be used to determine when disturbances such as movement interfere with accurate measurement. These can be used to suppress measurements to avoid giving false outputs. In fact, various methods exist for filtering disturbances from the measurement signal. However, it is preferable to increase the likelihood of a correct measurement rather than suppressing false outputs. For this reason, subjects may be instructed to remain "still," but this is ambiguous and can easily be forgotten, especially if the subject is experiencing distress (e.g., in an emergency room setting). [Means for solving the problem]

[0009] The objective of the present invention is to provide a device for supporting improved respiratory monitoring, particularly a device for supporting higher-quality respiratory monitoring of a subject. Therefore, the objective of the present invention is to support accurate respiratory monitoring.

[0010] In a first aspect of the present invention, a device for supporting improved respiratory monitoring is presented, which includes: An input unit configured to acquire a set of image frames of the region of interest, A processing unit configured to analyze a set of image frames and estimate the momentum of an object in a first direction and a second direction, wherein the second direction is substantially perpendicular to the first direction, An evaluation unit configured to compare momentum in a first direction with momentum in a second direction and to determine the quality of respiratory monitoring based on the above comparison, It includes an output unit configured to output a quality signal indicating the quality of respiratory monitoring.

[0011] In a second aspect of the present invention, a system for supporting improved respiratory monitoring is presented, which system An imaging device configured to capture a set of image frames of a region of interest, A device to support improved respiratory monitoring, It has a user interface configured to provide the user with feedback on the quality of respiratory monitoring based on quality signals.

[0012] In a further aspect of the present invention, a computer program is provided which, when executed on a computer, includes program code means for causing a computer to perform the steps of the method disclosed herein, and a non-transient computer-readable recording medium which stores therein the computer program which, when executed on a processor, causes the method disclosed herein to be performed.

[0013] Preferred embodiments of the present invention are provided in the dependent claims. It should be understood that the claimed systems, methods, and computer programs have similar and / or identical preferred embodiments to those of the claimed apparatus, as provided in the dependent claims and disclosed herein.

[0014] This invention is based on the idea of ​​providing support to subjects (patients) and clinical staff in respiratory monitoring. In particular, it provides feedback on the quality of respiratory monitoring to enable better monitoring quality. In fact, the idea of ​​this invention is to provide guidance so that the subject (or other user) is in the optimal position, orientation, posture, and / or movement (static) so that respiratory monitoring is performed in the optimal manner.

[0015] To this end, the subject is monitored by an imaging device, such as a video camera. The imaging device is directed particularly towards the subject's region of interest, preferably a region where respiratory motion can be detected / measured. The recorded image frames are provided to a device that supports improved respiratory monitoring. The device's input unit is configured to acquire at least two image frames. The image frames do not need to be (directly) consecutive, but may be image frames from (arbitrary) different points in time. The processing unit then analyzes the image frames. In particular, the processing unit is configured to use the image frames to estimate the motion (of the subject depicted by the image frames) particularly in the region of interest. For example, the processing unit may be configured to detect motion in / with respect to at least multiple pixels (voxels) and / or groups of pixels (voxels) in the region of interest for at least two image frames. A standard motion estimation algorithm can be applied for this purpose. The processing unit can further set a first (reference) direction and a second (reference) direction substantially perpendicular to the first (reference) direction (in the image plane of the image set), and can project the detected motion into the first and second directions to estimate the momentum (of the object) in the first and second directions. Alternatively, the detected motion may be filtered with respect to the first and second directions. The momentum in the first and / or second directions can be given by motion signals (movement signals) corresponding to the first and / or second directions, respectively.

[0016] In the evaluation unit, the momentum in the two directions (anterior-posterior direction) is compared with each other. In other words, the two quantities are related to each other, and the relationship (comparison) of the quantities is, for example, the difference or the ratio of the quantities. Based on the result of the comparison (relationship), the evaluation unit determines the quality of the respiratory monitoring, in particular the value of the respiratory monitoring quality metric. In other words, the processing unit is configured to convert the result of the comparison into a feedback metric. For example, the result of the comparison can be linked to a predetermined quality (metric value), such as one stored in a lookup table. Based on the quality (in particular the quality metric value), the output unit provides a quality signal indicating the quality (metric value) of the respiratory monitoring.

[0017] The user interface may use quality signals to provide feedback on the quality of respiratory monitoring. The feedback is preferably given in (near) real-time, i.e., during respiratory monitoring. The user interface may have a graphical user interface, such as a display or touchscreen, which can display the feedback as a (target) value and / or range. For example, the feedback may be provided as a percentage, with 100 percent representing the best possible quality metric value. In other words, the feedback may be provided in the form of a quality index. Similarly, the user interface may have visual indicators in the form of one or more light sources, where the light sources can emit light in a predetermined color and / or a predetermined (time) pattern based on the quality signal. In other words, the feedback may take the form of other non-numerical indicators, such as "traffic lights," bullseyes (indicators), or audio cues. However, the user interface is not limited to these examples.

[0018] The feedback may be based on the object's position, orientation, posture, and / or movement (relative to the imaging device). Information regarding the object's position, orientation, posture, and / or movement can be obtained from the set of images provided by the imaging device itself or from other sources.

[0019] Providing feedback to subjects during respiratory monitoring allows them to receive information about the quality of the monitoring and potentially receive direct advice on how to improve it. This is due to the fact that the measurement accuracy of non-contact respiratory monitoring is particularly improved when the subject is still. Furthermore, providing feedback allows the subject to concentrate on something, which helps prevent distractions that can occur from remaining still (i.e., distractions that may cause unwanted movements, such as movements that interfere with respiratory monitoring).

[0020] It should be noted that the imaging device of the system may or may not be the same as the device used for non-contact respiratory rate monitoring. The imaging device of the system according to the present invention is primarily used to capture the movement of the subject, particularly the movement associated with the subject's respiration (inspiration and / or expiration). However, this does not necessarily mean that the images captured by the above device are further used to obtain information about respiratory rate, respiratory velocity, or other vital signs related to respiration. In fact, the imaging device of the system may be used to improve respiratory monitoring, and a (separate) radar device or camera may be used to perform respiration-related measurements.

[0021] In the context of the present application as disclosed herein, the term “patient” is not limited to users suffering from a disease, but may be used in general to refer to any user or subject.

[0022] In embodiments of a device for supporting improved respiratory monitoring, the processing unit is configured to estimate the momentum of an object by setting a motion vector reference point in one image frame and determining a motion vector based on the displacement of the vector reference point in another image frame. In particular, the processing unit is configured to estimate the momentum of an object in a first direction by setting a first motion vector reference point in one image frame and determining a first motion vector based on the displacement (i.e., translation) of the first vector reference point in a first direction in another image frame. Similarly, the processing unit can estimate the momentum of an object in a second direction by setting a second motion vector reference point in one image frame and determining a second motion vector based on the displacement of the second vector reference point in a second direction in the other image frame. Preferably, the first and second motion vector reference points are the same. Furthermore, it is preferable that the images in which the first and second motion reference points are set are the same, and that the images used to determine the displacement of the points are the same. This ensures that momentum in two directions is detected simultaneously at the same location. Motion vectors are also called translation vectors or displacement vectors. Generally, a processing unit can be configured to estimate the momentum of an object by determining a series of motion vectors based on a series of displacements of a vector reference point in a series of image frames. In this way, forward and backward motion, particularly typical of respiration, can be detected / captured.

[0023] In general, any available algorithm can be used to determine the first and second motion vectors in an image frame. The respiratory effort can be tracked, for example, using motion detection from optical flow. Other examples of such algorithms include ProCor, kernel tracking, and face tracking. In fact, some algorithms do not (directly) determine the displacement of the reference vector points. However, such displacement can be determined in any of these methods, for example, by determining the difference in positions detected in two separate images. However, in this application, these positions are considered motion vector reference points.

[0024] In another embodiment, the momentum is the absolute value of the motion integrated over time. Thus, the momentum can be given by the modulus (or series of moduli) of the motion vector, and the modulus (series of moduli) is integrated over time, particularly over the time between the first and last image frames considered (i.e., between the image frames to which the motion vectors correspond). More precisely, the momentum (or change in motion) in the first (second) direction is given by the modulus of the first (second) motion vector, and this modulus is integrated over the time between the image frames to which the first (second) motion vector corresponds.

[0025] However, the momentum can be given by only the absolute value of the motion, particularly the modulus of the motion vector (i.e., without time integration).

[0026] In general, in order to evaluate the quality of respiratory measurements, integration over time may be omitted. The respiratory signal is usually rather a (proxy for) position than speed. In other words, the measurement signal related to respiratory monitoring is usually given in units of position. However, in camera-based measurements, the speed (or displacement of the motion vector reference point between frames) is generally easier to measure accurately than the absolute position. Therefore, in camera-based respiratory measurements, there is usually an integration step to move from speed to position, where the (position) signal is used for respiratory peak detection and / or rate calculation. Thus, by including the integration step, the quality of the final (position) signal determined can represent the actual quality in a more reliable manner.

[0027] In another embodiment, the evaluation unit is configured to calculate the ratio of the momentum in the first direction to the momentum in the second direction and determine the quality of respiratory monitoring based on the ratio. For example, the evaluation unit may be configured to calculate the ratio of the time integral value of the absolute value of the motion in the first direction to the time integral value of the absolute value of the motion in the second direction. More specifically, the evaluation unit may calculate the ratio of the time integral value of the modulus of the motion vector in the first direction to the time integral value of the modulus of the motion vector in the second direction. In other words, the evaluation unit may calculate the ratio of the time integral value of the modulus of the movement signal corresponding to the first direction to the time integral value of the modulus of the movement signal corresponding to the second direction.

[0028] If the value of the above ratio exceeds or is the same as a threshold value (the value is, for example, 1), the respiratory signal based on the momentum in the first direction, or more generally, the respiratory signal linked to the first direction (especially the movement of the object in the first direction) can be considered reliable. The above threshold value can depend on the selected type of respiratory monitoring and can thus be set based on the type of respiratory monitoring.

[0029] In general, any directional amplitude or other indicator of the momentum of an object (e.g., an indicator generated in an intermediate step that determines the motion vector) can be considered a motion vector. Thus, the motion of an object can also be represented by a series of directional amplitudes. Accordingly, the evaluation unit may be configured to compare the momentum in a first direction with the momentum in a second direction by determining the ratio of the directional amplitude in a first direction to the directional amplitude in a second direction, in particular by using the ratio of the moduli of the above amplitudes, the moduli of which is integrated over time (similar to the motion vector).

[0030] Motion vectors / directional amplitudes are not limited to images from a 2D camera, but can also be applied to (and determined from) image frames captured by a 3D and / or thermal camera or a high-frequency sensor.

[0031] In one embodiment of the apparatus, the first direction is substantially parallel to the height direction of the object, and the second direction is substantially parallel to the width direction of the object (or vice versa). In particular, the processing unit may be configured to detect the height and width directions of the object (e.g., from an image frame) and compare these directions with the first and second directions, respectively. Accordingly, the processing unit may be configured to detect the deviation between the first direction and the height direction, and the deviation between the second direction and the width direction, respectively. Furthermore, the processing unit may be configured to determine how the first and / or second directions need to be changed, respectively, to coincide with the height direction and / or width direction, and to change / adapt these directions accordingly.

[0032] If the first direction is (substantially) the same as the height direction, and the second direction is (substantially) the same as the height direction, then respiratory movement (particularly chest and abdominal movement) is expected to be (substantially) limited to the second direction. Therefore, if the evaluation unit determines (as a result of the comparison) that the amount of movement in the second direction is much greater than the amount of movement in the first direction, then the quality of respiratory monitoring can be determined to be high. In fact, based on the comparison of both amounts of movement, the processing unit can determine a quality index, and the higher the amount of movement in the second direction compared to the first direction, the higher the quality index.

[0033] In yet another embodiment, the input unit, processing unit, evaluation unit, and output unit are configured to operate in real time. Therefore, image frame acquisition and quality signal provision occur quasi-instantaneously. In other words, the device is configured to perform real-time processing. If the device is part of a system with a user interface, this enables real-time feedback to the user regarding respiratory monitoring quality.

[0034] In further embodiments, the processing unit is configured to generate a respiration signal from a set of image frames, and the processing unit is configured to filter the respiration signal. The respiration signal may indicate the number of breaths of an object over a given time. In practice, the respiration signal may also be a signal indicating momentum in a first and / or second direction (and accordingly, a signal indicating the number of breaths within a given time). Thus, the respiration signal is also called a motion signal, and the motion signal can be subject to filtering, in particular frequency filtering.

[0035] In particular, the processing unit may be configured to estimate the number of respirations of an object in a time span between two sets of image frames by using, for example, a motion vector sequence, the estimated momentum of the object in a first direction and / or a second direction (in the above time span). However, the respiration signal may also be determined from the set of image frames in other ways. The respiration signal can be further filtered. For example, the processing unit may be configured to filter the respiration signal with respect to a predetermined frequency band, i.e., to filter out (and / or cut off) a predetermined frequency. For this purpose, a bandpass filter that allows frequencies in the expected range of vital signs, i.e., 5 to 60 respirations / min (for adults), can be used. Thus, the momentum (corresponding signals) in the first and / or second directions can be filtered.

[0036] Therefore, in order to more accurately determine the quality of respiratory monitoring, the evaluation unit may be configured to compare the momentum in a filtered first direction with the momentum in a (filtered) second direction and determine the quality of respiratory monitoring based on this comparison.

[0037] In another embodiment, the evaluation unit is configured to compare momentum in a first direction with a first threshold and / or momentum in a second direction with a second threshold, and the evaluation unit is configured to determine a gain coefficient based on either of the above comparisons. In particular, the evaluation unit may be configured to compare momentum in a first direction with a first threshold and determine a first gain coefficient based on this comparison. Similarly, the evaluation unit may be configured to compare momentum in a second direction with a second threshold and determine a second gain coefficient based on this comparison. In particular, it can determine whether and to what extent each momentum fits the threshold. Then, the corresponding gain coefficient can be determined. For example, if momentum in a first direction fits the first threshold, in particular if it is less than the first threshold, the gain coefficient can be set to a high value (low value). On the other hand, if momentum in a first direction does not fit the first threshold, in particular if it is the same as or greater than the first threshold, the gain coefficient can be set to a low value (high value). Generally, the gain coefficient can be any number, and is preferably normalized to the interval [0,1].

[0038] In particular, the evaluation unit may be configured to determine the quality of respiratory monitoring based on one of the gain coefficients described above. Indeed, as mentioned above, the gain coefficient represents a measure of movement. Generally, the greater the movement (the more detectable) in the direction of movement of the subject's chest during respiration, the better the quality of respiratory monitoring is expected to be. However, very large movements may not originate from respiratory movement but from other movements of the subject. Therefore, very large amounts of movement are likely to interfere with accurate measurement. In other words, very large detected / measured movement is an indicator of poor quality of respiratory monitoring (regardless of the comparison of movement). Therefore, it is advantageous to apply the gain coefficient to the determined quality (and thus the quality signal and feedback provided by the system).

[0039] In one embodiment of a system supporting improved respiratory monitoring, the imaging device includes any of the following: a camera, in particular a video camera; a radar sensor, in particular a Doppler radar sensor; an ultrasonic imaging device; a thermal camera, in particular a microbolometer camera; a laser Doppler imaging device; a speckle vibration measurement unit; and a depth camera, in particular a time-of-flight camera.

[0040] For example, if the subject's face is visible, a thermal infrared camera can detect temperature differences and changes in front of the nose / mouth. Depth cameras enable the acquisition of 3D images. Thus, time-of-flight (ToF) cameras enable depth sensing. However, depth reconstruction can also be achieved by using multiple (video) cameras (stereo principle). Regardless of the type of imaging device described above, respiratory measurements using any of these devices benefit from limited body movement and the assistance provided by devices that support improved respiratory monitoring.

[0041] In another embodiment of the system, the feedback provided by the user interface further includes advice to improve the quality of respiratory monitoring, relating to the position, orientation, posture, and / or movement of the subject. The advice may include signal notifications to the subject to remain still or to move to a position more easily visible to the system's imaging device. In particular, the advice includes instructions to the subject to change its position, orientation, and / or posture such that a first direction is substantially parallel to the subject's height direction and a second direction is substantially parallel to the subject's width direction (or vice versa). Thus, the device may have instructions to the subject to rotate 90°, change orientation, stand upright, or lie down, etc.

[0042] To provide the above feedback, an improved respiratory monitoring device, particularly the processing unit of such device, may be configured to detect the height and width directions of an object and compare these directions to a first and second direction, respectively. Thus, the processing unit may be configured to detect the deviation between the first direction and the height direction, and the deviation between the second direction and the width direction, respectively. Furthermore, the processing unit may be configured to determine how the first and / or second directions need to be modified, respectively, to match the height and / or width directions, and to adapt the directions accordingly. Similarly, the processing unit may be configured to determine how the height and / or width directions of an object need to be modified, respectively, to match the first and / or second directions. Based on this, the processing unit may be configured to provide signals that instruct the object on how to change its position, orientation, posture, and / or movement so that the first direction is the same as the height direction, and / or the second direction is the same as the width direction. Based on these signals, the system's user interface can provide the above advice.

[0043] In yet another embodiment of the system, the user interface is configured to provide real-time feedback, and / or the user interface is configured to provide feedback as visual, auditory, and / or haptic feedback.

[0044] Because the device supporting improved respiratory monitoring can be configured to operate in real time, quality signals can be provided in real time. Therefore, the user interface of this system can provide real-time feedback on the quality of respiratory monitoring. Thus, healthcare professionals and patients can immediately obtain feedback on the quality of ongoing respiratory monitoring and intervene immediately to improve the monitoring. This enables faster, more reliable, and more accurate clinical diagnoses.

[0045] System feedback can be provided in various forms and thus can be adapted to the subject's health condition and the chosen monitoring method. For example, if the subject is visually impaired or if clinical staff are unable to supervise the monitor due to other tasks, the user interface may have a speaker to provide audio feedback regarding the quality of respiratory monitoring. Similarly, the user interface may have a screen showing the optimal monitoring position and the subject's actual position. Thus, the subject can see if their position is optimal and how to move to get into the optimal position for respiratory monitoring. As another example, during respiratory monitoring, the subject may stand on a vibrating platform, where the platform vibrates if the subject's position, orientation, posture and / or movement does not allow for high-quality monitoring.

[0046] Preferably, feedback is provided continuously. However, feedback can also be provided at (time) intervals, particularly at regular (time) intervals. [Brief explanation of the drawing]

[0047] [Figure 1] This figure shows a schematic diagram of one embodiment of the apparatus according to the present invention. [Figure 2] This figure shows a subject (patient) 50 during respiration and the orthogonal reference direction according to the present invention. [Figure 3] This figure shows a flowchart of a first embodiment of the method according to the present invention. [Figure 4] This figure shows a flowchart of a step embodiment of the first embodiment of the method according to the present invention. [Figure 5] This figure shows a diagram of the first set of motion signals representing the momentum in the horizontal and vertical directions, and their comparison. [Figure 6] This figure shows a diagram of a second set of motion signals representing the momentum in the horizontal and vertical directions, along with their comparison. [Figure 7]This diagram shows a third set of movement signals, each representing the amount of movement in the horizontal and vertical directions, and their comparison. [Figure 8] This figure shows a schematic diagram of one embodiment of the system according to the present invention. [Figure 9] This figure shows various diagrams illustrating the feedback regarding the quality of respiratory monitoring provided by the system according to the present invention. [Figure 10] This figure shows a diagram illustrating advice for supporting respiratory monitoring performed by System 1 of the present invention. [Modes for carrying out the invention]

[0048] These and other aspects of the present invention will become apparent from the embodiments described below and will be explained with reference to those embodiments.

[0049] Figure 1 shows a schematic diagram of one embodiment of the apparatus 10 according to the present invention. The apparatus 10 has an input unit 14, a processing unit 16, an evaluation unit 20, and an output unit 24. The apparatus 10 may have circuits that perform the functions of the apparatus, such as a processor, processing circuit, computer, dedicated hardware, etc. In another embodiment, separate units or elements that together represent the circuits may be used.

[0050] In this embodiment, the input unit 14 is configured to acquire a set of image frames 12, in particular a stream of images of the region of interest of the subject 50. The set of image frames 12 can be acquired, for example, from an imaging device or storage device. For this purpose, the input unit 14 can be coupled or connected directly or indirectly (e.g., via a network or bus) to the imaging device or storage device. The input unit 14 can be a communication interface or data interface (wired or wireless), such as a Bluetooth interface, Wi-Fi interface, LAN interface, HDMI interface, direct cable connection, or any other suitable interface that enables data transfer to the device 10. The acquired set of image frames 12 may in particular show the chest of the subject 50. The set of image frames 12 can be provided to the processing unit 16.

[0051] The processing unit 16 may use part or all of the region of interest to determine the motion of an object between at least two (not necessarily consecutive) image frames 12. In particular, the processing unit 16 may analyze the image frames 12 (particularly in the chest region) to estimate the momentum of the object's chest in two directions that are substantially orthogonal to each other. Specifically, the processing unit 16 estimates the momentum of the chest in a first direction and in a direction (second direction) orthogonal to the first direction. The momentum can be provided by the distance the chest moves in the first and second directions, respectively. Motion data 18 with momentum estimates in both directions (e.g., a motion signal indicating motion in the first direction and a motion signal indicating motion in the second direction) is provided to the evaluation unit 20, which compares the momentum in both directions with each other. In other words, the evaluation unit 20 compares the distance the chest moved in the first direction with the distance the chest moved in the second direction. Based on the above motion comparison, the evaluation unit 20 determines the quality metric 22 for monitoring the object. Both the processing unit 16 and the evaluation unit 20 may be any type of means configured to process the image frame 12, estimate the momentum of the object in a first and second direction using the frame, and determine the quality of respiratory monitoring by comparing the momentum. The processing unit 16 and the evaluation unit 20 can be implemented in software and / or hardware, such as a programmed processor, computer, laptop, PC, or workstation.

[0052] Using the respiratory monitoring quality / quality metric 22 determined by the evaluation unit 20, the output unit 24 provides a quality signal 26. The output unit 24 may generally be any interface that provides the generated signal, for example, to transmit it to another device, or to provide it for acquisition by another device, for example, to transmit it to another computer, or to transfer it directly to a user interface 40 for display. It may therefore generally be any (wired or wireless) communication or data interface.

[0053] Figure 2 shows an object 50 during respiration and a reference direction perpendicular to it. The solid line represents the object in an exhaling state. The dashed line represents the object in an inhaling state, i.e., the lungs are expanded, the pleural cavity is enlarged, and therefore the rib cage is protruding outward. In fact, respiratory movement (due to inspiration / expiration) is expected to be observed primarily as lateral movement when the object is sitting or standing. This is indicated by the double arrow in Figure 2, which is parallel to the x-axis. On the other hand, when the object inhales or exhales, there is expected to be little to no movement in the y-direction. Therefore, a metric for respiratory movement, i.e., movement of the object's body (particularly the chest and abdomen), can be obtained by dividing the integral of the absolute vertical displacement (or "displacement distance," i.e., displacement in the y-direction) measured within a certain displacement window by the integral of the horizontal displacement (x-direction). If this ratio is significantly greater than 1, it indicates that the vertical movement is stronger than the horizontal movement. In this case, the detected movement may not be due to breathing, but rather to other movements of the subject. In other words, breathing monitoring should not be trusted if this ratio exceeds 1. The effectiveness of the above ratio is strongest when the x and y axes are selected so that they are (substantially) parallel to the width and height directions of the subject, where the width direction of the subject can be defined, for example, by a straight line connecting the left and right shoulders of the subject. Similarly, the height direction of the subject can be defined by a straight line indicating the distance from the head to the toes.

[0054] Generally, another set of directions (preferably orthogonal) can be selected as directions that are neither parallel nor perpendicular to the height of the object, such as directions x' and y' as shown in Figure 2.

[0055] Figure 3 shows a flowchart of a first embodiment of Method 100 according to the present invention. The steps of the method can be performed by apparatus 10, where the main steps of the method are performed by processing unit 16 and evaluation unit 20. The method can be implemented, for example, as a computer program executed on a computer or processor.

[0056] In the first step 102 of method 100, a set of image frames 12 of the region of interest is acquired (received or retrieved). The image frames 12 can be acquired directly from a camera or from a storage device such as a buffer, memory, or image repository, for example, from records of the subject stored in a hospital document management system, for example, via a wired or wireless network.

[0057] In the second steps 104 and 104', the image frames are analyzed with respect to the motion of an object in the region of interest in two different directions. Specifically, in step 104, the momentum of the object in the first direction is estimated, and in step 104', the momentum of the object in the second direction is estimated, where the second direction is substantially perpendicular to the first direction. For this purpose, any commonly available algorithm can be used. Examples of motion estimation algorithms include ProCor, optical flow, kernel tracking, and face tracking. Using these algorithms, motion vectors in the first and second directions can be determined, respectively. In this embodiment of method 100, steps 104 and 104' are performed in parallel. However, the above steps can also be performed in a sequential order, with step 104 followed by step 104', or vice versa.

[0058] Once the momentum in the first and second directions is estimated, both momentums are compared in step 106. In step 108, a respiratory monitoring quality metric is determined based on the comparison in step 106. In step 110, a quality signal is generated based on the quality metric and provided for further processing.

[0059] Figure 4 shows a flowchart of an embodiment of step 104 of the first embodiment of Method 100. In this embodiment, step 104 is divided into three substeps 1042, 1044, and 1046. As described above, in step 104, the momentum of the object in the first (reference) direction is estimated. In particular, in the first substep 1042, a motion vector reference point is set in one of the image frames. The motion vector is determined based on the displacement / translation of the above vector reference point in another (consecutive) image frame. In substep 1044, the absolute value of the motion vector is determined. In other words, the modulus of the motion vector is determined. Then, the modulus of the motion vector is integrated over time.

[0060] Figures 5, 6, and 7 show sets of figures illustrating motion signals and their comparisons, respectively, indicating horizontal and vertical displacement. In particular, the figures show the momentum of a portion of the object, which is represented, for example, by a motion vector reference point. Signals indicating momentum in different directions are derived from time series of horizontal and vertical motion vectors (i.e., from a set of multiple image frames). In the upper left figure (of each figure), the horizontal movement of (a portion of) the object is represented by the horizontal motion signal. The upper center figure shows the absolute value of the horizontal motion signal. The lower left and lower center figures show the vertical motion signal and its absolute value, respectively. As can be seen from the figures in Figure 5, especially the center figure, the amplitude of the vertical motion signal is considerably larger than that of the horizontal motion signal. This means that the detected motion of the object progresses more vertically than horizontally. In a given case, the quality metric for determining the quality of respiratory monitoring of the object is given by the ratio of the time integral of the absolute vertical motion to the time integral of the absolute horizontal motion. This ratio, or quality metric, is shown in the figure on the right. As can be seen from this figure, the value of the quality metric is significantly greater than 1. In this case, it can be inferred that there is a high probability that respiration can be obtained (in a reliable manner) from the vertical signal.

[0061] Figure 6 shows the horizontal movement of (part of) the object, i.e., the (horizontal) movement signal, in the upper left figure. The absolute value of the horizontal movement signal is shown in the upper center figure. The lower left and lower center figures show the vertical movement signal and its absolute value, respectively. As can be seen, the horizontal movement signal is significantly different from the vertical movement signal. However, the amplitudes of the above signals are similar. On the right, a figure is shown showing the ratio of the time integral of the absolute amount of vertical movement to the time integral of the absolute amount of horizontal movement, and this ratio represents the quality (metric) of the respiratory monitoring. The metric is shown to be greater than 1. Therefore, there is a reasonable possibility that a reliable determination of respiratory features can be obtained from the vertical signal.

[0062] Figure 7 shows the horizontal movement of (part of) the object, i.e., the (horizontal) movement signal, in the upper left figure. The absolute value of the horizontal movement signal is shown in the upper center figure. The lower left and lower center figures show the vertical movement signal and its absolute value, respectively. According to the measured signals shown in the figures, the (absolute) horizontal movement signal dominates the (absolute) vertical movement signal. The quality metric, given by the ratio of the time integral of the absolute vertical movement to the time integral of the absolute horizontal movement, is less than 1. Therefore, it is unlikely that respiration monitoring can be reliably obtained from the vertical movement signal.

[0063] Figure 8 shows a schematic diagram of a first embodiment of System 1 according to the present invention. In this embodiment, System 1 includes an imaging device 30, a device 10 that supports improved respiratory monitoring, and a user interface 40.

[0064] The imaging device 30 is configured to perform respiratory monitoring of a subject and, in this embodiment, has a video camera. However, generally, the imaging device 30 may be a radar sensor, an ultrasonic imaging device, a thermal camera, a laser Doppler imaging device, a speckle vibration measurement unit, or a time-of-flight camera, as conventionally used to acquire image data of a subject, for example, image data of a specific body region of the subject. The imaging device 30 may be implemented, for example, in software and / or hardware, as a processor or computer, or in a processor or computer. For example, a programmed processor may be included in the imaging device 30, which can execute computer programs that can be stored in memory accessed by the processor.

[0065] In this embodiment, a video camera is directed towards the chest of a subject to capture an image stream having a set of target image frames 12 over a predetermined period of time. The imaging device 30 provides the image frames 12 to the device 10, which analyzes the image frames 12 and provides a quality signal 26 indicating the quality of the respiratory monitoring of the imaging device 30.

[0066] In this embodiment, the user interface 40, which is a touchscreen, then provides feedback on the quality of respiratory monitoring based on the quality signal. In particular, the touchscreen is configured to display the quality, for example, with a quality index. Alternatively or additionally, feedback on the quality of the respiratory signal may be provided by the user interface 40 in a color-coded manner.

[0067] Figure 9 shows various diagrams illustrating feedback regarding the quality of respiratory monitoring provided by System 1 according to the present invention. In particular, the diagram shows a quality index q over a respiratory monitoring time t. The quality index q indicates the quality of respiratory monitoring by the imaging device 30 of System 1. In the diagram on the left, the quality index is quite low and is located only in the dark gray shaded area. This means that the quality of respiratory monitoring is low and therefore respiration is being monitored in an unreliable manner. In one embodiment, to more favorably visualize the meaning of the displayed quality index, the darkly shaded area can be shaded in red and the lightly shaded area in green.

[0068] In the central and right-hand figures, the quality index q can be considered to represent reliable respiratory monitoring because the index is clearly in the faint shadow region, meaning the quality index exceeds a predetermined threshold (indicated by the transition from the dark shadow region to the faint shadow region). However, since the value of q is generally higher in the central figure than in the right-hand figure, the quality of respiratory monitoring corresponding to the central figure can be considered higher than the quality of respiratory monitoring corresponding to the right-hand figure.

[0069] Figure 10 illustrates advice to support respiratory monitoring performed by System 1 according to the present invention. In particular, Figure 10 shows the screen of the system's user interface 40 in three different situations. The left screen shows a position template (dashed line) corresponding to the optimal position (and orientation) of the subject's head and shoulders, which enables optimal respiratory monitoring. Thus, the above screen of the user interface can be considered a guide or advisory means for where the subject should position itself. Furthermore, as depicted in the center screen, the user interface can provide further feedback regarding the subject's actual position (solid line). In other words, the feedback screen can display a camera view of the subject (abstracted by a solid line) with the contour of the optimal subject position overlaid. As can be seen from the center screen, the subject's actual position does not coincide with the optimal position indicated by the template. However, from the display of both positions, the subject is advised on how to move to take the optimal position. The right screen shows the subject's position and the position template in a matching state. Thus, the right screen shows the position in which the subject is in the optimal position for respiratory monitoring.

[0070] Although the present invention has been illustrated and described in detail in the drawings and description, such illustrations and descriptions are descriptive or illustrative and not limiting. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and practiced by those skilled in the art who practice the claimed invention from the drawings, description and appended claims.

[0071] In the claims, the word “having” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude plurality. One element or other unit may perform the functions of multiple items described in the claims. The mere fact that a means is described in different dependent claims does not imply that a combination of these means cannot be used advantageously.

[0072] Any reference numerals used in the claims should not be construed as limiting the scope of the invention.

Claims

1. A device that supports improved respiratory monitoring, An input unit that acquires a set of image frames of the target region of interest, A processing unit that analyzes the set of image frames to estimate the momentum of the object in a first direction and a second direction, wherein the second direction is substantially perpendicular to the first direction. An evaluation unit that compares the momentum in the first direction with the momentum in the second direction and determines the quality of respiratory monitoring based on the comparison, A device having an output unit that provides a quality signal indicating the quality of the respiratory monitoring.

2. The apparatus according to claim 1, wherein the processing unit estimates the momentum of the object by setting a motion vector reference point in one of the image frames and determining a motion vector based on the displacement of the vector reference point in another image frame.

3. The apparatus according to claim 1 or 2, wherein the momentum is obtained by integrating the absolute value of the momentum over time.

4. The apparatus according to any one of claims 1 to 3, wherein the evaluation unit calculates the ratio of momentum in the first direction to momentum in the second direction, and determines the quality of the respiratory monitoring based on the ratio.

5. The apparatus according to any one of claims 1 to 4, wherein the first direction is substantially parallel to the height direction of the object, and the second direction is substantially parallel to the width direction of the object.

6. The apparatus according to any one of claims 1 to 5, wherein the input unit, the processing unit, the evaluation unit, and the output unit operate in real time.

7. The apparatus according to any one of claims 1 to 6, wherein the processing unit generates a respiration signal from the set of image frames, and the processing unit filters the respiration signal.

8. The evaluation unit further compares the momentum in the first direction with a first threshold and / or the momentum in the second direction with a second threshold. The apparatus according to any one of claims 1 to 7, wherein the evaluation unit determines the gain coefficient based on any of the comparisons.

9. The apparatus according to claim 8, wherein the evaluation unit determines the quality of the respiratory monitoring based on the gain coefficient.

10. A system that supports improved respiratory monitoring, An imaging device that captures a set of image frames of the region of interest, The apparatus according to any one of claims 1 to 9, A system having a user interface that provides the user with feedback on the quality of the respiratory monitoring based on quality signals.

11. The system according to claim 10, wherein the imaging device comprises any of the following: a camera, in particular a video camera; a radar sensor, in particular a Doppler radar sensor; an ultrasonic imaging device; a thermal camera, in particular a microbolometer camera; a laser Doppler imaging device; a speckle vibration measurement unit; and a depth camera, in particular a time-of-flight camera.

12. The system according to claim 10 or 11, wherein the feedback further includes advice for improving the quality of the respiratory monitoring, the advice relating to the position, orientation, posture and / or movement of the subject.

13. The user interface provides real-time feedback and / or, The system according to any one of claims 10 to 12, wherein the user interface provides the feedback as visual, auditory, and / or tactile feedback.

14. In methods to support improved respiratory monitoring, The steps include obtaining a set of image frames of the region of interest, A step of analyzing the set of image frames to estimate the momentum of the object in a first direction and a second direction, wherein the second direction is substantially perpendicular to the first direction; A step of comparing the momentum in the first direction with the momentum in the second direction and determining the quality of respiratory monitoring based on the comparison, A method comprising the step of providing a quality signal indicating the quality of the respiratory monitoring.

15. A computer program having program code means that, when executed on a computer, causes the computer to perform the steps of the method according to claim 14.

Images