Devices, systems, and methods for determining respiratory information of a subject

Abstract

The present invention relates to a device, system and method for determining respiratory information of a subject, the device comprising an image input for acquiring a thermal image of the subject, a processing unit for determining respiratory information of the subject from the thermal image and an output for outputting the determined respiratory information of the subject, the processing unit determines the respiratory information of the subject by identifying in the acquired thermal image respiratory pixels and / or pixel groups indicative of temperature variations related to respiration, determining from among the respiratory pixels and / or pixel groups respiratory flow pixels and / or pixel groups indicative of respiratory flow, where one or more spatial filters are applied to the respiratory pixels and / or pixel groups to distinguish the respiratory flow pixels and / or pixel groups from respiratory motion pixels and / or pixel groups indicative of respiratory motion, and determining the respiratory information of the subject from the respiratory flow pixels and / or pixel groups and / or the respiratory motion pixels and / or pixel groups.

Description

[Technical Field]

[0001] The present invention relates to a device, system, and method for determining respiratory information of a subject, particularly a human or an animal. [Background technology]

[0002] Respiratory rate (RR) is one of the most important vital signs, and a decrease in it may indicate that immediate action is needed. While the movement of the subject usually indicates breathing, respiratory movement can persist in cases of airway obstruction, such as in sleep apnea, so the most informative measurement is airflow. Non-invasive measurement of these apneic events during sleep is currently an unmet need.

[0003] Apnea is defined as a cessation of breathing lasting at least 20 seconds, or, in infants, at least 10 seconds if accompanied by bradycardia and / or desaturation, the latter commonly referred to as apnea of ​​prematurity. In adults, the term sleep-disordered breathing is generally used to include the occurrence of apnea and hypopnea.

[0004] In the Neonatal Intensive Care Unit (NICU), the standard method for monitoring respiration is chest impedance or impedance pneumography. The typical standard method for adults is polysomnography (PSG), which requires the use of several sensors to accurately monitor the patient's vital signs. During PSG, both respiratory flow and respiratory movement are monitored using thermistors near the nostrils and / or oral cavity, and two belts on the chest and abdomen to monitor respiratory trends.

[0005] The need to monitor both respiratory flow and respiratory movement arises from differences in specific types of apnea. Apnea can generally be classified into central apnea, obstructive apnea, and mixed apnea in both infants and adults. In central apnea, breathing is completely absent, and this is evident in both respiratory flow and respiratory movement. Obstructive apnea is a type of apnea in which there is still a noticeable effort to breathe during the apnea, but there is no flow at all. Mixed apnea is a combination of the other two types of apnea. To detect and classify the type of apnea, it is necessary to monitor both respiratory flow and respiratory movement.

[0006] Currently available methods and techniques for monitoring respiration are, however, invasive, can cause discomfort and skin irritation, and may disrupt the patient's sleep. Therefore, non-invasive solutions are being considered as alternatives to contact methods. Within the category of non-invasive techniques, the use of thermal cameras is the only method capable of monitoring both respiratory flow and respiratory movement, although respiratory movement can be detected using different imaging techniques in visible light, near-infrared (NIR), and radar.

[0007] Thermal cameras can be an alternative to contact monitoring methods for respiratory flow and respiratory movement. However, to obtain respiratory flow signals that enable the detection of apnea, body landmark detection is generally required. Known solutions distinguish between contributions from thermal respiratory flow and contributions from respiratory movement. These require body part detection using only a thermal camera, or body part detection by combining a thermal camera with another type of camera, such as an RGB or NIR camera. The detection of specific body parts is extremely complex and may be too fragile to rely on in a hospital setting. The presence of blankets or sheets covering the body and certain sleeping positions increase this complexity in adults, and even more so in infants.

[0008] To avoid identifying landmarks, solutions have been proposed for automatically selecting pixels containing respiratory information in the camera. However, these methods, when used with thermal imaging, cannot distinguish between the contribution of respiratory flow and the contribution of respiratory motion. Therefore, the output becomes a mixture of flow and motion signals, making it impossible to accurately identify apnea.

[0009] LORATO ILDE et al., "Multi-camera infrared thermography for infant respiration monitoring," BIOMEDICAL OPTICS EXPRESS, vol.11, no.9, September 1, 2020, disclose an algorithm for merging multiple thermal camera views and automatically detecting pixels containing respiratory motion or respiratory flow using three features. [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The object of the present invention is to provide a device, system, and method for determining a subject's respiratory information in a non-invasive and accurate manner. [Means for solving the problem]

[0011] In a first aspect of the present invention, a device for determining respiratory information of a subject, wherein the device is An image input unit for acquiring a thermal image of a target, wherein the thermal image shows at least a portion of the target's facial area and / or its surrounding area. A processing unit for determining the respiratory information of a subject from thermal images, An output unit for outputting the determined respiratory information of the target and Equipped with, The processing unit, Identifying respiration pixels and / or pixel groups that show temperature fluctuations related to respiration in the acquired thermal image, Determining respiratory flow pixels and / or pixel groups from among respiratory pixels and / or pixel groups, wherein one or more spatial filters are applied to the respiratory pixels and / or pixel groups in order to distinguish them from respiratory motion pixels and / or pixel groups that indicate respiratory motion, Determining the target respiratory information from respiratory flow pixels and / or pixel groups, and / or respiratory motion pixels and / or pixel groups. A device is presented that determines the subject's respiratory information.

[0012] In a further aspect of the present invention, a system for determining the respiratory information of a subject, wherein the system is A thermal sensor unit for collecting a thermal image of a target, wherein the thermal image shows at least a portion of the target's facial area and / or its surrounding area. A device disclosed herein for determining the respiratory information of a subject from thermal images, An output interface for outputting the target's respiratory information and A system is presented that includes the following features.

[0013] Further aspects of the present invention provide a corresponding method, a computer program, a computer program including program code means for causing a computer to perform steps of the method disclosed herein when the computer program is executed on a computer, and a non-temporary computer-readable recording medium storing therein a computer program product that causes the method disclosed herein to be performed when executed by a processor.

[0014] Preferred embodiments of the present invention are defined in the dependent claims. It should be understood that the claimed methods, systems, computer programs, and media have preferred embodiments similar to and / or equivalent to the claimed devices defined in the dependent claims and disclosed herein.

[0015] The present invention enables the non-invasive determination of respiratory information in animals or human subjects. Respiratory information generally includes any information related to respiratory (or breathing) characteristics, such as respiratory flow information, respiratory effort information, obstructive apnea, central apnea, and mixed apnea. For example, respiratory flow, particularly signals indicating the presence of respiratory flow, are determined.

[0016] A 2D thermal sensor array, such as a thermal sensor unit (also called a thermographic camera, infrared camera, thermal imaging camera, or thermal imager), is used for this purpose. The present invention is based on the idea that it is possible to distinguish between motion-induced thermal fluctuations and respiratory flow-induced thermal fluctuations, since motion-induced thermal fluctuations (detected as respiratory motion pixels and / or pixel groups (one pixel group consists of multiple adjacent pixels)) occur on a temperature gradient, and respiratory flow-induced thermal fluctuations (detected as respiratory flow pixels or pixel groups) similarly occur in flat regions of the thermal image. For this distinction of RR frequency fluctuations, spatial filters can be used efficiently.

[0017] While exercise-induced thermal variability (i.e., time-varying values ​​or time signals of respiratory motion pixels and / or pixel groups) can be helpful in determining respiratory rate, substantially eliminating exercise-induced variability in respiratory motion pixels and / or pixel groups allows for the determination of other respiratory information, such as respiratory flow signals. This not only enables non-invasive monitoring of the subject's respiration and respiratory effort, but also allows for the detection of the presence and type of apnea, and respiratory arrest due to apnea.

[0018] It should be noted that the expression "pixel and / or pixel group" should be understood in the context of the present invention as a single pixel (only), or a pixel group (only), or a mixture of pixels and pixel groups. A pixel group should be understood as a plurality of pixels, particularly a plurality of adjacent pixels. In general, a pixel group may even include a complete image, i.e., all the pixels of an image. Thus, for example, the feature that one or more spatial filters are applied to respiratory pixels and / or pixel groups should be understood as including the option that one or more spatial filters are applied to a single pixel, or to a pixel group, or to a mixture of pixels and pixel groups, or to a complete image (i.e., all the pixels of an image).

[0019] In other words, a respiratory pixel group should generally be understood as a group of pixels presenting respiratory information, i.e., those in which the pixel values vary over time in a quasi-periodic manner within a specific frequency range. Spatial filtering implies that the pixels are combined with their spatially adjacent pixels which do not all have to convey respiratory information. Moreover, it may sometimes be easier to spatially filter the entire image and then evaluate the effect on the respiratory signal (temporal variation of the pixels).

[0020] According to a preferred embodiment, the processing unit applies at least two spatial filters, particularly at least two Gabor filters having at least two different spatial filtering directions and / or one or more spatial frequencies, to the respiratory pixels and / or pixel groups. Depending on the situation, two different spatial filters having two different spatial filtering directions and a common spatial frequency are applied. In other embodiments, two or more spatial filters having two or more different spatial frequencies are applied.

[0021] Direction (or orientation) and spatial frequency are common settings for spatial filters such as the Gabor filter. By varying the direction / orientation, it is possible to analyze the variations in the thermal image from different orientations. Assume that the respiratory motion pixels are configured in columns, and thus a set of spatial filters for analyzing the variations in the thermal image from different orientations is applied. It can be expected that a high response value is obtained in the filtering performed in the direction orthogonal to the columns, while the response values for the filters in other directions are lower or even zero. The fact that the respiratory flow pixels are not configured in columns but in a region that appears more circular allows the assumption that the response value for the spatial filter for respiratory flow is independent of the direction of the filter. Therefore, a set of spatial filters is preferably applied, and then the response values are multiplied to automatically identify the respiratory flow.

[0022] In another embodiment, the processing unit identifies the respiratory motion pixels and / or pixel groups by using one or more spatial filters among the respiratory pixels and / or pixel groups, and assumes that the remaining pixels and / or pixel groups of the respiratory pixels and / or pixel groups are the respiratory flow pixels and / or pixel groups. This enables the rapid and efficient identification of the respiratory motion pixels and / or pixel groups. Generally, it is possible to use a single spatial filter such as a 2D low-pass filter and compare the results with the case without filtering. Therefore, an effect can be seen in the respiratory motion signal, but no significant effect can be seen in the respiratory flow signal.

[0023] In another embodiment, the processing unit determines the respiratory signal from the respiratory pixels and / or pixel groups and eliminates or reduces the contribution of the respiratory motion pixels and / or pixel groups to the respiratory signal in order to obtain the respiratory flow signal as information about the respiratory flow rate. As described above, the use of one or more spatial filters makes it possible to distinguish between respiratory motion and respiratory flow rate in the thermal image so that information about the respiratory flow rate (i.e., a signal representing the respiratory flow rate of the subject) can be determined by the use of the present invention.

[0024] The processing unit further determines the respiratory signal by combining pixels and / or pixel groups that individually exhibit time-period or pseudo-periodic fluctuations in pixel values ​​within the respiratory range, particularly by averaging such pixels and / or pixel groups, or by selecting pixels and / or pixel groups that individually exhibit time-period or pseudo-periodic fluctuations in pixel values ​​within the respiratory range. For this purpose, for example, a single pixel with the best SNR is selected.

[0025] There are several options available to the processing unit regarding how to determine respiratory motion pixels and / or pixel groups, and respiratory flow pixels and / or pixel groups. In one embodiment, respiratory motion pixels and / or pixel groups are determined by detecting pixels and / or pixel groups from among the respiratory pixels and / or pixel groups that represent edges or temperature gradients in the acquired thermal image. In another embodiment, respiratory flow pixels and / or pixel groups are determined by detecting pixels and / or pixel groups from among the respiratory pixels and / or pixel groups that do not represent edges or temperature gradients in the acquired thermal image. In yet another embodiment, respiratory motion pixels and / or pixel groups are determined as respiratory pixels and / or pixel groups whose respiratory signal intensity changes significantly after spatial filtering of the heat, and respiratory flow pixels and / or pixel groups are determined as respiratory pixels and / or pixel groups whose respiratory signal remains substantially the same after spatial filtering.

[0026] Edges in thermal images identify regions and pixels close to significant temperature changes. Respiratory motion in thermal images is visible only when there is contrast present due to a detected temperature difference. Edges identify pixels at the edge of a temperature difference. For example, if the face is approximately 37°C and the blanket is approximately 30°C, then pixels at the edge between the face and the blanket are identified as edges and may contain respiratory motion. Edges can be identified using conventional edge detectors (e.g., detecting gradients as features). From the potential selection of pixels, all pixels on the edge can then be removed to eliminate the contribution of pixels containing exercise-induced breathing.

[0027] To discard exercise-induced respiration, it is possible to completely eliminate edge pixels, but generally, it is not necessary to eliminate non-moving edge pixels. If the intensity of the temporal variation does not depend on the direction of the spatial filter, they are not exercise-induced, and they can be counted as flow-induced respiration pixels or pixel groups.

[0028] In another embodiment, the processing unit determines a respiratory flow pixel or pixel group by determining one or more features from the set of features of the acquired thermal image, and the set of features is Pseudo-periodicity, which indicates the height of the spectral peak of the signal in the respiratory pixel, Respiratory count clusters, which show clusters of pixels with similar frequencies, The gradient that indicates the edge in the thermal image, A correlation value that indicates whether the time-domain signal of a pixel is correlated with the time-domain signal of a breathing pixel, The covariance value shows the covariance between the time-domain signal of a pixel and the time-domain signal of a respiratory flow pixel. Includes.

[0029] The processing unit further determines the flow map by combining two or more features from a set of features and applies one or more spatial filters to the flow map to determine respiratory flow pixels and / or pixel groups. This enables efficient determination of respiratory flow pixels and / or pixel groups.

[0030] As already stated, the use of the present invention determines various information related to the subject's respiration. The processing unit therefore determines one or more of the following: respiratory flow information, respiratory effort information, obstructive apnea, central apnea, and mixed apnea, in order to determine respiratory flow information representing the subject's respiratory flow, in particular by averaging the time-domain signals of respiratory flow pixels and / or pixel groups. In general, it is even possible to determine other respiratory information, such as respiratory rate.

[0031] In one embodiment, the processing unit specifically detects respiratory motion pixels and / or pixel groups, and detects apnea in the subject based on the respiratory motion pixels and / or pixel groups and the respiratory flow pixels and / or pixel groups.

[0032] To sense a second image for additional use in determining respiratory information, another sensor unit, such as an RGB or NIR camera, may optionally be used further. Thus, in one embodiment, the image input unit further acquires a second image of the subject, acquired from radiation in a wavelength range between 400 nm and 2000 nm, particularly visible light or near-infrared light, which is lower wavelength than the radiation from which the thermal image was acquired, and the second image substantially shows at least a portion of the same area as the thermal image. The processing unit then identifies respiratory motion pixels and / or pixel groups in the acquired second image and determines respiratory information using the respiratory motion pixels and / or pixel groups identified in the acquired second image and the respiratory pixels and / or pixel groups identified in the acquired thermal image.

[0033] In a preferred embodiment, the processing unit further detects which of the respiratory pixels and / or pixel groups determined in the thermal image are not present in the second image, and determines respiratory flow pixels and / or pixel groups from among the respiratory pixels and / or pixel groups that are present in the thermal image but not in the second image.

[0034] The thermal sensor unit preferably detects radiation within a wavelength range of 2000 to 20000 nm, particularly within a wavelength range of 5000 to 20000 nm. For this purpose, a thermal camera or thermal sensor array is used.

[0035] In embodiments configured to sense radiation in a wavelength range shorter than the radiation from which the thermal image was acquired, an optional second sensor unit is provided. It is preferable to sense radiation in a wavelength range between 400 nm and 2000 nm, such as visible light or near-infrared light, in order to acquire a second image of the object from that radiation.

[0036] These and other aspects of the present invention will become apparent from the embodiments described below, and will be clarified by referring to those embodiments. [Brief explanation of the drawing]

[0037] [Figure 1] This figure shows a schematic diagram of one embodiment of the system according to the present invention. [Figure 2] This figure shows a schematic diagram of one embodiment of the device according to the present invention. [Figure 3] This figure shows a flowchart of one embodiment of the method according to the present invention. [Figure 4] This figure shows a schematic diagram of another embodiment of the system according to the present invention. [Figure 5] This figure shows an exemplary image obtained from real thermal imaging, with pixels automatically selected, and the corresponding respiratory signal. [Figure 6] This figure shows an exemplary image of the nostril region manually selected, along with the corresponding respiratory signal. [Figure 7] This figure shows the results obtained when features were applied to thermal images. [Figure 8] This figure shows a schematic diagram of the concept of one embodiment of the present invention. [Figure 9] This figure shows three thermal images in the first window and an example of their features. [Figure 10] This figure shows three thermal images in the following window, along with an example of their features. [Modes for carrying out the invention]

[0038] Figure 1 shows a schematic diagram of one embodiment of a system 1 for determining respiratory information of a subject according to the present invention. The subject is generally any human being, such as patients, newborns, and the elderly, but especially any human being who requires close monitoring of respiration, such as neonates, premature infants, or patients with respiratory diseases or dysfunctions.

[0039] Respiratory information generally refers to any information related to the subject's respiration. One or more of the following are particularly important: respiratory flow information, respiratory effort information, obstructive apnea, central apnea, mixed apnea, and respiratory rate. Determining the type of apnea, which is important for determining whether immediate action is required, is useful for determining respiratory flow information, which represents the subject's respiratory flow, and for distinguishing such respiratory flow information from respiratory movement information, which represents the subject's respiratory movement.

[0040] System 1 is primarily used in hospital settings, in patient monitoring settings, or in other settings where the necessary equipment is available or already available. System 1 can also be used at home, where it may be useful to give the device to patients who are suspected of having sleep apnea and are being examined by their primary care physician. Trying the device at home for a few nights may aid in diagnosis.

[0041] System 1 comprises a thermal sensor unit 10 for collecting a thermal image of a subject, wherein the thermal image shows at least a portion of the subject's facial area and / or its surrounding area. For example, the area of ​​the nose (especially the nostrils) and / or the oral cavity or areas close to them, as well as adjacent areas (such as parts of a pillow, blanket, or clothing adjacent to the nostrils and / or oral cavity), are of particular importance and should be shown in the thermal image. The thermal sensor unit 10 is generally any means or unit capable of collecting a thermal image by sensing radiation within a corresponding wavelength range, particularly 2000 to 20000 nm, preferably 5000 to 20000 nm, such as a thermal camera or thermal sensor array.

[0042] System 1 further comprises a device 20 disclosed herein and described below for determining the respiratory information of a subject from thermal images. Details of device 20 are described below.

[0043] System 1 further comprises an output interface 30 for outputting respiratory information of a subject. The output interface 30 is generally any means or user interface that outputs information in a visual or audible format, such as in text format, as an image or graphic, or as sound or spoken words. For example, the output interface 30 may be a display, loudspeaker, touchscreen, computer monitor, smartphone or tablet screen, etc.

[0044] System 1 further optionally includes a second sensor unit 40 for sensing radiation in a wavelength range shorter than the radiation from which the thermal image was acquired, particularly in the wavelength range between 400 nm and 2000 nm, such as visible light or near-infrared light. From the radiation sensed by the second sensor unit 40, a second image of the subject, for example, an RGB image or an NIR image, further used in certain embodiments for determining respiratory information, is generated. The second sensor unit 40 is therefore, for example, an RGB or NIR camera.

[0045] Figure 2 shows a schematic diagram of one embodiment of the device 20 according to the present invention.

[0046] Device 20 comprises an image input unit 21 for acquiring a thermal image of a subject, wherein the thermal image shows at least a portion of the subject's facial field and / or surrounding area. The image input unit 21 is either directly coupled to or connected to the thermal sensor unit 10 (and, if available, a second sensor unit 40), or acquires (i.e., retrieves or receives) the respective images from storage, a buffer, a network, or a bus. The image input unit 21 is therefore a (wired or wireless) communication interface or data interface, such as a Bluetooth interface, a WiFi interface, a LAN interface, an HDMI® interface, a direct cable connection, or any other suitable interface that enables signal transmission to Device 20.

[0047] Device 20 further comprises a processing unit 22 for determining the respiratory information of a subject from a thermal image. The processing unit 22 is any kind of means for processing the image and determining the respiratory information. The processing unit 22 is implemented in software and / or hardware, for example, as a programmed processor or computer, or as an application on a user device such as a smartphone, smartwatch, tablet, laptop, PC, or workstation.

[0048] Device 20 further comprises an output unit 23 for outputting determined respiratory information of a subject. The output unit 24 is generally an arbitrary interface that provides determined information, for example, by transmitting it to another device or making it available for retrieval by another device (e.g., a smartphone, computer, tablet, etc.). The output unit is therefore generally an arbitrary (wired or wireless) communication or data interface.

[0049] Figure 3 shows a flowchart of one embodiment of Method 100 according to the present invention. The steps of Method 100 are performed by Device 20, and the main steps of the Method are performed by Processing Unit 22. The Method is implemented, for example, as a computer program running on a computer or processor.

[0050] In the first step 101, a thermal image of the object is acquired. Optionally, in some embodiments, a second image (such as an RGB or NIR image) is further acquired.

[0051] In the second step 102, respiratory information of the subject is determined from the thermal image. This involves several steps.

[0052] In step 103, respiratory pixels and / or pixel groups that indicate temperature fluctuations related to respiration are identified in the acquired thermal image. In step 104, respiratory flow pixels and / or pixel groups that indicate respiratory flow rate are determined from among the respiratory pixels and / or pixel groups. For this purpose, one or more spatial filters, such as Gabor filters, are applied to the respiratory pixels and / or pixel groups to distinguish them from respiratory motion pixels and / or pixel groups that indicate respiratory motion. In step 105, the desired respiratory information of the subject is determined from the respiratory flow pixels and / or pixel groups and / or respiratory motion pixels and / or pixel groups.

[0053] In the final step 106, the determined respiratory information for the subject is output.

[0054] Figure 4 shows a schematic diagram of another embodiment of the system 200 according to the present invention. The system 200 comprises a sensor means 201 (representing a thermal sensor unit 10 in system 1) to non-invasively capture at least a portion of the radiation of an object using, for example, a 2D array of thermal sensors and output pixel data 202. A device 20 in system 1 is provided, in particular a processing means 203 and a focusing means 206 that implement its processing unit 22. The processing means 203 calculates a breathing signal (respiration signal) 205 of an object from the pixel data 203 of the sensor means 201, in particular from breathing pixels 204. The focusing means 206 reduces or substantially eliminates the contribution of pixels containing exercise-induced breathing from the breathing signal 205 and outputs a remaining breathing-flow signal (respiratory flow signal) 207.

[0055] In one embodiment, an algorithm is used to automatically identify pixels containing respiration. Pixels containing respiration signals can be located independently of their origin (respiratory motion or respiratory flow). An exemplary implementation of the algorithm uses one or more features, for example, a combination of three features, to locate core pixels, i.e., pixels representing the subject's respiration (which may be motion-induced or flow-induced) with the highest quality according to criteria. Exemplary criteria may be amplitude, signal-to-noise ratio, spectral purity, spectral skewness, or a combination of several criteria. Respiratory pixels (or pixel groups) are then determined, for example, based on the correlation between the pixels and core pixels. Each feature is based on the characteristics of the respiration signal and the characteristics of the respiration pixel.

[0056] The first characteristic is pseudoperiodicity. Pseudoperiodicity generates a map in which each pixel value corresponds to the peak height of the normalized spectrum. This map should identify the location of pixels containing the respiratory signal, based on the fact that respiration is periodic.

[0057] The second feature is the respiratory rate cluster. This second feature is based on the observation that respiratory pixels are not isolated but constitute clusters. A map is constructed in which each pixel value corresponds to the frequency of peaks in the spectrum (for example, the value of pixel 1 is 50 breaths per minute). A 2D nonlinear filter is applied to this map, giving higher weights to nearby pixels that have similar frequencies.

[0058] The third feature is the gradient. This feature requires the use of an edge detector to locate edges in the thermal image, since respiratory motion pixels are located only on edges. A standard edge detector can be used for this feature. For respiratory flow pixels, the reciprocal of the gradient is used, based on the knowledge that respiratory motion pixels are located on edges, but respiratory flow pixels are not usually located on edges.

[0059] A fourth optional feature is the correlation map. The correlation map is a map containing pixels with time-domain signals that have an absolute correlation value higher than 0.6 (or another set value) with signals selected to be respiratory signals (extracted from so-called core pixels). Core pixels are found as pixels corresponding to the maximum value when the pseudoperiodicity and respiratory rate clusters are multiplied together by the gradient. Thus, the correlation map is a binary map (0 and 1), where 1 indicates a possible location of a respiratory pixel. However, it is unknown whether a pixel is a respiratory motion pixel or a respiratory flow pixel.

[0060] A fifth optional feature is the covariance map. The covariance map is constructed when the flow core pixels (i.e., the pixels most likely to be respiratory flow pixels) are known. Once the covariance map is known, there are pixels that are out of phase with the respiratory flow signal extracted from the flow core pixels. These pixels are respiratory motion pixels that have been excluded from the possible selectable pixels (respiratory motion exists in two variants in the thermal image, one that is nearly in phase with the flow signal and the other that is nearly out of phase depending on its position on the pixel). Respiratory flow pixels may exhibit an additional (smaller) phase shift caused by and corresponding to the heat capacity of tissue or bedding that changes its temperature due to respiratory flow.

[0061] The core pixel is found by multiplying the features (the first three features in this example) and selecting the pixel corresponding to the maximum value after multiplication. This core pixel is then used to find all other breathing pixels based on their correlation values. The absolute values ​​of the correlation coefficients obtained between the core pixel's signal and the signals of all the other pixels are placed in a 2D map called a correlation map. The breathing signal is then generated by averaging together the pixels that have absolute correlation coefficients higher than a given value, for example, 0.7 or 0.8.

[0062] In general, an input map for spatial filtering can be constructed using a smaller (or larger) set of features. However, due to the presence of noise and the difficulty of generalizing solutions for entirely different recordings, the more features there are, the better the results. One or more features are used to find all pixels that contain respiration, which could be pseudoperiodic or respiratory count clusters, for example. From this set of pixels that contain respiration, pixels that contain respiratory flow will be found. Once these pixels containing respiration are available, a spatial filter or the reciprocal of the gradient can be applied to directly eliminate respiratory motion. Therefore, preferably, at least two features are used, one feature used to find all respiration pixels and one strategy or feature used to find respiratory flow pixels among these respiration pixels (e.g., using a spatial filter or by eliminating edges).

[0063] Figures 5-7 illustrate the idea of ​​the present invention by using simulated obstructive apnea (where airflow is absent but respiratory effort is still present) added to real thermal images collected from infants.

[0064] Figure 5 shows an exemplary image obtained from a real thermal image with pixels automatically selected (pixels in the nostrils are indicated by P, and motion pixels at the edge between the blanket and the face are indicated by P'). Furthermore, the respiratory signal obtained from this image is shown. Since respiratory motion pixels are also selected, manually added obstructive apnea is undetectable.

[0065] Figure 6 shows an exemplary image in which the nostril region (indicated by N) is manually selected. Furthermore, the respiratory signal obtained from this image is shown, demonstrating the presence and generally detectable nature of obstructive apnea.

[0066] By combining information related to respiratory motion pixels with information related to respiratory flow pixels, it is possible to construct features that focus solely on detecting flow pixels. Figure 7 shows the results obtained when these features are applied to thermal imaging. The automatically selected pixels (indicated by F) relate only to the flow range, in contrast to the automatically selected pixels P in Figure 5. In this case, simulated obstructive apnea is visible and can be detected using the present invention.

[0067] More specifically, this exemplary result was obtained by applying a bank of Gabor filters to a previously acquired correlation map. By multiplying all response values ​​for the Gabor filters, a portion of the correlation map, independent of the filter orientation, is amplified. The respiratory flow pixels can therefore be automatically identified.

[0068] Other embodiments identify pixels containing flow by using optics to focus only on the nasal region or by positioning a camera near the nostrils. This is either quite complex, as it requires detecting the nasal region first, or having the camera nearby can be uncomfortable for the subject. Moreover, much stronger flow signals may exist elsewhere, for example, in thermal images on a pillow close to the subject's nose. These stronger signals cannot benefit from using a nasal detector, but according to the present invention, these signals can be used to obtain a much stronger SNR or enable flow detection even when the nostrils are not visible.

[0069] In another exemplary implementation of the present invention, the identification of respiratory flow pixels (hereinafter also referred to as RF pixels) is based on the five features already partially described above, combined with a novel use of a bank of Gabor filters. These filters make it possible to take advantage of the characteristics of RF pixels, namely that their pixels generally occur in a 2D smooth region. Using the selected pixels, the RF signal and flow-based respiratory rate (RR) can be obtained.

[0070] Furthermore, obstructive apnea was simulated in multiple video segments, and known methods for detecting respiratory arrest (COB) were used (described in Lorato, I., Stuijk, S., Meftah, M., Verkruijsse, W., de Haan, G. Camera-Based On-Line Short Cessation of Breathing Detection. 2019 IEEE / CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019, pp. 1656-1663). This allows for a comparison of the detectability of obstructive apnea in different respiratory signals acquired from thermal imaging (i.e., mixed respiratory signals (MR signals, i.e., respiratory signals derived from both RF and respiratory movement (RM)) and RF signals).

[0071] These steps are summarized in Figure 8, which shows a schematic diagram of the concept of one embodiment of the present invention. RF pixels are automatically detected in the thermal image (Figure 8A) (Figure 8B) and used to calculate the RF signal and flow-based RR (Figure 8C). Furthermore, the locations of the RF pixels are manually annotated (Figure 8D). To simulate the occurrence of OA, the annotated RF pixels are replaced with noise (Figure 8E). A COB detector is used to compare the performance of OA detectability between the RF signal and the MR signal (Figure 8F). Exemplary implementations are described below.

[0072] First, preprocessing is performed. Thermal images from one or more camera views, for example, three camera views, are merged on the same image plane, as seen in Figure 8A, thereby obtaining a single image with a resolution of 180 × 80, i.e., M × L. To compensate for the non-uniform sampling rate, each pixel time-domain signal was interpolated using 1D linear interpolation. The resulting frame rate is 9 Hz, which is close to the average frame rate of the infrared camera.

[0073] In the next step, respiratory flow detection is performed. The embodiments described above are used for the automatic detection of RF pixels. In one embodiment, a set of, for example, five features (three of which have already been described above, and in other embodiments, more or fewer features may be used) is combined to identify the RF pixels. The flow core pixel, i.e., the pixel most likely to belong to the RF pixels (the pixel representing the respiratory flow of the subject with the highest quality according to the criteria, although there may be pixels representing exercise-induced respiration with an even higher score on that criterion), is selected because it is used as the basis for the calculation of one of the features. For precise selection of the flow core pixel, a spatial filter (e.g., a Gabor filter) is used. The time-domain signal of each pixel in each window is x m、l (nT s This is called n=0+(j-1) / T, where (m, l) indicates the pixel position and n=0+(j-1) / T s , 1+(j-1) / T s ..., N+(j-1) / T s In an exemplary implementation, each window is identified by an integer j and consists of N=72 consecutive samples within an 8-second fragment that slides in 1-second steps. The sampling period is T. s is T s = Equivalent to 0.111 seconds.

[0074] Gabor filters are well-known bandpass filters used in image processing for texture and edge detection. The kernel is formed from a sinusoidal carrier and a 2D Gaussian envelope. Several Gabor filters can be generated by varying the spatial frequency of the sine wave and the orientation of the filter. By applying a set of filters to an image, edges and textures can be enhanced. Banks of Gabor filters are applied by varying the orientation and / or spatial frequency, with the aim of identifying the locations of flow pixels that should have similar response values ​​for all orientations, by examining the distribution characteristics of RF and RM pixels. On the other hand, for RM pixels, higher response values ​​are often expected in certain directions, sometimes along curved lines. For orientation and spatial frequency of the filter, a set of parameters, λ = 3, 4, ..., 8 pixels / cycle and θ = 10°, 20°, 30°, ..., 170°, is empirically selected. Since the flow rates visible in the nasal cavity / oral cavity and / or on the fibers typically create regions affected by flow rates of different sizes, multiple spatial frequencies are selected so that this method can work with both of these flow rates.

[0075] By convolving the input map using each Gabor filter, the banks of Gabor filters are applied to the input map, called the Flow Map, as defined below. In particular,

number

number

[0076] therefore,

number

number

number

number

number

number

number

number

number

number

number

[0077] To acquire MR signals from thermal imaging, the other four features in equation (3) are developed. All of these features are described above as the first to fifth features (pseudoperiodicity, respiratory rate clusters, gradient, correlation map, and covariance map).

[0078] These features are designed to locate MR pixels, but can be adapted for the identification of RF pixels. Q is called pseudo-periodicity and is based on the estimation of the peak height of the normalized spectrum. W is called RR cluster and is based on the application of a 2D nonlinear filter for the detection of nearby pixels with similar frequencies. G is the gradient that identifies the edges of the thermal image. These three features are used to identify the core pixel, i.e., the pixel that best represents the MR signal. Once the core pixel is found, the Pearson correlation coefficient is used to locate all other pixels that contain the respiratory signal. By averaging these pixels together, the MR signal is obtained. The Pearson correlation coefficients obtained between the core pixel and all other pixels are placed in a correlation map and indicated by C. Using the correlation map obtained from the core pixel, the location of the MR pixel can be identified. This map has an empirical threshold ξ1 for an absolute value equal to 0.6.

number

number

number

[0079] Therefore, RF pixels are detected, and flow core pixels are used only in the first window of each video segment. Then, [Numerical] all non-zero pixels in are considered RF pixels. [Numerical] P flow is thus a set containing the positions of the detected RF pixels. The conditions for RF pixel detection are quite strict. It is possible that no pixels are found at all. In that case, the previously selected RF pixels are used in the current window as well. P flowThe RF signal is obtained by averaging all the RF pixels contained within together. An example of features in the first window is shown in Figure 9. Features obtained in the next window are shown in Figure 10. In these figures, the location of the annotated RF pixels is shown along with the periphery. Figures 9 and 10 demonstrate the advantages of using a covariance map that excludes inverted RM pixels. As a result, the flow map shown in Figure 10 does not include RM pixels compared to the flow map shown in Figure 9. The MR signal is also obtained from the image and used for comparison purposes.

[0080] In the next step, obstructive apnea detection is performed. Using a COB detector (disclosed, for example, in the literature by Lorato, I., Stuijk et al. mentioned above), the detectability of simulated obstructive apnea is evaluated as shown above. The COB detector assumes that COB can be detected by monitoring abrupt amplitude changes and that COB is based on a comparison of short-term and long-term standard deviations. Adaptations are made to the COB detector regarding the length of the windows for calculating the two standard deviations. The duration of these windows is generally selected based on the targeted COB. In particular, the window for calculating the short-term standard deviation is close to the minimum COB duration of 10 seconds, while the window for the long-term standard deviation, which is calculated as the median of the short-term standard deviations, may be longer than the COB duration. Otherwise, the long-term standard deviation dynamically adapts to the standard deviation during the apneic event (i.e., detecting the cessation of the event while apnea is still continuing). In another implementation, the short-term standard deviation is calculated using an 8-second window, which is the same sliding window technique used for RR estimation. The long-term standard deviation is calculated within a 15-second window (other cycles may also be applicable). This window may be shortened to 11 seconds, considering the fact that this is closer to the designed duration of obstructive apnea, but it may also be kept longer to easily adapt to unsimulated cases.

[0081] The RF and MR signals are acquired, for example, using a dataset with simulated obstructive apnea, as described above. The COB detector is applied to a reference RF (RefRF) signal (acquired by averaging all annotated RF pixels together), which is the benchmark for achievable results when monitoring RF, with respect to the RF signal acquired by applying the proposed method, and with respect to the MR signal to highlight the limitations of monitoring this type of signal when the purpose is apnea detection.

[0082] In another embodiment, in addition to the thermal sensor unit (also called the first sensor unit), a second sensor unit (40, see Figure 1), such as an optical camera, is used to orient substantially the same scene including the object. The idea is that the thermal camera orients both respiratory flow and respiratory motion, while the optical camera can orient only respiratory motion. The combination allows for the separation of the two respiratory signals.

[0083] While exercise-induced thermal fluctuations can be helpful in determining the respiratory rate from a thermal camera, using data from an optical camera allows for the measurement of the respiratory flow signal (provided that the respiratory flow signal is present in the captured image, which may depend on the camera view, and that multiple cameras viewing the object from multiple angles may be used to ensure the visibility of the respiratory flow). Furthermore, illumination means (e.g., dedicated lighting means or mere ambient light) are used to illuminate the object with radiation within the wavelength range captured by the second sensor unit.

[0084] In such embodiments, the processing unit of this device (22 in Figure 2) is configured to determine corresponding pixel data between pixels from the first sensor unit and pixels from the second sensor unit, and to search for image regions in the thermal image that exhibit respiratory information that is not present in the corresponding regions in the second image collected by the second sensor unit. The respiratory flow rate is calculated by calculating the respiratory signal based only on those pixels.

[0085] The first and second sensor units are oriented so that they capture the same portion of the scene containing the same part of the object. If there is an inevitable parallax between their views, image alignment techniques are used to align the images to define the corresponding pixels.

[0086] Both thermal and optical imagers are capable of detecting respiratory motion, and both generally find this motion on the edges of their respective images, but it should be taken into account that optical cameras typically find far more image edges than thermal cameras do. Thermal images generally have little detail, with strong edges occurring mainly at the boundaries between the subject and its environment. Optical cameras also generally "see" these edges, but they locate far more edges because the reflection of light produces images with much richer detail.

[0087] Therefore, while respiratory flow is available in relatively flat regions of the thermal image, considerable detail may still be present in the optical image. When removing pixels from detail regions, too many pixels may be removed from the thermal image from regions containing only respiratory flow. However, the remaining pixels are almost certainly in flat optical regions, which should be flat thermal regions. If respiratory flow occurs in such regions of the thermal image, those pixels can be used to output an (initial) respiratory flow signal.

[0088] A further finding is that, since there is little possibility of using pixels from thermal images, the resulting signal may have an insufficient signal-to-noise ratio (SNR). This can, however, be prevented by recognizing that flow-based thermal fluctuations and motion-based thermal fluctuations also have different signal phases and shapes. The pixel exhibiting the strongest pseudoperiodic signal is used to find a core pixel, i.e., a pixel with a similar phase and shape (using correlation values), and the core pixel is combined with similar pixels to obtain a robust (high SNR) respiratory signal. Core pixels generally have respiratory motion, as respiratory motion is often stronger. To find similar pixels to combine with for improved SNR of the respiratory flow output signal, the combined pixels of the "initial respiratory signal" can be used as new "core pixels".

[0089] The proposed devices, systems, and methods have yielded promising results in the automatic identification of respiratory flow pixels in thermal images and the acquisition of RF signals. Furthermore, these devices, systems, and methods enable the detection of different types of respiratory information, particularly apnea and the determination of the type of apnea.

[0090] The practical implementation should aim to maximize RF visibility in the thermal image. An array of cameras may be used to ensure visibility of the oral cavity and / or nasal cavity region in the image. Furthermore, the subject should preferably be in a supine position.

[0091] The present invention is particularly applicable to patient monitoring in non-invasive vital signs monitoring. The present invention can be used in a variety of scenarios and environments, including (but not limited to) hospitals, specialized sleep centers, and home care (of infants).

[0092] While the present invention has been illustrated and described in detail in the drawings and the above description, such illustrations and descriptions should be considered illustrative or typical, and not limiting. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and practiced by those skilled in the art in carrying out the claimed invention, based on a study of the drawings, this disclosure, and the appended claims.

[0093] In the claims, the word “equipped with / possessing” does not exclude other elements or steps, and a singular element does not exclude plural elements. A single element or other unit may perform the functions of several items described in the claims. The mere fact that several means are described in different dependent claims does not imply that combinations of these means cannot be used advantageously.

[0094] Computer programs may be stored / distributed on suitable non-temporary media, such as optical storage media or solid-state media, supplied together with or as part of other hardware, as well as in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0095] No reference numeral in the claims should be construed as limiting the scope.

Claims

1. A device for determining the respiratory information of a subject, wherein the device is An image input unit for acquiring a thermal image of the subject, wherein the thermal image shows at least a portion of the facial area and / or surrounding area of ​​the subject; A processing unit for determining the respiratory information of the subject from the thermal image, An output unit for outputting the respiratory information of the target that has been determined. Equipped with, The aforementioned processing unit In the acquired thermal image, identify the respiration pixels and / or pixel groups that show temperature fluctuations related to respiration, Determining respiratory flow pixels and / or pixel groups from among the respiratory pixels and / or pixel groups, wherein one or more spatial filters are applied to the respiratory pixels and / or pixel groups in order to distinguish the respiratory flow pixels and / or pixel groups from respiratory motion pixels and / or pixel groups that indicate respiratory motion, Determining the respiratory information of the subject from the respiratory flow pixels and / or pixel groups, and / or the respiratory motion pixels and / or pixel groups. A device that determines the respiratory information of the subject.

2. The processing unit applies at least two spatial filters to the breathing pixels and / or pixel groups, each having at least two different spatial filtering directions and / or one or more spatial frequencies. The device according to claim 1.

3. The processing unit identifies the respiratory motion pixels and / or pixel groups within the respiratory pixels and / or pixel groups by using one or more spatial filters, and assumes that the remaining pixels and / or pixel groups of the respiratory pixels and / or pixel groups are the respiratory flow pixels and / or pixel groups. The device according to claim 1 or 2.

4. The processing unit determines a respiratory signal from the respiratory pixels and / or pixel groups and removes or reduces the contribution of the respiratory motion pixels and / or pixel groups to the respiratory signal in order to obtain a respiratory flow signal as information regarding respiratory flow. The device according to claim 1.

5. The processing unit determines the respiratory signal by combining pixels and / or pixel groups that individually exhibit time-period or pseudo-periodic fluctuations in pixel values ​​within the respiratory range, particularly by averaging such pixels and / or pixel groups, or by selecting pixels and / or pixel groups that individually exhibit time-period or pseudo-periodic fluctuations in pixel values ​​within the respiratory range. The device according to claim 4.

6. The aforementioned processing unit From among the aforementioned respiratory pixels and / or pixel groups, the respiratory motion pixels and / or pixel groups are determined by detecting pixels and / or pixel groups that represent edges or temperature gradients in the acquired thermal image, and / or From among the aforementioned respiratory pixels and / or pixel groups, the respiratory flow pixels and / or pixel groups are determined by detecting pixels and / or pixel groups that do not represent edges or temperature gradients in the acquired thermal image, and / or Respiratory motion pixels and / or pixel groups are determined as respiratory pixels and / or pixel groups whose respiratory signal intensity changes significantly due to the spatial filtering of heat, and respiratory flow pixels and / or pixel groups are determined as respiratory pixels and / or pixel groups whose respiratory signal remains substantially the same due to the spatial filtering. To do The device according to claim 1.

7. The processing unit determines a respiratory flow pixel or pixel group by determining one or more features from the acquired set of features of the thermal image, and the set of features is The pseudoperiodicity that indicates the height of the spectral peak of the signal in the respiratory pixel, Respiratory count clusters, which show clusters of pixels with similar frequencies, The gradient that indicates the edge in the thermal image, A correlation value that indicates whether the time-domain signal of a pixel is correlated with the time-domain signal of a breathing pixel, The covariance value shows the covariance between the time-domain signal of a pixel and the time-domain signal of a respiratory flow pixel. including, The device according to claim 1.

8. The processing unit determines a flow map by combining two or more of the features from the set of features, and applies one or more spatial filters to the flow map to determine the respiratory flow pixels and / or pixel groups. The device according to claim 7.

9. The processing unit determines, in particular, one or more of the following: respiratory flow information, respiratory effort information, obstructive apnea, central apnea, and mixed apnea, in order to determine respiratory flow information representing the target respiratory flow by averaging the time-domain signals of the respiratory flow pixels and / or pixel groups. The device according to claim 1.

10. The processing unit further detects respiratory motion pixels and / or pixel groups, and detects the apnea of ​​the target based on the respiratory motion pixels and / or pixel groups and the respiratory flow pixels and / or pixel groups. The device according to claim 1.

11. The image input unit further acquires a second image of the target obtained from radiation within a shorter wavelength range than the radiation from which the thermal image was acquired, particularly visible light or near-infrared light, within a wavelength range between 400 nm and 2000 nm, and the second image substantially shows at least a portion of the same region as the thermal image. The processing unit further identifies respiratory motion pixels and / or pixel groups in the acquired second image, and determines the respiratory information using the respiratory motion pixels and / or pixel groups identified in the acquired second image and the respiratory pixels and / or pixel groups identified in the acquired thermal image. The device according to claim 1.

12. The processing unit further detects which of the respiratory pixels and / or pixel groups determined in the thermal image are not present in the second image, and determines the respiratory flow pixel and / or pixel group from among the respiratory pixels and / or pixel groups that are present in the thermal image but not in the second image. The device according to claim 11.

13. A system for determining the respiratory information of a subject, wherein the system is A thermal sensor unit for collecting a thermal image of the subject, wherein the thermal image shows at least a portion of the facial area and / or surrounding area of ​​the subject. A device according to claim 1 for determining the respiratory information of the subject from the thermal image, An output interface for outputting the respiratory information of the aforementioned target and A system that includes these features.

14. A method for determining the respiratory information of a subject, wherein the method is A step of acquiring a thermal image of the subject, wherein the thermal image shows at least a portion of the facial area and / or surrounding area of ​​the subject. The steps include determining the respiratory information of the subject from the thermal image, A step of outputting the respiratory information of the target that has been determined. It has, The respiratory information of the subject is The steps include identifying respiratory pixels and / or pixel groups that show temperature fluctuations related to respiration in the acquired thermal image, A step of determining respiratory flow pixels and / or pixel groups from among the respiratory pixels and / or pixel groups, wherein one or more spatial filters are applied to the respiratory pixels and / or pixel groups in order to distinguish the respiratory flow pixels and / or pixel groups from respiratory motion pixels and / or pixel groups that indicate respiratory motion, A step of determining the respiratory information of the subject from the respiratory flow pixels and / or pixel groups, and / or the respiratory motion pixels and / or pixel groups. The method determined by

15. A computer program comprising program code means for causing a computer to perform steps of the method of claim 14 when executed on the computer.

Images