Method and device for determining one or more relevant image areas

Abstract

A method (100) for determining one or more relevant image regions, comprising the following steps: obtaining a recording sequence; extracting (120) signals from the recording sequence to obtain a first signal sequence; obtaining a second signal sequence; performing a joint evaluation (130) of the first signal sequence and the second signal sequence; and determining a mask (140) with the one or more relevant image regions based on the joint evaluation (130).

Description

Exemplary embodiments of the present invention relate to a method and an associated device for determining one or more relevant image regions. Further exemplary embodiments relate to a corresponding computer program and a system comprising the apparatus mentioned with a spirometer and / or a camera. Preferred embodiments relate to a method for determining one or more image areas of a person or in particular of a torso in order subsequently to monitor the breathing system of the person.In Germany alone, respiratory system diseases such as asthma or chronic obstructive pulmonary diseases (COPD) are responsible for 20.2% of all causes of incapability and 10.1% of the day of the disease in 2021 [1] The diagnosis of pulmonary diseases takes place according to the current gold standard by measurements of the volume flow through, for example, spirometer or bodyplethysmograph. Both methods require direct contact with the patient through a mouthpiece. The use of a mouthpiece can lead to some problems:• Discomfort to patients during insertion or holding of the mouthpiece,• Difficulties in sealing the mouthpiece• Restrictions in mouth / airway which make insertion or use of a mouthpiece difficult,• Allergic reaction to the material of the mouthpiece,• Outlay by cleaning or changing the mouthpiece.A physician can then make a diagnosis based on the volume flow diagram. Changes from the normal state are shown here. However, these measuring methods do not make it possible to identify the anatomical regions which are involved in the respiration process. Asynchronous muscle weakness or failure of a lung lobe thus cannot be localized.The most common medical studies for assessing lung function are spirometry and whole body psychometry. Both methods are carried out by pulmonary practitioners in order to determine deviations of the measurement parameters from standard values and thus to diagnose diseases such as COPD or asthma.Spirometry is the most common method for routine pulmonary study. In this case, lung volumes and the volume flow are measured. In modern spirometry, so-called open systems are used which measure the flow rate, also referred to as flow, of the gases over time. This method is also referred to as pneumotachography. The patient inhales and exhausts ambient air via a mouthpiece. A spirometric sensor measures the mean flow velocity in the breathing tube [2]. The respiratory volumes can be determined from the surrogate signal of the temporal flow rate via integration.In expanded lung function diagnostics, bodyplethysmographs, also referred to as whole bodyplethysmographs, are used for differential diagnostics and determination of the non-ventilation respiratory volumes, such as the residual volume. The patient moves into a closed chamber for this purpose. Respiration is then measured via volume and pressure changes in the chamber interior. Present bodyplethysmographs are mainly based on the volume-constant method in which the patient breathes the chamber air through a flow sensor [2]. A shutter mechanism (shutter) can briefly interrupt the breathing.The measuring principle is based on the law of Boyle and Mariotte: In the case of changes in state of an ideal gas, the volume is inversely proportional to the pressure at a fixed number of particles and constant temperature. Thus, an increase in the thorax volume, due to the rising rib cage during inspiration, causes a reduction in the volume in the chamber with simultaneously increasing chamber pressure. The ventilation during exhalation is interrupted via the shutter. Thus, the alveolar pressure at the mouthpiece can be measured. The functional residual capacity as a potential measurement parameter can subsequently be determined via the product of a calibration constant with the ratio of the measured variables chamber pressure and mouth pressure.Clinical evaluation of respiratory movements is currently being performed by manual evaluation of respiratory movement (engl. Manual Assessment of Respiratory Motion, MARM) [3]. MARM is a palpatoric method based on subjective chest movements experienced by the examiner. Another method, respiratory inductance plethysmography (engl. Respiratory Induction Plethysmography) [4] is based on the attachment of chest straps around the human upper body. The extent of the chest during respiration is proportional to the lung volume and can be measured by self-induction of coils in the chest straps. The chest straps are attached to the thorax and abdomen to differentiate the corresponding movements.Further contactless methods are used to evaluate the work of the individual compartments. This includes opto-electronic plethysmography (Eng. Optoelectronic Plethysmography, OEP) [5]. Markers clearly visible to the OEP are applied to the upper body of the patient, which are then automatically registered by software. After extensive calibration, the structure of the rib cage can be calculated by reconstruction methods. The number of markers is not fixed and is stated in the literature in a range from 5 [6] to 89 [7] markers. For the OEP, several cameras are needed to record the markers and their spatial movements. Examples of OEP are shown in Figures 1a, 1b and 1c and 2.FIG. 1 ashows an apparatus for optoelectronic plethysmography comprising a plurality of cameras 10 for observing an object 12. The plurality of cameras output their camera signal to a calculation unit 14, which calculates a volume 16 on the basis thereof. The volume 16 is shown in FIG. 1 b.The volume 16 can be plotted, for example, with respect to signals to be extracted, here e.g. signals V CW, V RCP, V RC, A, V AB. These signals are shown, for example, in FIG. 1c.The object 12 can also be observed over time by means of the cameras 10, with the result that the calculated volume 16 is calculated for different points in time and the signals V CW to V AB can be plotted over time (cf. FIG. 1 c ).As already explained above, the observation of the torso and thus also of the signals V CW to V AB takes place on the basis of spatial movements of the torso as object 12. The markers are represented by intersections in Fig. 1b. The resulting model with values emphasized by hatching is shown in FIG. 2. as can be seen from FIG. 2, not all regions of interest contribute to depth-based plethysmography. On the basis of the hatching, a subdivision into three subregions is carried out, namely lung rib cage R 1, abdominal rib cage R 2 and abdominal space R 3. The points with the reference symbols M represent the markers. It is disadvantageous that this marker-based method requires the prior application of these markers and is thus associated with an increased preparation effort. The spatial resolution of the analysis of the breathing mechanics is defined by the distances of the markers to each other and thus cannot be extended arbitrarily.Depth information can be determined by means of structured light methods, active stereoscopy and time-of-light methods without the application of markers. Contactless measuring methods use the change of the distance information of the upper body to the camera in order to draw conclusions about the respiration rate on the one hand by the periodic change and about the respiratory volumes by the absolute distance changes. For this, the selection of a region of interest (engl. Region of interest, ROI) of central importance to measure only the thoracic distance change and exclude other parts such as arbitrary head movements.Such an ROI can be selected, for example, by level-set segmentation [9]. Spatial and temporal information is used to define the ROI. The shape of the human breast is used as a priori information and the segmentation of adjacent time windows as temporal information. The result here is a mask that selects relevant portions for respiration. However, this method does not allow the relative contributions to respiration to be illustrated. It is not possible to emphasize which contribution individual regions have to the total respiration. These are disadvantages of the method. The level set segmentation is illustrated in FIG. 3.FIG. 3 illustrates the selection of a region of interest by level segmentation, wherein FIG. 3 ashows a differential image of successive depth images, FIG. 3 bshows the result of the level set segmentation and FIG. 3 cshows the result including the spatial relationship. In FIG. 3 d, further inclusion of temporal information is shown [9].The selection of the ROI (region of interest) via the amplitude

[10] is shown in FIG. 4. Here, the selection is based on a preselection of an ROI as a rectangular surface, which is identified by the reference symbol ROI. In this simplified variant, the determination of the region of interest (ROI) is made as a selection of sub-regions (cf. hatched regions). The regions are color-coded with corresponding guides according to their amplitude strengths from strongly shaded (strongly) to weakly shaded (weakly)

[10] These are selected from top to bottom until 90% of the amplitude value of the total amplitude of the ROI is reached. This approach could allow visual highlighting of the strips based on their contribution. However, the summation only analyzes a linear relationship in the respiratory interaction-a disadvantage of the method. Another limitation is the preference for selecting the regions from top to bottom without being able to determine that the upper regions have a stronger contribution in the thoracic level than the abdominal regions. Also, different ratios on both sides of the body (right and left) cannot be analyzed by this algorithm.A further automatic segmentation is used by

[11] . Background and foreground pixels are removed from the mean image of the entire measurement by thresholding. Contiguous segments are formed row and column by column. Finally, an anatomical adaptation is carried out in order to adapt the ROI to the torso height of the test persons. It is thus possible to select an ROI, but not to delimit and emphasize corresponding involved regions. The fact that the contribution of individual regions cannot be illustrated is a disadvantage of this method. Since the thresholding is distance-based alone, pixels are selected which are at the distance to the expected upper body.However, it is not taken into account here whether the corresponding region also actively contributes to the respiration. Finding connected components (row-wise or column-wise) likewise has no influence on this, just like taking into account anatomical structures.FIG. 5 illustrates a method with five method steps for selecting an ROI by thresholding (limit value formation) the averaged images, followed by the selection of contiguous components in rows and columns and an anatomical correction, as explained in

[11] .In the field of remote photoplethysmography (rPP), there are approaches that automatically derive an ROI in the face. The difficulty for these algorithms is to segment only the free skin pixels and exclude pixels with non-living or low information content, such as hairs or eyeglasses. The corresponding information content of the pixels can be represented, as shown in FIGS. 6 and 7, for example. For example, the signal-to-noise ratio

[12] , the heart rate

[13] , correlation coefficients

[14] , or a combination of different metrics

[15] can be used as information content. For this purpose, the individual elements can be evaluated either pixel by pixel [12-141 or voxel-based

[15] . Pixel-by-pixel evaluation is likewise to be understood as a summary of further rectangular shapes. However, these approaches have not yet been used for creating an individual body mask for determining respiration parameters in contactless plethysmography.It is an object of the present invention to provide a concept for determining a mask having one or more ROI regions.The method is solved by the subject matter of the independent claims.Embodiments of the present invention provide a method for determining one or more relevant image areas. The method comprises the steps of obtaining a capturing sequence, e.g. from a 3D depth camera;• extraction of signals from the capture sequence to obtain a first signal sequence;• obtaining a second signal sequence;• performing a joint assessment of the first signal sequence and the second signal sequence; and• Determining a mask with the one or more relevant image areas based on the joint assessment.According to embodiments, the recording sequence can be a plurality of frames (over time), so that the first signal sequence and / or the second signal sequence then also have more signals over time. In the first signal sequence, according to exemplary embodiments, the signals can be generated from depth information of the recording sequence, so that these represent, for example, a movement, in particular a micromovement, based on breathing. For example, with this first signal sequence, the movement is observed at a pixel or a pixel region. According to embodiments, there are two different variants for the second signal sequence. According to a variant, the second signal sequence can also be obtained from the recording sequence. In this respect, the step of obtaining the second signal sequence comprises extraction of signals from the recording sequence. For example, in this variant, the signals of the first signal sequence then belong to a first pixel or image regions and the signals of the second signal sequence belong to second pixels or image regions. According to further exemplary embodiments, it would also be conceivable for the second signal sequence to be obtained on the basis of a reference signal. In this respect, the step of obtaining the second signal sequence is characterized by obtaining signals of a reference signal.These two signal sequences can be evaluated together, e.g., by some sort of cross-correlation. Therefore, according to embodiments, the joint evaluation may comprise a determination of Pearson correlation coefficients or Mutual information.Embodiments of the present invention are based on the finding that advantageously a plurality of signal sequences are obtained under one measurement data set, which signal sequences, when evaluated together, indicate one or more image regions which are connected to one another. Assuming that the one or more signal sequences both permit conclusions to be drawn about the movement to be observed and thus about the one or more relevant image regions, it is advantageously possible to identify the relevant image regions and thus to use them for further processing, such as analysis or diagnosis.The essential advantages are that the regions to be examined do not have to be predefined. That is, it does not have to define individual subareas by, for example, applying markers. The measuring method can be carried out parallel to spirometry and thus requires no additional time in practice. The most important advantage is that the proposed method is capable of objectively determining the breathing mechanics of different regions of the thorax. On the basis of this information, it is possible to define a region of interest which can record more precise measurement signals and thus even more precisely allows respiration parameters to be measured in a contactless manner. Furthermore, the information can be used to identify regions in diagnosis or therapy which, for example due to restrictive disorders, have only a smaller contribution to the breathing mechanics. Thus, restrictive pulmonary diseases such as pulmonary fibrosis or thorax deformities can potentially be identified earlier and more accurately.According to an embodiment, the mask may be a mask comprising a person or a torso of a person. According to exemplary embodiments, the mask contains one or more relevant regions in the recording sequence.As already explained above, the joint evaluation takes place by a type of correlation. Here, as already mentioned at the beginning, the determination of Pearson correlation coefficients or of Mutual information (common information) takes place in the first and in the second sequence. According to embodiments, the step of joint evaluation may comprise the substeps of determining a correlation matrix M. This can describe, for example, the correlation coefficient r for the individual signals s with respect to one another, wherein the correlation matrix M is described by the following formula:.The coordinates of a recording of the recording sequence are denoted by x and y (corresponding to the x and y dimensions of the chip of the camera or of the recording), wherein the counters i and j are selected. t is the time for the measurement time T. It should be noted that the formula (2) contains a thresholding ("if s x,y= 0 for any t", which occurs if s x,y belongs to the background at this time), wherein this is not absolutely necessary, so that the formula (1) results without optional thresholding.As already mentioned above in the introduction, a reference signal can be used. This can be, for example, a spirometer signal. The assessment using the reference signal then comprises, for example, generating the mutual information between the reference signal and the first sequence. For example, signal components with the same behavior are thus correlated with one another even if the signals can have a different character, that is to say a different type. In the evaluation, according to embodiments, a cross-correlation can be used, such that the step of evaluating comprises the substep of determining cross-correlation coefficients between the first signal sequence and the second signal sequence. The correlation matrix can be determined, for example, on the basis of the following formula:The coefficients x, y, s, t, T are as explained above. The general function f describes, for example, cross-correlation or mutual information. The combination of two different signal types can therefore be considered advantageous since relevant image regions are thus recognized in the first signal type (image signal), which also have a change in the second signal type. As mentioned above, if the second signal type represents a spirometer signal or the like which is associated close to the region to be observed, namely the respiratory muscles, the information concerning the relevance is transferred from the second signal type to the first signal type and thus very accurately identifies the relevant regions.In the situation mentioned (first signal sequence+reference signal) or else generally, the problem may arise that the first signal sequence and the second signal sequence are not synchronous. A possible constellation here is when a reference signal, such as a spirometer signal, is used as the second signal sequence. Therefore, the method may comprise the step of synchronizing the first signal sequence with the second signal sequence. Possible steps here are down sampling of the first and / or the second signal sequence onto the second and / or first signal sequence in order to adapt the sampling rates. Additionally or alternatively, the method may comprise the step of determining a time shift via cross-correlation between the first signal sequence and the second signal sequence.A further exemplary embodiment relates to the method explained above, wherein the method has the step of background detection, in particular background detection via a threshold value method of the distances.As mentioned at the beginning, the method explained above advantageously enables the determination of one or more relevant regions in an image or an image sequence, such as a 3D depth recording. This can be used for different purposes, for example also for a subsequent diagnostic method. In this respect, the method here represents a calibration step which can precede a diagnostic method but also another method, e.g. based on image recognition. According to exemplary embodiments, the explained method therefore precedes a recognition method, in particular a diagnostic method for extracting diagnostic-relevant information from the one or more image regions. Based on optoelectronic plethysmography, respiratory volumes and the partial contributions can advantageously be measured.According to exemplary embodiments, the method can also be computer-implemented. In this respect, an exemplary embodiment relates to a computer program for carrying out the method or individual steps of the method when the method runs on a computer.A further embodiment relates to an apparatus for determining one or more relevant image areas. The device comprises an interface and a processor. The interface is configured to receive a recording sequence. The processor is configured to extract signals from the capture sequence to obtain a first signal sequence and is configured to further obtain a second signal sequence. The processor is furthermore configured to perform a joint evaluation of the first signal sequence and the second signal sequence in order to determine a mask with the one or more relevant image regions.A further embodiment relates to a system comprising a device as just explained, and a spirometer or another sensor. According to a further exemplary embodiment, the system can have the device and a camera, in particular a 3D depth camera. Of course, according to further exemplary embodiments, it would also be conceivable for the system to comprise the device, the sensor and the camera.Embodiments of the present invention will be explained with reference to the accompanying drawings. The following are shown: FIG. 1 a shows a schematic block diagram of an optoelectronic plethysmography; FIGS. 1 b- 1 c are schematic representations for illustrating the measured values; FIG. 2 illustrates a selection of regions of interest in depth-based plethysmography; FIGS. 3 a- 3 d are schematic illustrations for illustrating the selection of a region of interest by level-set segmentation; FIG. 4 is a schematic illustration to illustrate the determination of the final region or interest; FIG. 5 is a schematic flow diagram with associated result representations for a method for selecting a region of interest by thresholding the averaged images followed by a selection of contiguous components in rows and columns; FIG. 6 is a schematic illustration to illustrate the selection of a region of interest in the face (selection of individual pixels by heart rate, which is generated by the signal of the corresponding pixels

[13] ; FIG. 7 shows a schematic flow diagram for illustrating a selection of a region of interest in the face via the information content of hierarchical voxels

[15] ; FIG. 8 shows a schematic flow diagram for explaining the determination of one or more relevant image regions according to exemplary embodiments; FIG. 9 shows a schematic flow diagram to illustrate the sequence of signal processing according to extended exemplary embodiments; FIG. 10 shows a schematic illustration of an individual patient mask generated information via Mutual according to exemplary embodiments; FIG. 11 shows a schematic illustration for defining a threshold value in order to determine the relevant contributions to the breathing mechanics in a histogram according to exemplary embodiments; and FIGS. 12 a- 12 b show schematic representations of patient masks of a region of interest (FIG. 12 a use of the patient mask as a region of interest for improved signal extraction, FIG. 12 b use of the patient mask for an identical assessment) for explaining exemplary embodiments.Before exemplary embodiments of the present invention are explained below with reference to the attached drawings, it should be noted that elements and structures having the same function are provided with the same reference numerals, so that the description thereof can be applied or interchanged with one another.FIG. 9 shows a schematic flow diagram of the method 100 for determining one or more relevant image regions, here collectively as a mask or individual patient mask. The method 100 has the four basic steps 110, 120, 130 and 140.In the first step 110 of the method 100, a recording, such as a 3D recording, is obtained. This is, for example, a recording sequence. In the subsequent step 110, this recording sequence is extracted with one or more signals in order to obtain a first signal sequence. The signal sequence can optionally be subjected to signal processing in the subsequent step 125. Furthermore, a second signal sequence is obtained either from the 3D recording at steps 110 and 120 or by a parallel-running method string with optional steps 160 and 165. It will be assumed from the following initial consideration that the second signal sequence is likewise obtained as part of the recording in step 110 and extracted in the signal extraction 120 in order to obtain the second signal sequence. The first and the second signal sequence are then evaluated in the subsequent metric 130 (i.e. thus after the optional step of the signal preprocessing 125). The evaluation aims to determine, for example, Mutual information, that is to say information associated with one another. In this respect, in step 130, a joint evaluation of the first and the second signal sequence takes place. According to exemplary embodiments, it can be determined here that a plurality of image regions, such as a plurality of pixels, behave similarly over time, that is to say over the recording sequence, with the result that these potentially relevant image regions offer. In the last step 140, a mask having one or more relevant image regions is determined on the basis of the preceding evaluation.According to an alternative variant, instead of the second signal sequence, a type of reference signal can be obtained obtained obtained from the recording (cf. step 110). This step is provided with the reference sign 160 and in principle runs parallel, i.e. particularly parallel in time, to the step 110. The reference signal can be, for example, a signal of a spirometer. This can be preprocessed analogously to the signal processing 125 by means of parallel signal processing 165, in order to then be supplied to the metric in step 130. In the metric 130, the joint evaluation takes place again.The method is explained below by way of example with reference to an example. For the method 100, the person to be measured sits on a chair at a distance of 1.5 m from a 3D depth camera (Orbecc Astra Pro, cf. step 110). The camera is oriented so that it is at the height of the chest. The test subject can additionally breathe through a spirometer (cf. step 160), also within a bodyplethysmograph. A computer processes (see steps 120, 125 and 165) the signals of the camera and the spirometer. The person breathes at rest and performs breathing maneuvers when required. The upper body is to be kept still. The test person should not speak in the measurement time.In step 120, signals are extracted from the images (see 110). In this case, a measurement signal s x,y[ t] is formed for each pixel (x, y). The measured value at time t corresponds to the pixel intensity of the image at point (x,y) at time t. In order to save computing effort and to avoid the effects of noise, the signals can be extracted exclusively at the locations (cf. 120) at which the test person is located. Optional background detection, e.g. via a threshold value method of distance, can be used here. This has the aim of saving computation time and increasing the signal quality.The result of the evaluation 130, i.e. the obtaining 140 of an individual patient mask MA, is shown, for example, in FIG. 10. The final metric of each signal is plotted at the position of the corresponding region and thus an individual patient mask is generated, as shown in FIG. 10 by way of example.FIG. 10 shows individual patient masks (generally mask MA) generated information of the individual pixel signals using the reference spirometer signal via Mutual (cf. step 160). For the mask, the metric (Mutual Information) is scaled to 100% of the maximum value that is obtained during the evaluation.FIG. 9 shows a sequence of optional signal processing (cf. step 125) if the signals extracted from the 3D recordings and the reference signals are not time-synchronous signals. First, down sampling 125d and 165d, respectively, is performed to adjust the sampling rate of the reference signal to the sampling rate of the extracted signals. Subsequently, a determination of the time shift is made via cross-correlation 125k or 165k. Starting from this, the two signals (first signal sequence and second signal sequence) can be synchronized (cf. steps 125 sand 165 s, respectively). It should be noted that the measurement signals s x,y and the reference signal of a spirometer y are present synchronously. For adaptation at different sampling rates, down sampling can be carried out. The time shift between both signals can be determined and compensated by cross-correlation. This optional operation is illustrated in Figure 9.The step of the metric (cf. 130) is explained in detail below. It should be noted that this embodiment is a possible embodiment, other embodiments or modifications being possible in accordance with further exemplary embodiments.The individual signals are then evaluated on the basis of statistical features or statistical parameters, such as the Pearson correlation coefficient or the Mutual Information. For this purpose, a correlation matrix M can be set up which describes the correlation coefficients r of the individual signals s with respect to one another. The coordinates of the image are x and y, with the counters i and j. If the corresponding pixel belongs to the rear or foreground for an arbitrary point in time t over the measurement time T, the value 0 is assigned for this pixel by the previous optional segmentation and the calculation can be shortened.Optionally, a reference signal y, e.g. via a spirometer, which is used in parallel for measurement by the test person, can be used. For this method, the statistical correlation of the signals with this reference signal is measured (generally the function f), for example by means of mutual information or cross-correlation coefficients or other statistical measures or statistical calculations. Generally, a function f (e.g., without thresholding here) is applied as follows: where y represents a reference signal and f may be a general function (e.g., mutual information or cross-correlation coefficients). The formula describes the calculation of the mask at the location x,y,.By means of further processing methods, the mask can be chosen, for example, as a new region of interest, which allows signals to be extracted only from the regions which are actually involved in the breathing process. This selection is made completely automatically by the method presented. One possibility for creating such an ROI is represented in FIG. 11 via a threshold value method, which only selects the contributions with the highest metric.An optional step of the metric (see step 130) is shown in FIG. 11, as to how the relevant regions for the mask are determined on the basis of this metric (see step 140). This is done by means of the threshold value method just addressed.FIG. 11 shows a histogram, here a so-called "kernel density estimation (KDE) histogram". This serves as a metric for determining the threshold value for the mutual information. A threshold value S serves to distinguish the relevant areas from the irrelevant areas. This threshold value can be determined automatically or defined in advance.Referring now to FIGS. 12 aand 12 b, the creation of the mask in step 140 is shown. If a new region of interest is created, a result is shown, for example, in FIG. 12 a. In addition, it is possible to represent a mask without further processing only on the basis of the color-coded metrics, as in FIG. 12 b. The proposed method makes anatomical structures visible, since their contribution to the respiratory dynamics is measured and displayed. This can potentially be used to detect diseases such as COPD and plan therapies.It should be noted at this point that even if it was assumed in the above exemplary embodiments that either the correlating image information items from different signals, for example at different positions in the image and / or different images, or the correlation of image information items with a further sensor signal was explained, the method described also fundamentally functions with other signal sequences. In this respect, the reference signal does not necessarily have to be a sensor signal of a respiration sensor, but can also use another sensor signal, such as a microphone signal. Since the recognition of a region of interest is relevant in particular for image sensors, this is a preferred exemplary embodiment. According to exemplary embodiments, the region of interest for medical applications (life science) can be carried out in order, for example, to calibrate a subsequent measurement method or diagnostic method accordingly. According to further exemplary embodiments, however, this concept is also possible for other image recognition methods, such as, for example, the observation of a person, for example during sports or during driving. Here, it would be conceivable, for example, for a heart rate or the like to be determined instead of the breathing signal via the exhalation air.Exemplary embodiments of the present invention are implemented in particular in software, wherein, in accordance with further exemplary embodiments, an associated apparatus is also created. This device has at least one interface for receiving the image information, optionally also an interface for receiving further data, such as the sensor data, e.g. the spirometer. Furthermore, the device comprises a processor which carries out the processing of the data and, as a result, outputs the mask for the region of interest. According to exemplary embodiments, this device or the processor can also be used for further purposes, for example for the subsequent analysis / diagnosis.Worldwide, 2017,544.9 million people suffered from chronic airway diseases. 3.9 million people died due to such disease

[16] . Death victims from the still persistent SARS-Cov-2 pandemia are estimated to be 6.9 million (Spring 2023)

[17] . Thus, chronic airway diseases are among the most common causes of death worldwide.The field of application of this invention extends in the field of pulmonary healing. Respiratory disorders can thus be diagnosed. In addition, a visual analysis of the dynamics of the upper body and of the breathing mechanism is possible. The contributions of individual regions to the respiratory volumes can be represented. Potentially interested companies are, for example: ResMed, Fisher & Paykel Healthcare, Vyaire Medical, MGC Diagnostics, Hamilton Medical, Ganshorn Medicine Electronic and Geratherm Respiratory GmbH. The aging population is a growth driver in the medical technology market so the research and development expenses in 2023 for the market-grade U.S. and Euopian medical technology companies are about 30 billion U.S. dollars

[18] .As explained above, some embodiments are implemented in software.Although some aspects have been described in connection with a device, it is understood that these aspects also represent a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in connection with or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps may be performed by a hardware apparatus (or using a hardware apparatus) such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the most important method steps may be carried out by such an apparatus.Depending on certain implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example a floppy disk, a DVD, a Blu-ray disk, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or another magnetic or optical memory, on which electronically readable control signals are stored, which can cooperate or cooperate with a programmable computer system such that the respective method is carried out. Therefore, the digital storage medium may be computer readable.Some embodiments according to the invention thus comprise a data carrier having electronically readable control signals which are capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.In general, embodiments of the present invention can be implemented as a computer program product having a program code, wherein the program code is operative to perform one of the methods when the computer program product runs on a computer.The program code can also be stored on a machine-readable carrier, for example.Other embodiments include the computer program for performing any of the methods described herein, wherein the computer program is stored on a machine readable carrier. In other words, an exemplary embodiment of the method according to the invention is thus a computer program which has a program code for carrying out one of the methods described herein when the computer program runs on a computer.A further embodiment of the methods according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded.A further exemplary embodiment of the method according to the invention is thus a data stream or a sequence of signals which represents or represent the computer program for carrying out one of the methods described herein. The data stream or sequence of signals may be configured, for example, to be transferred over a data communication link, for example, over the Internet.A further embodiment comprises a processing device, for example a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.A further embodiment comprises a computer on which the computer program for carrying out one of the methods described herein is installed.A further embodiment according to the invention comprises an apparatus or a system which is designed to transmit a computer program for carrying out at least one of the methods described herein to a receiver. The transmission can be carried out electronically or optically, for example. The receiver may be, for example, a computer, a mobile device, a storage device, or similar device. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the recipient.In some embodiments, a programmable logic device (e.g., a field programmable gate array, an FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be universally usable hardware such as a computer processor (CPU) or hardware specific to the method, such as an ASIC.The above-described embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented herein by way of description and explanation of the embodiments.Literature: Literature1. Health Report 2022. Analysis of Incapability Data: Risk Psycho: Such as depressions, anxiety and stress burden the heart; Storm, A.; Schumann, M.; Marsonic, J.; Hildebrandt-Heene, S.; Nolating, H.-D., Eds.; medhochzwei: Heidelberg, 2022, ISBN 978-3-86216-919-1. 2. buless, C.; Schlegel milk, R. M.; Kramme, R. Pulmonary Function Diagnostics. In Medical Technology: Method - Systems - Information Processing, 5, Fully Revised and Extended Edition; Kramme, R., Ed.; Springer: Berlin, 2017, pp 159-180, ISBN 3662487713. 3. Cowteney, R.; van Dixhoorn, J.; Cohen, M. Evaluation of breeding pattern: comparison of a Manual Assessment of Respiratory Motion (MARM) and respiratory induction plethysmography. Appl. Psychophysiol. Biofeedback 2008, 33, 91-100, doi:10.1007 / s 10484-008-9052-3. 4. Retory, Y.; Niedzialkowski, P.; Picciotto, C. de; Bonay, M.; Petitjean, M. New Respiratory Inductive Party Lithography (RIP) Method for Evaluating Ventilatory Adaptation during Mild Physical Activities. PLoS ONE 2016, 11, doi:10.1371 / journal.pon.0151983. 5. massaroni, C.; Carraro, E.; Vianello, A.; Miccinilli, S.; Morrone, M.; Levai, I.K.; Schena, E.; Saccomandi, P.; Sterzi, S.; Dickinson, J.W.; et al. Optoelectronic Plethysmography in Clinical Practice and Research: A Review. Respiration 2017, 93, 339-354, doi:10.1159 / 000462916. 6. Ripka, W. L.; Ulbricht, L.; Geschahn, P. M. Application of a photogrametric kinetic model for prediction of lung volumes in adolescents: A pilot study. BioMed Engineering Online 2014, 13, doi:10.1186 / 1475-925X-13-21. 7. feitosa, L.; Britto, M. de; Alieverti, A.; Noronha, J. B.; Andrade, A. D. de. Accuracy of optoelectronic plethysmography in childhood exercis-induced asthma. Journal of Asthma 2019, 56, 61-68, doi:10.1080 / 02770903.2018.1424196. 8. Zoumot, Z.; Lomauro, A.; Alieverti, A.; Nelson, C.; Ward, S.; Jordan, S.; Polkey, M.I.; Shah, P.L.; Hopkinson, N.S. Lung Volume Reduction in Emphysema Improves Chest Wall Asynchrony. Chest 2015, 148, 185-195, doi:10.1378 / chest.14-2380. 9. Oh, K.; Shin, C. S.; Kim, J.; Yoo, S. K. Level-Set Segmentation-Based Respiratory Volume Estimation Using a Depth Camera. IEEE Journal of Biomedical and Health Information 2019, 23, 1674-1682, doi:10.1109 / JBHI.2018.2870859. 10. imano, W.; Kameyama, K.; Hollingdal, M.; Refsgaard, J.; Larsen, K.; Topp, C.; Kronborg, S. H.; Gade, J. D.; Dinesen, B. Non-Contact Respiratory Measurement Using a Depth Camera for Electrically People. Sensor (Basle) 2020, 20, doi: 10.3390 / s20236901. 11. Eastadabas, S.; Sebkhi, N.; Zhang, M.; Rahim, S.; Anderson, L.J.; Lee, F.-H.; Ghovanloo, M.A. Vision-Based Respiration Monitoring System for Passive Airway Resistance Estimation. IEEE Transactions on Biomedical Engineering 2016, 63, 1904-1913, doi:10.1109 / TBME.2015.2505732. 12. calvo-Gallego, E.; Haan, G. de. Automatic ROI for Remote Photoplethysmography using PPG and Color Features. Proceedings of the 10th International Conference on Computer Vision Theory and Applications. International Conference on Computer Vision Theory and Applications, Berlin, Germany, 11-14 Mar. 2015; SCITEPRESS-Science and Technology Publications, 2015; pp 357-364, ISBN 978-989-758-089-5. 13.Gibert, G.; D'Alessandro, D.; Lancet, F. Face detection method based on photoplethysmography. In 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance. 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Krkow, Poland, 27-30 Aug. 2013; IEEE, 2013; pp 449-453, ISBN 978-1-4799-0703-8. 14. Kwon, S.; Kim, J.; Lee, D.; Park, K. ROI analysis for remote photoplethysmography on facial video. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2015, 2015, 4938-4941, doi:10.1109 / embc.2015.7319499. 15. Wang, W.; Stuijk, S.; Haan, G. de. Unsupported Subject Detection via Remote PPG. IEEE Trans. Biomed. Eng. 2015, 62, 2629-2637, doi:10.1109 / TBME.2015.2438321. 16. Soriano, J. B.; Kendrick, P. J.; Paulson, K. R.; Gupta, V.; Abrams, E. M.; Adeoyin, R. A.; Adhikari, T. B.; Advani, S. M.; Agrawal, A.; Ahmadian, E.; et al. Prevalence and attributeable health burds of chronic respiratory diseases, 1990-2017: a systematic analysis for the Global Burds of Disease Study 2017. The Lancet Respiratory Medicine 2020, 8, 585-596, doi:10.1016 / s2213-2600(20)30105-3. 17. world Health Organization. World health statistics 2023: monitoring health for the SDGs, Geneva, 2023. Available online: https: / / apps.who.int / iris / rest / bitstreams / 1502907 / retrieve (accessed on 28 January 2023). 18. Statista. Research and development expenses of U.S. and European Medical Engineering Companies in 2009-2022 (in Billion U.S. Dollar): Chart. Available online: https: / / de.statista.com / statistics / data / study / 203486 / survey / issue-fuel-research-and-development-American-and-European-medical-technology company / (accessed on 14 November 2023).References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureResMed, Fisher & Paykel Healthcare, Vyaire Medical, MGC Diagnostics, Hamilton Medical, Ganshorn Medicine Electronics and Geratherm Respiratory GmbH

[0061] Storm, A.; Schumann, M.; Marschall, J.; Hildebrandt-Heene, S.; Nolating, H.-D., Eds.; medhochzwei: Heidelberg, 2022, ISBN 978-3-86216-919-1

[0075] Buess, C.; Schlegel milk, R. M.; Kramme, R. Pulmonary Function Diagnostics. In Medical Engineering: Method - Systems - Information Processing, 5, Fully Revised and Extended Edition; Kramme, R., Ed.; Springer: Berlin, 2017, pp 159-180, ISBN 3662487713

[0075] Cowney, R.; van Dixhoorn, J.; Cohen, M. Evaluation of breeding pattern: comparison of a Manual Assessment of Respiratory Motion (MARM) and respiratory induction plethysmography. Appl. Psychophysiol. Biofeedback 2008, 33, 91-100, doi:10.1007 / s 10484-008-9052-3

[0075] Retort, Y.; Niedzialkowski, P.; Picciotto, C. de; Bonay, M.; Petitjean, M. New Respiratory Inductive Party Lithography (RIP) Method for Evaluating Ventilatory Adaptation during Mild Physical Activities. PLoS ONE 2016, 11, doi:10.1371 / journal.pon.0151983

[0075] Massaroni, C.; Carraro, E.; Vianello, A.; Miccinilli, S.; Morrone, M.; Levai, I.K.; Schena, E.; Saccomandi, P.; Sterzi, S.; Dickinson, J.W.; et al. Optoelectronic Plethysmography in Clinical Practice and Research: A Review. Respiration 2017, 93, 339-354, doi:10.1159 / 000462916

[0075] Ripka, W. L.; Ulbricht, L.; Gun, P. M. Application of a photogramtric kinetic model for prediction of lung volumes in adolescents: A pilot study. BioMed Engineering Online 2014, 13, doi:10.1186 / 1475-925X-13-21

[0075] Feitosa, L.; Britto, M. de; Alieverti, A.; Noronha, J. B.; Andrade, A. D. de. Accuracy of optoelectronic plethysmography in childhood exercis-induced asthma. Journal of Asthma 2019, 56, 61-68, doi:10.1080 / 02770903.2018.1424196

[0075] Zoumot, Z.; Lomauro, A.; Alieverti, A.; Nelson, C.; Ward, S.; Jordan, S.; Polkey, M. I.; Shah, P. L.; Hopkinson, N. S. Lung Volume Reduction in Emphysema Improves Chest Wall Asyncy. Chest 2015, 148, 185-195, doi:10.1378 / chest.14-2380

[0075] Oh, K.; Shin, C. S.; Kim, J.; Yoo, S. K. Level-Set Segmentation-Based Respiratory Volume Estimation Using a Depth Camera. IEEE Journal of Biomedical and Health Information 2019, 23, 1674-1682, doi:10.1109 / JBHI.2018.2870859

[0075] Imano, W.; Kameyama, K.; Hollingdal, M.; Refsgaard, J.; Larsen, K.; Topp, C.; Kronborg, S. H.; Gade, J. D.; Dinesen, B. Non-Contact Respiratory Measurement Using a Depth Camera for Electrically People. Sensor (Basle) 2020, 20, doi: 10.3390 / s20236901

[0075] Eastadabas, S.; Sebkhi, N.; Zhang, M.; Rahim, S.; Anderson, L. J.; Lee, F. -H.; Ghovanloo, M. A Vision-Based Respiration Monitoring System for Passive Airway Resistance Estimation. IEEE Transactions on Biomedical Engineering 2016, 63, 1904-1913, doi:10.1109 / TBME.2015.2505732

[0075] Calvo-Gallego, E.; Haan, G. de. Automatic ROI for Remote Photoplethysmography using PPG and Color Features. Proceedings of the 10th International Conference on Computer Vision Theory and Applications. International Conference on Computer Vision Theory and Applications, Berlin, Germany, 11-14 Mar. 2015; SCITEPRESS-Science and Technology Publications, 2015; pp 357-364, ISBN 978-989-758-089-5

[0075] Gibert, G.; D'Alessandro, D.; Lancet, F. Face detection method based on photoplethysmography. In 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance. 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Krkow, Poland, 27-30 Aug. 2013; IEEE, 2013; pp 449-453, ISBN 978-1-4799-0703-8

[0075] Kwon, S.; Kim, J.; Lee, D.; Park, K. ROI analysis for remote photoplethysmography on facial video. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2015, 2015, 4938-4941, doi:10.1109 / embc.2015.7319499

[0075] Wang, W.; Stuijk, S.; Haan, G. de. Unsupported Subject Detection via Remote PPG. IEEE Trans. Biomed. Eng. 2015, 62, 2629-2637, doi:10.1109 / TBME.2015.2438321

[0075] Soriano, J. B.; Kendrick, P. J.; Paulson, K. R.; Gupta, V.; Abrams, E. M.; Adeoyin, R. A.; Adhikari, T. B.; Advani, S. M.; Agrawal, A.; Ahmadian, E.; et al. Prevalence and attributeable health burds of chronic respiratory diseases, 1990-2017: a systematic analysis for the Global Burds of Disease Study 2017. The Lancet Respiratory Medicine 2020, 8, 585-596, doi:10.1016 / s2213-2600(20)30105-3

[0075]

Claims

Method (100) for determining one or more relevant image areas, comprising the following steps: obtaining (110) a recording sequence; extracting (120) signals from the recording sequence to obtain a first signal sequence; obtaining a second signal sequence; carrying out a joint assessment (130) of the first signal sequence and the second signal sequence; and determining (140) a mask (MA) with the one or more relevant image areas based on the joint assessment (130).The method (100) of claim 1, wherein the step of obtaining the second signal sequence is characterized by obtaining (160) a reference signal.The method (100) of claim 1, wherein the step of obtaining the second signal sequence is performed by extraction (120) of signals from the capture sequence; and / or wherein the step of obtaining the second signal sequence is performed by extraction (120) of signals from the capture sequence, wherein signals of the first signal sequence belong to a first pixel or image area and the signals of the second signal sequence belong to second pixels or image areas.The method (100) according to any one of the preceding claims, wherein the mask (M) comprises a person or a torso of a person or the mask (M) includes one or more relevant areas in the capturing sequence.Method (100) according to one of the preceding claims, wherein the joint evaluation (130) comprises the determination of statistical parameters, in particular Pearson correlation coefficients and / or Mutual information.Method (100) according to one of the preceding claims, wherein the step of performing a common evaluation (130) comprises the substeps of determining a correlation matrix M or wherein the step of performing the common evaluation (130) comprises the substep of determining a correlation matrix M which describes the correlation coefficients r for the individual signals s with respect to one another, wherein the correlation matrix M is described by the following formula: M [ x, y ] = ∑ i ∑ j r s x, y, s i, j wherein coordinates of a recording of the recording sequence are characterized by x and y with the counters i and j, wherein t represents a time point over the measurement time T.The method (100) of any preceding claim with reference to claim 2, wherein the reference signal comprises a spirometer signal.Method (100) according to one of the preceding claims, wherein in the evaluation (130), in particular in the evaluation (130) using a reference signal, the generation of a mutual information and / or the determination of cross-correlation coefficients between the first signal sequence and the second signal sequence or other statistical calculations takes place, wherein the mask is determined on the basis of the following formula: M [ x, y ] = f ( s x, y, y ) The method (100) of any preceding claim, wherein the method (100) comprises the step of synchronizing the first signal sequence with the second signal sequence; and / or wherein the method (100) comprises the step of down sampling the first or second signal sequence onto the second or first signal sequence to adjust the sampling rates; and / or wherein the method (100) comprises the step of determining a time shift via a statistical method or cross correlation between the first signal sequence and the second signal sequence.The method (100) of any preceding claim, wherein the acquisition sequence represents the acquisition sequence associated with a 3D depth acquisition.Method (100) according to one of the preceding claims, wherein the method (100) has the step of background detection, in particular background detection via a threshold value method of the distances.Method (100) according to one of the preceding claims, wherein the method (100) for determining one or more relevant image areas precedes a diagnostic method for extracting (120) diagnostic relevant information from the one or more image areas.Computer program for carrying out the method (100) according to one of the preceding claims, when the method (100) runs on a computer.Device for determining one or more relevant image areas, having the following features: an interface for obtaining a recording sequence; a processor which is designed to extract signals from the recording sequence in order to obtain a first signal sequence and is designed to obtain a second signal sequence; wherein the processor is designed to carry out a joint evaluation (130) of the first signal sequence and the second signal sequence in order to determine a mask (M) with the one or more relevant image areas.System comprising a device according to claim 14 and a spirometer and / or a camera, in particular a 3D depth camera.

Images