Method for automatically locating specific tissue type during endoscopic intervention and associated image recording system

Abstract

The invention proposes a new technical solution by means of which a specific tissue type (6) within the surgical region (9) can be automatically positioned within the captured image (3) by means of the image capture system (1). For this purpose, the image ranges (4) within the image (3) showing the desired tissue type (6) are identified and selected by means of two temporally successive selection steps A and B. In this case, in the framework of the first selection step A, not only the image range (5) is spatially delimited, but also the spectral information to be processed and / or the pixels to be processed of the respective image (3) are optionally reduced. Then, only in a second selection step B, the target tissue type (6) that is actually to be determined is identified within the image (3) and is selected as a second image range (4), whereby the user can finally find the tissue type (6) within the region (9) observed with the image recording system (1).

Description

Technical Field

[0001] The invention relates to a method for automatically locating a specific target tissue type within at least one image, which is recorded with a (particularly medical) image recording system. This can occur, for example, during an endoscopic intervention performed with an endoscope as part of the image recording system. The method according to the invention should simplify or generally allow, first of all, to distinguish the target tissue type from other types of tissue within the image with a higher degree of reliability and thus to locate the target tissue type within the surgical field (being observed with the image recording system). Background Art

[0002] It is already known in the prior art to make specific tissue types to be identified (e.g. tumors) visible with the help of fluorescent colorants, so that cancerous tissue in live video images can be recognized by the surgeon by corresponding color marking. In addition, the surgeon also uses the fact that the mechanical strength of cancerous tissue is different from that of healthy tissue to find and locate the cancerous tissue by touch. However, in these methods, the doctor often relies on his experience and anatomical knowledge to locate a specific tissue type within the surgical field with the help of an endoscope.

[0003] Nervous tissues are critical structures in the human body, where damage to nerves during surgery can lead to different consequences, including chronic pain or even loss of function of different organic structures, with a consequent loss of quality of life for those affected.

[0004] Especially when performing surgery in the head and neck area, careful identification of nerves is essential in order to avoid nerve damage. However, currently, nerves are identified by surgeons solely on the basis of anatomy and appearance, possibly in combination with electromyography (EMG), wherein in practice, nerve damage is often discovered after the operation has been completed. Moreover, there are already scientific publications, such as the article "In Vivo Nerve Identification in Head and Neck Surgery Using Diffuse Reflectance Spectroscopy" published by Lanhout et al. in the specialist journal "Laryngoscope Investigative Otolaryngology" in 2018, which propose optical methods such as diffuse reflectance spectroscopy (DRS) and fluorescence spectroscopy (FS) for the targeted identification of nerves.

[0005] Therefore, in order to locate peripheral nerves, different technical methods have been followed in the prior art: One of these methods is the method of autofluorescence (AF), in which use is made of the fact that different tissue types show fluorescence of different intensities when excited with a suitable wavelength. In this case, the method of autofluorescence is based on the fact that natural fluorophores (such as fat pigments, collagen or elastin) are present in the human body, which can be excited to autofluorescence when illuminated accordingly with UV light. However, this method therefore not only makes nerves visible, but also other tissue types that show autofluorescence properties, such as ligaments or fat tissue. In medical applications, however, nerves must be clearly distinguished from these other tissue types in order to prevent nerve damage during surgery. Therefore, in the method of autofluorescence, a decisive disadvantage is that not only nerve tissue, but also other tissue types show autofluorescence properties, so that they are also visible in autofluorescence imaging. Therefore, the described method of autofluorescence is not specific enough to detect and locate nerve tissue exclusively and selectively.

[0006] On the other hand, doctors often use acquired image impressions, experience and anatomical knowledge in the endoscopic localization of nerves. However, these impressions are all subjective, and it is the anatomy that can show differences between different patients. For these reasons as well, various technologies are being tested in order to be able to locate nerve tissue more reliably and more reproducibly during endoscopic interventions and thus avoid damage to the nerves during the intervention.

[0007] As mentioned, another possible method that can be used for nerve detection is the method of diffuse reflectance spectroscopy (DRS). In this method, reflection spectra with a high wavelength resolution of, for example, 1 nm are typically used in order to distinguish different tissue types from each other based on their "spectral fingerprints". However, the corresponding measurement probe is placed directly on the tissue or even penetrated into the tissue. Therefore, in DRS, the probe must be in direct contact with the tissue and accordingly only provides information at specific positions in the instantaneous visible range. However, this is not feasible for endoscopic applications, especially when larger surgical areas must be examined. In other words, it is not possible to locate a specific tissue type within the entire field of view of the endoscope using this method. Summary of the invention

[0008] Starting from this background, the object of the present invention is to provide an improved concept with which specific tissue types, in particular neural tissue, can be localized within a surgical area that is being observed, for example, with an endoscope.

[0009] In order to achieve this object, according to the invention, the features of claim 1 are provided in a method. In particular, according to the invention, in order to achieve this object, it is proposed in a method of the type mentioned at the outset that in a first selection step, at least one first image range is automatically identified and selected within the at least one image by means of a first imaging method. In this case, in addition to displaying the target tissue type, the at least first image range also displays at least one other secondary tissue type that is different from the target tissue type. In other words, the selectivity of this first preselection is not yet sufficient to locate the target tissue type in the image with sufficient accuracy. However, the result of the first selection step can already be an image with a reduced amount of information.

[0010] Therefore, according to the invention, in a second selection step, at least one second image range (as a sub-range) is identified and selected within the at least one first image range that has been preselected by means of a second imaging method that is different from the first imaging method (which can be, as described for the first imaging method, preferably an optical method, respectively). Here, the at least one second image range then mainly or even exclusively displays the target tissue type to be determined. In other words, the target tissue type can thereby be located sufficiently selectively within the surgical area based on the second image range. Therefore, the result of the second selection step can be an image with a further reduced amount of information. Therefore, the image obtained in this way can in particular be a spectral image with the desired information, i.e. the relative position and extension of the target tissue type within the original at least one image.

[0011] The method can therefore comprise two image processing steps which follow one another in time, namely a first selection step and a second selection step.

[0012] In particular, in the subsequent third method step, the at least one second image region thus identified can be highlighted for the user in the at least one image or in an image corresponding thereto, for example a VIS image (which can be, for example, a still image or, for example, a changing live image). This can be achieved by means of a specific false color or, for example, by means of a so-called "image overlay".

[0013] This method allows for pixel-precise and automatic (without any input from the user) conclusions to be drawn as to whether the respective pixel shows the target tissue type. In a specific application, it is possible for the at least one second image range to show a lower resolution than the image on which it is based, which is observed by the operating doctor and / or is being recorded by the image recording system (which can be, for example, a VIS live image or an IR live image).

[0014] Furthermore, the method according to the invention can in principle be used not only in open medical interventions, but also in endoscopic interventions inside the body. The image recording system used in the method can include, for example, an endoscope, a microscope or a macroscope, since it is not important in which way the image data are optically detected.

[0015] Thus, the method according to the invention can be used to identify specific tissue types quickly (in particular in real time based on the live video image data stream), with high reliability and with spatial resolution within an image recorded by an image recording system, in particular within a live video image data stream. The main advantage in this regard in application is improved surgical guidance: In this way, in particular damage to nerve tissue can be prevented, so that collateral damage can be effectively prevented. In addition, the method can help the doctor make decisions in specific situations during surgery.

[0016] In particular, when locating peripheral nerves during endoscopic surgery, the surgeon can thus be assisted by the use of the technology. In this regard, the method according to the invention takes into account the specific boundary conditions that exist during endoscopic interventions, in particular the need for contactless identification of the tissue type to be located. The method enables contactless and reliable localization of nerves in the entire current and changing field of view of the endoscope.

[0017] As described above, the present invention provides for at least two different imaging methods (which can also be understood as "measurement methods") or a combination of measurement methods in order to identify a specific tissue type, such as nerve tissue, within the image being recorded and to locate the specific tissue type within the image. As mentioned above, this approach is particularly advantageous for locating nerves of the peripheral nervous system.

[0018] For example, in a first method step, it is possible to first classify in the entire image by means of autofluorescence which first image areas show tissue types with a high autofluorescence activity. Subsequently, in a second method step, those second image areas which show / contain the type of tissue to be identified (target tissue type) can be identified and located within the first image areas identified in this way. In this second method step, the following imaging methods can be used, for example: (a) fluorescence using externally introduced fluorophores; (b) polarization analysis; (c) laser speckle contrast imaging (LSCI); and (d) color ratio methods.

[0019] The combination of different imaging techniques for localizing specific tissue types is a novel approach that enables, in particular, a specific differentiation between neural tissue and other tissue types, as they are observed in typical surgical situations, in particular during endoscopic interventions. In addition, it is advantageous that localization can be achieved in the entire field of view of the endoscope and that no contact with the tissue is necessary for this purpose.

[0020] A second possible application of the invention is the location and display of blood vessels. In many surgeries, it is extremely disruptive if blood vessels are damaged and the bleeding must first be stopped before the operation can continue. What can be disruptive here is in particular the fouling of the endoscope optics with blood. If the second selection step is designed for the identification of blood vessels, this unintentional damage can also be avoided with the method according to the invention.

[0021] In contrast to the DRS method mentioned at the outset, the method according to the invention, in particular within the framework of the second method step, can provide that only a small number of spectrally narrowly defined wavelength ranges (i.e., specific narrowly defined color ranges / colors) are used / analyzed in order to distinguish different tissue types from one another. Thus, it is not necessary to record a complete continuous spectrum with high resolution, which is usually only possible with the aid of a complex and difficult to miniaturize spectrometer. Thus, in particular, the second method step can be carried out without the use of a spectrometer, as will be explained in more detail.

[0022] Since the method according to the invention can be carried out contactlessly, the used measuring optical system, which can be integrated into the endoscope used in the method, does not have to be placed directly on the tissue, but can be kept at a distance from the tissue during the measurement, as is the case in typical endoscopic interventions. This has the advantage that the corresponding positioning of the nerve can be carried out in the entire (active) field of view of the endoscope / microscope / macroscope.

[0023] The target tissue type to be determined can be understood as a primary tissue type in the focus of attention. Here, the second image range may have excluded (i.e., just not displayed or only displayed in a negligible range) at least one secondary tissue type (which may belong to a group of tissues to which the target tissue type also belongs). In addition, the first image range may exclude (i.e., just not displayed or only displayed in a negligible range) at least another tertiary tissue type present / displayed in the entire image. Conversely, both the primary tissue type and at least one secondary tissue type may be displayed / present in the at least one first image range.

[0024] In a typical application, in an image captured by an image capture system, for example, arteries, veins, adipose tissue, muscle, nerve tissue, liver or intestine can be seen / displayed as different tissue types, but objects outside the body, such as metal objects (such as surgical instruments) can also be seen / displayed. Therefore, with the help of a first (pre) selection step, spatial pre-filtering can be performed within the image being captured and thus those image ranges that show the target tissue type to be located and may also show other tissue types are identified, and the other tissue types are optically at least similar to the target tissue type to be located, that is, for example, provide autofluorescence signals like the target tissue type to be located. Therefore, the first image range can display, for example, a specific group of tissue types (this can be, for example, a specific subgroup of a specific tissue type). In addition, the first image range may have excluded at least one tissue type that is present / displayed in the (entire) image (for example, a tissue type that does not provide an autofluorescence signal).

[0025] The described method can provide the surgeon with position information based on the second selection step, i.e. at least one second image range identified in the recorded image, in particular in the current live image. This position information enables the surgeon to plan and / or execute subsequent treatment method steps. However, the method according to the invention itself is not aimed at the execution of this treatment step, but only at the provision of information for preparing for subsequent treatment and / or diagnosis, such as a specific tissue resection that is optically observed with an endoscope and performed with a separate surgical tool. Therefore, in the method according to the invention, it can be provided in particular that no treatment or diagnostic step is carried out, but only information for planning and preparing for subsequent treatment and / or diagnostic steps is provided.

[0026] In this case, the endoscope mentioned can be part of an image recording system, wherein the latter can also include, for example, a monitor and / or a camera control unit.

[0027] The captured image may be a single image (a “still image”) or, for example, part of a live video image data stream that is or was continuously recorded by the image capture system.

[0028] The first measurement method can be, for example, imaging based on diffuse reflectance spectroscopy (DRS) or, for example, imaging by means of autofluorescence (AF). In the latter AF method, the fluorophores naturally present in human tissue are excited by means of (typically invisible) excitation light (for example in the UV wavelength range) into light radiation, which can then be detected by a sensor and is typically at 500 nm. After evaluating this autofluorescence signal recorded by means of the image recording system, the first image range selected in this way can, for example, only show fat tissue and nerve tissue. In this case, the excitation light can be excluded from the imaging by means of an excitation light filter.

[0029] As mentioned, the method according to the invention can in particular use the so-called measured color ratios (English color ratios) in order to distinguish different tissue types from each other. Here, such a color ratio can be understood as, for example, the ratio of the corresponding intensity of an image point to two different wavelength ranges (especially with, for example, a corresponding bandwidth of less than 200 nm, preferably less than 100 nm and / or at least 10 nm). In order to determine such a wavelength ratio (a synonym for this is colorratios / color ratio), firstly corresponding measured intensity values ​​from the various wavelength ranges (which may partially overlap in some cases) are required. Such spectrally selected intensity values ​​can be obtained in two basic ways:

[0030] (i) On the one hand, wavelength sensitivity can be implemented in the camera of the system. According to this approach, for example, hyperspectral sensors can be used, wherein there are different technologies for implementing such sensors. In addition, so-called "snapshot sensors" that only provide a small number (for example 16) of different color channels are also suitable for the approach according to the invention. This is because a very high spectral resolution is generally not required in the approach according to the invention. However, in the image recording system according to the invention, it is also possible to realize the detection of different wavelength ranges, for example with the help of beam splitters and wavelength-selective (for example dichroic) layers; in this case, conventional color and / or monochrome image sensors can be used. In this way, a spatial separation of the different wavelengths can be achieved, so that two different spectral wavelength ranges can be directed to different (spatially separated) image sensors of the system and detected separately there by the sensors.

[0031] (ii) On the other hand, the wavelength sensitivity (in addition to or as an alternative to (i)) can also be based on a (temporally modulatable) light source used with the image recording system during imaging. For example, only one specific wavelength range can always be emitted from the light source in a temporal sequence. This temporally varying spectral illumination can thus lead to a temporal separation of different wavelength ranges, so that - even when using only a single image sensor - different spectral wavelength ranges can be detected separately by the sensor at different points in time.

[0032] With all these methods it is therefore possible to determine the required intensity values ​​in different wavelength ranges which are necessary for calculating the color ratios.

[0033] However, unlike in the conventional scheme of hyperspectral imaging, it is not necessary to detect the complex color fingerprint of a specific tissue type with high spectral resolution. More precisely, according to the present invention, it is possible and more effective to first determine how a specific color ratio (color ratios) is manifested in a specific tissue type based on experience. Subsequently, when performing the method according to the present invention, the current tissue type can be directly, reliably and quickly inferred more based on the color ratio of this measurement (and in particular based on this empirical data), without having to detect and evaluate complex spectral measurement data in a broadband manner (for example, in the entire visible range + subsequent non-visible wavelength range). Here, the reduction in complexity is achieved by selectively selecting the corresponding spectral range considered when calculating the corresponding color ratio. This selection depends on the tissue type to be determined and can be determined by preliminary experiments.

[0034] Therefore, in the described method using color ratios, the challenge is to identify suitable (as narrow as possible) spectral ranges that should be used to distinguish between different tissue types, that is, to select specific color ratios that should be calculated in a corresponding selection step. These spectral ranges must be able to localize a specific tissue type in their entirety, without individual differences in the tissue (between different patients) or other interfering influences having a negative impact on the localization. However, due to the very large differences between the numerous tissue types present in humans, the main challenge is to empirically identify a sufficiently large number of suitable spectral ranges. However, this can be achieved by preliminary experiments and then taken into account when designing the image acquisition system.

[0035] The invention also offers important advantages here, since the solution space is already greatly reduced by the first method step / selection step, so that in the second step, the desired specific differentiation of tissue types can only be achieved by means of optical methods (e.g. DRS) and / or based on measured color ratios. By spatial pre-filtering, for example based on AF, in the first selection step, the image range to be differentiated can be limited in such a way that the DRS method or another imaging or measurement method (e.g. the determination of the color ratios mentioned) - even with only a low spectral resolution - provides usable results in the second selection step.

[0036] Accordingly, this object can also be achieved by further advantageous embodiments according to the dependent claims.

[0037] The first selection step and / or the second selection step can thus be performed automatically by the image recording system, in particular based on an image evaluation of the at least one image. Image evaluation can also be understood, for example, as a pixel-based calculation of color ratios for individual image regions.

[0038] Furthermore, the first and second selection steps may be performed continuously in real time.

[0039] Furthermore, the target tissue type can be visually highlighted, for example in a live video image data stream. This makes it possible for the user to locate the target tissue type particularly easily within the area being viewed with the image recording system.

[0040] Furthermore, it is advantageous for the robust detection of the target tissue type if the first selection step and / or the second selection step are performed based on spectral partial information separated spatially or temporally from the entire spectral information of the at least one image. In other words, the image processing on which the first and / or the second selection step is based is applied only to a part of the entire spectral information with respect to the entire spectral information provided by the corresponding image. Thus, in particular, the at least one image can, for example, cover the entire visible wavelength spectrum (and can therefore be a VIS image), while the first selection step and / or the second selection step are applied to the following spectral image, which, although showing the same scene / same image part as the at least one VIS image (in which the target tissue type should ultimately be displayed to the user), reproduces only a part of the entire spectral information or information different therefrom. Thus, for example, the first and / or the second selection step can be applied to infrared images, in particular fluorescence images, whose spectral information is outside the visible range. After selecting the second image range within the spectral image, these second image ranges can be displayed to the user in the VIS image.

[0041] Here, in particular, if the amount of image information (e.g., the amount of image pixels to be processed) and / or the amount of spectral information (e.g., the amount of color channels to be evaluated / per pixel) processed in the second selection step is reduced by means of the first selection step, the identification and selection of the second image range by means of the second selection step can be reliably and quickly achieved. Thus, by means of the first selection step A, the solution space of possible tissue types to be distinguished in step B can be reduced. The fewer tissue types to be distinguished, the less information in the form of wavelength ranges is required for this in the second selection step B. In practice, there are also cases in which the differentiation of tissue types can only be achieved in this way according to the solution of the present invention (because it is usually impossible to distinguish with the available hardware by means of a single selection step). In the image acquisition system itself, this means that fewer wavelength ranges must be recorded / detected by the sensor. This can, for example, lead to a reduction in the number of wavelength ranges that are (originally) required in time to robustly distinguish the target tissue, which are generated by means of a modulated light source, or, for example, a small number of color channels provided by the system's hyperspectral sensor is sufficient to reliably arrange tissue identification. In other words, the solution according to the present invention can significantly reduce the requirements on the system's hardware and thus open up new fields of application. With the solution according to the invention, it may be sufficient to read out, for example, only 16 color channels / spectral bands of an image sensor, instead of 100 different channels as in known solutions requiring high spectral resolution. This also shows a specific technical advantage of the invention.

[0042] Therefore, from another perspective, the method according to the invention can be designed so that only a limited spectral range is evaluated in the first selection step and / or in the second selection step. Here, this limited spectral range can be different from, in particular smaller than, or not overlap with the (in particular wider) spectral range (e.g. the entire visible range) that is ultimately displayed to the user in the at least one image.

[0043] According to a possible implementation of the method of the invention, it is provided that, within the framework of the second selection step, a so-called color ratio is calculated for different image points of the at least one image based on at least two different color channels of the image recording system. Here, the corresponding color ratio gives the intensity ratio of at least two spectral ranges for the corresponding image point or image part. It is preferred that the corresponding spectral range based on which the color ratio is determined has a minimum width of at least 10 nm, preferably at least 15 nm or even at least 20 nm. This distinguishes this approach from the use of conventional spectrometers for tissue analysis, in which the spectral resolution is typically a few nanometers or even less than 1 nm. Here, the preselection provided by the first selection step makes it possible to achieve a lower spectral resolution while reliably distinguishing the target tissue type. For example, this approach can already be implemented using the three color channels of a conventional RGB sensor, which each detects a spectral range greater than 50 nm, which also overlaps in some cases.

[0044] A preferred application of the method provides that the target tissue type is neural tissue and / or that in addition to showing the neural tissue, the at least first image range also shows adipose tissue. Thus, the method is used here to distinguish neural tissue and adipose tissue in a first selection step in order to subsequently distinguish the neural tissue from the adipose tissue in a second selection step.

[0045] For example (particularly for the purpose of differentiating neural tissue as explained just above), it can be provided that the first imaging method is based on autofluorescence imaging and the second imaging method is based on diffuse reflectance spectroscopy. In this embodiment, the UV excitation light used for exciting the AF can be kept away from the image sensor of the image recording system by means of an excitation light filter.

[0046] Without the use of the first imaging method and the resulting preselection, it would be very difficult to identify and locate, for example, neural tissue directly based on the measured reflected light with the help of the DRS method: because, on the one hand, there are many different tissue types, which each show different typical colors that can be determined in diffuse reflection. In addition, the color of tissues may also vary from patient to patient; for example, the blood supply may be better or worse, or specific discolorations may occur, in particular yellowing of specific tissues, for example due to metabolic diseases. All of this makes it very troublesome to locate the target tissue type directly using the DRS method. For this purpose, a spectrometer that can spectrally analyze diffuse reflection accurately on a few nanometers must be used. However, this is very troublesome or even impossible to achieve precisely in endoscopic applications, because the spectrometer has a huge space requirement.

[0047] The presented approach can be implemented, for example, in an image processing algorithm which is deterministic and which, due to the preliminary reduction of the solution space carried out with the first measuring method, can determine specific tissue types, for example neural tissue, quickly and efficiently and with high reliability.

[0048] In this case, a preferred variant provides that diffuse reflection spectroscopy is modified so that it is measured without a spectrometer and / or contactlessly. This can be achieved, for example, by calculating a plurality of color ratios based on a plurality of color channels of an image recording system used in the method (in particular one or more color image sensors) based on corresponding measured values ​​determined with the image recording system in a specific wavelength range (per pixel). If, for example, three beam splitters and three color image sensors with three color channels are used, nine color channels can be generated that can be used to calculate different color ratios.

[0049] This solution can be implemented in hardware as follows, for example:

[0050] System 1: Broadband illumination, for example by means of a xenon lamp; a beam splitter is used in the image recording system so that at least two image sensors can be used to detect different wavelength ranges. In this case, at least one of the two image sensors can be a color image sensor with multiple color channels. This approach can form 3, 4, 5, 6 or even more different color channels. Therefore, by correspondingly operating the signals of the corresponding color channels with one another, multiple color ratios can also be calculated per pixel. In this case, four parameters are determined for a specific color ratio: if, for example, a ratio is formed from the wavelength ranges 470±5 nm and 620±10 nm, four values ​​are given for the corresponding start and end values ​​of the corresponding wavelength intervals (e.g. 465, 475; 610, 630 in the above example).

[0051] In this approach, the target tissue type is thus identified based on the measured color ratio. This approach operates purely empirically: unlike the prior art to date, the DRS method according to the invention is not intended to be used to determine obvious physiological parameters, such as the fat content or the beta-carotene content or the ratio of oxygen-saturated hemoglobin to unsaturated hemoglobin (which depends on the oxygen saturation of the blood). The approaches known to date therefore use spectrometers and DRS to determine such physiological parameters and only then deduce the tissue type from these specific physiological parameters.

[0052] In contrast, the solution according to the invention proposes to determine the target tissue type directly (without calculating physiological parameters) from the measured color ratio.

[0053] System 2: In the simplest case, the image acquisition system used in the solution according to the invention can have only one monochrome image sensor, but different illumination light sources, for example three different LEDs, can be used. If the light sources are operated discontinuously, for example pulsed, they emit illumination light with different spectra at different points in time. In this way, image signals that can then be used to calculate color ratios can also be detected in different spectral ranges in a spatially resolved manner. However, RGB sensors or HSI (hyperspectral imaging) sensors can also be used in this method according to the invention. However, unlike the case of using (in some cases only) broadband illumination, a beam splitter is not required here, because the spectral splitting is not carried out in space, but in time. Nevertheless, a beam splitter can also be used here to improve the spectral resolution, for example by means of a beam splitter and a corresponding dichroic filter for each LED, which detects the illumination light and divides it again spatially and spectrally.

[0054] Furthermore, the necessary spectral information that must be collected in order to determine the color ratios can also be determined with the aid of a scanning scheme in which the spectral image information from an image point is spatially distributed over an image sensor used in an image recording system, for example using a scanning mirror or under time-varying illumination.

[0055] According to another embodiment of the method according to the invention explained above, it can be provided that the first imaging method is based on autofluorescence imaging, while the second imaging method is also based on fluorescence, but on fluorescence generated by introduced fluorophores in the object area (observed by the image recording system); that is, the fluorescence imaging in the second selection step is here realized using at least one fluorophore introduced into the area observed by the image recording system. In this specific embodiment, two fluorescence signals (fluorescence signal of autofluorescence and signal generated by means of the at least one introduced fluorophore) can be distinguished, because AF is excited by UV light, while, for example, when using ICG (indocyanine green acid) as a possible fluorophore, it works with an excitation wavelength of 800 nm, and the fluorescence signal is therefore not 500 nm as in autofluorescence, but about 830 nm. In other words, the corresponding fluorescence signals evaluated in the first or second selection step may therefore appear differently in the spectrum. Therefore, in this embodiment, it is first possible to preselect by means of AF imaging, and then to identify the second image area within the first image area by means of fluorophore-based imaging. In this way, the surgeon can also perform the resection in such a way that, for example, the nerve tissue is not damaged.

[0056] However, the method according to the invention can also be implemented in the opposite case, i.e. if first a first selection step is performed based on fluorescence (i.e. based on fluorophores introduced into the body from the outside) and the at least one second image range is subsequently determined based on the detected AF signal.

[0057] According to another embodiment of the method according to the invention explained above, it can be provided that the first imaging method is based on autofluorescence imaging and the second imaging method is based on laser speckle imaging (LSI). In this case, the general term "laser speckle imaging" can be understood to include, in particular, methods such as laser speckle contrast imaging (LSCI) and laser speckle contrast analysis (LASCA). In this case, the method can therefore be characterized by working with coherent illumination in the second selection step in order to generate the desired speckle pattern, while incoherent illumination is used in the first selection step. Based on interference, microscopic movements of tissue, such as the extent of blood vessels or their pulsation, can thus be detected. It has been shown here that neural tissue often has well-supplied connective tissue, which is traversed by very thin blood vessels, so that neural tissue can also be selectively identified by means of LSI. That is to say, in this specific embodiment, the differentiation of the target tissue type is carried out in the second selection step by means of interference motion analysis.

[0058] According to yet another embodiment of the method according to the invention explained above, it can be provided that the first imaging method is based on autofluorescence imaging and the second imaging method is based on polarization. This specific embodiment is particularly suitable for determining nerve tissue, since nerves are typically present in the form of fine nerve cords that all extend in the same direction and therefore have a high degree of order, which can be automatically detected in the surgical setting by means of polarization imaging and corresponding image evaluation.

[0059] In summary, it can be seen that according to the general concept of the invention, the first and second measuring or imaging methods are respectively different and in a second selection step (i) different fluorescence signals or (ii) optically detected motion signals (in the case of LSI) or (iii) (optically detected) information about the structure of the tissue (e.g. in the case of polarization) is used for further tissue differentiation. It should be taken into account that tissue-specific (yet to be developed) fluorophores do not have to be used, but conventional fluorophores can be used or even the use of (external) fluorophores can be completely omitted.

[0060] The invention thus solves the basic problem that the aforementioned measuring methods are often not specific enough on their own to localize specific tissue types, such as neural tissue, in surgical situations. It must be taken into account that, especially in the case of endoscopic interventions, there are technical boundary conditions regarding the spectral resolution that can be achieved with corresponding image recording systems. The two-step approach of the invention now solves this problem by significantly reducing the solution space in a first selection step and then, in a second selection step, still being able to reliably differentiate and localize tissue types even with only limited spectral information.

[0061] The present invention will now be described in more detail based on embodiments, but the present invention is not limited to these embodiments. Other configurations of the present invention can be obtained from the subsequent description of preferred embodiments in combination with the general description, the claims and the drawings.

[0062] In the following description of various preferred embodiments of the invention, elements which correspond in their function are provided with identical reference numerals even in the case of different designs or shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The figure shows:

[0064] Figure 1 An example of an image recording system according to the invention is shown, with which the method according to the invention can be applied.

[0065] Figure 2 shows a schematic diagram of the essential method steps of the method according to the invention using a temporally constant illumination and a spatial division of the individual spectral image information,

[0066] Figure 3 A further schematic flow chart for explaining the method according to the invention is shown,

[0067] Figure 4 Another image recording system is shown, but with temporally modulated illumination,

[0068] Figure 5 Shown in use Figure 4 A possible embodiment of the method according to the invention in the case of an image recording system is shown in FIG.

[0069] Figure 6 A further possible implementation of the method according to the invention is shown using an HSI (hyper spectral imaging) camera. DETAILED DESCRIPTION

[0070] Figure 1An image recording system 1 is shown, which comprises an endoscope 2 with an image sensor 18 and an associated light source 15, with which both illumination light 11 and invisible UV excitation light 10 can be emitted onto a region 9 observed with the endoscope 2. The image sensor 18 of the endoscope 2 continuously acquires images 3 in the form of a video image data stream, which are processed by a subsequent camera control unit 13 and subsequently reproduced on a monitor 14 in the desired form.

[0071] As in Figure 2 As schematically shown in FIG. 1 , the endoscope 2 also has a beam splitter 17 with associated spectral optical filters, which enable the spectral image information recorded in total with the optical system of the endoscope 2 to be spatially divided and assigned to the corresponding image sensors 18. Therefore, in the lower imaging path marked with the letter D, conventional image processing is performed on the visible wavelengths in order to thus generate the visible processed image D (VIS image) shown. In addition, the camera control unit 13 also implements Figure 2 The upper image processing path shown in FIG. 1 comprises two image processing steps A and B. The image information, which is spectrally selective in certain cases, is detected by the endoscope 2, more precisely by the beam splitter and the upper image sensor 18, and is shown here in the upper left image. In this image, in addition to the nerves as the target tissue type 6 to be determined, the secondary tissue type 7 (i.e., fat tissue) as well as other tertiary tissue types 8 can also be seen. The primary tissue type 6 and the secondary tissue type 7 to be located belong to the same tissue group and produce approximately comparable autofluorescence signals. Therefore, if in Figure 1 If the light source 15 shown in FIG. 1 illuminates the region 9 with a suitable excitation wavelength in the UV range, both tissue types 6 and 7 emit corresponding autofluorescence signals, which then reach the Figure 2 in the upper image processing chain.

[0072] In order to now identify the desired target tissue type 6, i.e. the nerve pathway within the surgical region 9, and to highlight / visualize it for the surgeon so that he can locate or find the nerves in the surgical region 9, the invention provides that in a first selection step A, the first image range denoted by reference numeral 4 is first automatically identified and selected by the camera control unit 13 in the current image 3 by means of autofluorescence imaging (and corresponding image processing of the images recorded in this way), as in Figure 2 As shown by the highlighting in the upper middle detail image H of . In addition to the target tissue type 6 , these first image ranges 4 also include different secondary tissue types 7 .

[0073] Only in a subsequent second selection step B, a plurality of second image regions 5 are detected and selected within the first image region 4 by the camera control unit 13, also automatically, by means of a second imaging method different from the autofluorescence method. Figure 2 The upper right detail image I of FIG. 5 is highlighted in black and contains mainly or even exclusively neural tissue, ie the desired target tissue type 6 .

[0074] As in Figure 2 As shown in , in a third method step C, the image information (VIS image) obtained in the lower image processing chain can then be fused with the additional information about the relative position of the target tissue type 6 within the image 3 obtained by means of the two selection steps A and B. As a result, as shown in the detail image F, a so-called overlay image F can be presented to the user, in which the identified image region 6 (ie the nerve to be located) is highlighted as an overlay in the VIS image E.

[0075] It should also be mentioned that in accordance with Figure 1 and Figure 2 In the embodiment of the present invention, work is performed using a temporally constant illumination, wherein the spectral partial information on which the first selection step A and the second selection step are based is spatially separated / divided from the entire spectral information (which is optically detected using the endoscope 2) by means of a beam splitter 17.

[0076] exist Figure 3 The above-explained individual steps of the method according to the invention and the resulting images are also shown again in the schematic overview of FIG. 1 : On the left, the image processing path for the VIS image E is shown, while the right arm shows the spectral image processing path, in which a first image range 4 is automatically identified by means of a first selection step A, and a second image range 5 is automatically identified within the same image range by means of a second selection step B. The raw spectral image G (with Figure 2 The spectral information included here (captured by the upper image sensor 18 in the VIS image E) is different from the spectral information detected in the VIS image E. That is, the VIS image E shows visible wavelengths, while the image information of the spectral images G / H / I includes fluorescence wavelengths. The fluorescence wavelengths may, for example, be outside the visible wavelength range.

[0077] If for example Figure 3If the first selection step A in the method is carried out by means of autofluorescence imaging, the first image range 4 can show fat tissue and nerve tissue, while other tissue types that are also visible in the image G, such as arteries, veins, muscles, organs (such as liver) or metal objects, have already been filtered out in step A, as shown (schematically) in the image H. The second selection step B can then be carried out, for example, based on diffuse reflectance spectroscopy (DRS), so that the image I and the fused subsequent image F then contain the position of the nerve as the desired image information to be determined. Therefore, a further method step C represents the image fusion between the image I and the image E. As a result, the image F thus shows a superimposed image, which consists of the VIS image E and the desired tissue information 6 about the exact position of the nerve in the surgical region 9 (as a superimposition).

[0078] Figure 4 A further possibility of using the method according to the invention is shown with an image recording system 1. Figure 1 The system differs in that the light source 15 is modulated in time, so that different wavelengths are emitted as illumination light 11 and / or as excitation light 10 onto the observed region 9 at different points in time.

[0079] like Figure 5 As shown, therefore, using Figure 4 The system 1 can realize (temporally) sequential image detection with only one image sensor 18, so that the spectral information of the corresponding images 3 is different, wherein all images 3 are recorded using the image sensor 18. If these spectral information collected in this way over a longer period of time are fused together by corresponding image processing, it is possible (e.g. Figure 5 ) generates a VIS image E, although only a specific portion of the visible wavelength range is used for imaging at a specific time point. Figure 5 In the upper image processing path of , the spectral component information can be used at a specific point in time to implement the selection steps A and B explained above and thus determine the position of the target tissue type 6 within the image 3. In other words, in Figure 4 and Figure 5 In the example of , the first selection step A and the second selection step B are therefore performed based on the spectral portion information temporally separated from the entire spectral information of the at least one image 3.

[0080] at last, Figure 6Another possible example of an implementation of the method according to the invention is shown, wherein a hyperspectral image sensor 18 is used here, which, in addition to visible wavelengths, can also record invisible wavelengths (for example in the UV and / or in the IR wavelength range). For this purpose, the HSI image sensor 18 can have corresponding spectral filters at the pixel level, for example. Thus, the endoscope 2 is used to optically detect and thus reach Figure 6 The entire spectral information of the HIS image sensor 18 is thus re-divided spatially, ie, based on a division of individual pixels of the image sensor 18. Figure 6 In the upper image processing path, selection steps A and B can also be implemented based on spectral part information, while Figure 6 In the lower image processing chain, spectral information different from this is used for imaging in order to generate an image E, which can be a VIS map in particular.

[0081] exist Figure 6 In the example, the thickness of the respective block arrows is used as an indicator for the width of the spectral range used in each case, i.e. the wider the arrow, the more spectral information is processed in the respective image processing path. It can thus be seen that a reduction of the spectral information is also carried out in the first selection step A, so that the second selection step B can be applied to a smaller spectral range (compared to the first selection step A). ​​This enables the use of much simpler optical measurement or imaging methods and image processing algorithms in the second selection step B, which is a great advantage in particular in endoscopic applications, when only a limited spectral resolution can usually be achieved, because no spectrometer is usually available. In addition, significantly less information must be processed, which makes it easier to apply the method in real time (for example in a live video image data stream).

[0082] In summary, a new technical solution is proposed, by means of which a specific tissue type 6 in a surgical area 9 can be automatically located within a captured image 3 using an image recording system 1. For this purpose, it is provided that image ranges 4, 5 within the image 3 that show the desired tissue type 6 are identified and selected by means of two temporally successive selection steps A and B. In this case, within the framework of the first selection step A, not only is the image range 5 spatially delimited, but also, if necessary, the spectral information to be processed and / or the pixels to be processed of the corresponding image 3 are reduced. Then, in the second selection step B, the target tissue type 6 that is actually to be determined is finally identified within the image 3 and selected as the second image range 4, so that the user can finally find this tissue type 6 within the area 9 observed using the image recording system 1 (see Figure 2 ).

[0083] Reference numerals list

[0084] 1Image capture system

[0085] 2 Endoscope

[0086] 3 images (taken with 2 / 1, especially live images as part of the video image data stream)

[0087] 4 first image range (automatically identified within 3)

[0088] 5 second image range (automatically recognized within 4)

[0089] 6 Target organization type (= first-level organization type)

[0090] 7 Secondary Organization Type (same organization group as 6, but different from 6)

[0091] 8 Tertiary Organization Type (different from 6 and 7; not in the same organizational group as 6)

[0092] 9 areas (especially the surgical area, use 1 / 2 for observation)

[0093] 10 Excitation light (especially invisible, such as in UV; used to stimulate light emission in 9)

[0094] 11 Illumination light (for traditional imaging with visible wavelengths)

[0095] 12 Fluorescence

[0096] 13 controller, especially designed as a camera control unit

[0097] 14 Monitor

[0098] 15 light sources (multi-spectral variation or time-spectral variation)

[0099] 16 Overlay

[0100] 17 beam splitter (with appropriate spectral filters if necessary)

[0101] 18 Image Sensor

Claims

1. A method for automatically locating a specific target tissue type (6) within at least one image (3) recorded with an image recording system (1), in particular during an endoscopic intervention performed with an endoscope (2) which is part of the image recording system (1), It is characterized in that - in a first selection step, at least one first image range (4) is automatically identified and selected within the at least one image (3) by means of a first imaging or measuring method, wherein the at least one first image range (4) shows, in addition to the target tissue type (6), at least one further secondary tissue type (7) which is different from the target tissue type (6), and - In a second selection step, at least one second image range (5) is identified and selected within the at least one first image range (4) that has been preselected by means of a second imaging or measuring method that is different from the first imaging or measuring method, preferably an optical imaging or measuring method, wherein the at least one second image range (5) primarily or even exclusively displays the target tissue type (6) to be determined.

2. The method according to claim 1, wherein: The first selection step and / or the second selection step is automatically performed by the image recording system (1) based on an image evaluation of the at least one image (3).

3. The method according to any one of the preceding claims, wherein: The first selection step and the second selection step are performed continuously in real time and the target tissue type (6) is visually highlighted in the live video image data stream, in particular in order to enable a user to locate the target tissue type (6) within the area observed with the image recording system (1).

4. The method according to any one of the preceding claims, wherein: The first selection step and / or the second selection step is performed based on spectral portion information separated spatially or temporally from the entire spectral information of the at least one image (3).

5. The method according to any one of the preceding claims, wherein: The first selection step reduces the amount of image information and / or spectral information that is processed in the second selection step.

6. The method according to any one of the preceding claims, wherein: In the context of a second selection step, color ratios are calculated for different image points of the at least one image (3) based on at least two different color channels of the image recording system (1). Preferably, the respective spectral range on which the color ratio is determined has a minimum width of at least 10 nm, preferably at least 15 nm or even at least 20 nm.

7. The method according to any one of the preceding claims, wherein: The target tissue type (6) is neural tissue, and / or - wherein the at least first image region (4) shows fat tissue in addition to neural tissue.

8. The method according to any one of the preceding claims, wherein: The first imaging or measuring method is selected from one of the following methods: - Diffuse reflectance spectroscopy (DRS); - autofluorescence imaging, in particular by illuminating the area (9) observed with the image recording system (1) with excitation light (10) and / or without introducing any colorants, markings or the like into the area (9) observed with the image recording system (1); - fluorescence imaging, in particular by illuminating the area (9) observed by the image recording system (1) with excitation light (10) and / or by introducing colorants or markings into the surgical area observed by the image recording system (1); - Laser Speckle Imaging (LSI), in particular by illuminating the area (9) observed by the image recording system (1) with coherent light; - Imaging based on polarization analysis, in particular, for this purpose the area (9) observed by the image recording system (1) is illuminated with polarized light and / or the at least one image (3) is recorded with the aid of a polarization filter.

9. The method according to any one of the preceding claims, wherein: The second imaging or measurement method is selected from one of the following methods: Diffuse reflectance spectroscopy (DRS), in particular differentiation between neural tissue and fat tissue based on diffuse reflectance spectroscopy, so that the at least one second image range (5) predominantly or even exclusively contains / shows neural tissue; - fluorescence imaging using at least one fluorophore introduced into the area observed with the image recording system; - Laser Speckle Imaging (LSI), in particular by illuminating the area (9) observed by the image recording system (1) with coherent light; - Imaging based on polarization analysis, in particular, for this purpose the area (9) observed by the image recording system (1) is illuminated with polarized light and / or the at least one image (3) is recorded with the aid of a polarization filter.

10. The method according to any of the preceding claims, wherein: In the framework of the second selection step, - a fluorescent signal that is spectrally different from the fluorescent signal used in the first selection step, or - an optically detected motion signal, or - Optically detected information about the structure of the tissue Used to further differentiate tissues in a second selection step.

11. The method according to any one of the preceding claims, - wherein the at least one first image range (4) is determined by computer implementation, - Especially with the help of artificial intelligence, - preferably, wherein The artificial intelligence has been trained with measurement data which were measured for different tissue types with a first imaging method, preferably also with an image recording system (1) used in the method, in particular an endoscope (2) used therein.

12. Medical image capture system (1), - in particular comprises an endoscope (2), a microscope or a macroscope, characterized in that the image recording system comprises a controller (13) which is configured to carry out the method according to one of the preceding claims based on image data recorded by the image recording system (1), in particular as a live video image data stream, Preferably and as a result of the method, information is output, in particular by displaying a second image range (5), which information enables the localization of at least one target tissue type (6) determined by means of the method within at least one image (3) recorded with the image recording system (1).

Images