Method for the automated localization of a specific tissue type during an endoscopic procedure and associated image acquisition system

Abstract

A new technical approach is proposed for the automated localization of a specific tissue type (6) within a surgical field (9) using an image acquisition system (1) within a captured image (3). This involves identifying and selecting, by means of two sequential selection steps A and B, those image areas (4) within the image (3) that show the desired tissue type (6). In the first selection step A, the image areas (5) are spatially delimited, and, if necessary, the spectral information and / or pixels of the respective image (3) to be processed are reduced.Only in the second selection step B is the target tissue type (6) to be determined finally identified within the image (3) and selected as the second image areas (4), so that a user can finally find this tissue type (6) within the area (9) observed with the image acquisition system (1).

Description

[0001] The invention relates to a method for the automated localization of a specific target tissue type within at least one image acquired with an image acquisition system (particularly a medical one). This can be done, for example, during an endoscopic procedure performed with an endoscope as part of the image acquisition system. The method according to the invention is intended to facilitate, or even make possible in the first place, the reliable differentiation of the target tissue type from other types of tissue within the image and thus its localization within a surgical field (currently being viewed by the image acquisition system).

[0002] It is already known in the art to visualize certain identifiable tissue types, such as tumors, using fluorescent dyes, so that the cancerous tissue can be identified in a live video image by corresponding color marking for the surgeon. Furthermore, surgeons also utilize the fact that the mechanical strength of cancerous tissue differs from that of healthy tissue to locate and pinpoint the cancerous tissue by palpation. However, with such approaches, the physician often relies on their experience and anatomical knowledge to locate a specific tissue type within a surgical field using an endoscope.

[0003] Nerve tissue is among the critical structures in the human body, and nerve damage during surgery can lead to various consequences, including chronic pain or even the loss of function of various organ structures. This is often accompanied by a significant loss of quality of life for the affected person.

[0004] Especially during head and neck surgery, careful nerve identification is essential to avoid nerve damage. Currently, surgeons identify nerves solely based on their anatomy and appearance, sometimes in combination with electromyography (EMG), often revealing nerve damage after surgery. However, scientific publications, such as the article "In Vivo Nerve Identification in Head and Neck Surgery Using Diffuse Reflectance Spectroscopy" by Langhout et al., published in 2018 in the journal "Laryngoscope Investigative Otolaryngology," propose optical methods like diffuse reflectance spectroscopy (DRS) and fluorescence spectroscopy (FS) for targeted nerve identification.

[0005] Several state-of-the-art technological approaches are currently being pursued for the localization of peripheral nerves. One such approach is autofluorescence (AF), which exploits the fact that different tissue types exhibit varying degrees of fluorescence when stimulated with appropriate wavelengths. Autofluorescence is based on the presence of natural fluorophores in the human body (such as fatty pigments, collagen, or elastin) that can be stimulated to autofluoresce under UV light. However, this method visualizes not only nerves but also other tissue types that exhibit autofluorescence, such as ligaments or adipose tissue. In medical applications, it is crucial to clearly distinguish the nerve from these other tissue types to prevent injury during surgery.The crucial disadvantage of the autofluorescence method is that tissue types other than nerve tissue also exhibit autofluorescence properties, meaning they too become visible in autofluorescence imaging. Therefore, the described autofluorescence method is not specific enough to detect and localize nerve tissue exclusively and selectively.

[0006] On the other hand, physicians often rely on visual impressions, experience, and anatomical knowledge when endoscopically locating nerves. However, these impressions are all subjective, and anatomy can vary considerably from patient to patient. For these reasons, various technologies are being tested to enable more reliable and reproducible localization of nerve tissue during endoscopic procedures, thereby preventing nerve injuries during the intervention.

[0007] Another possible method that can be used for nerve detection is, as mentioned, diffuse reflectance spectroscopy (DRS). This method typically uses reflection spectra with high wavelength resolution, for example, 1 nm, to distinguish different tissue types based on their "spectral fingerprint." However, the corresponding probe is placed directly on the tissue or even inserted into it. With DRS, a probe must therefore be in direct contact with the tissue and consequently only provides information at a specific point within the current field of view. This is not practical for endoscopic applications, especially when a larger surgical area needs to be examined. Localizing specific tissue types within the entire field of view of an endoscope is not possible with this method.

[0008] Based on this background, the invention aims to provide an improved technical approach with which a specific type of tissue, in particular nerve tissue, can be located within an operating area, which is, for example, being observed with an endoscope.

[0009] To solve this problem, the features of claim 1 are provided according to the invention in a method. In particular, it is thus proposed according to the invention to solve the problem in a method of the type mentioned at the outset that, in a first selection step, at least one first image area within the at least one image is automatically identified and selected by means of a first imaging method. Here, the at least first image area visualizes not only the target tissue type but also at least one other secondary tissue type that differs from the target tissue type. In other words, this first pre-selection is not yet sufficiently selective to locate the target tissue type with sufficient accuracy in the image. However, the result of the first selection step can already be an image with a reduced amount of information.

[0010] Therefore, according to the invention, in a second selection step, at least one second image area (as a sub-area) within the at least one already pre-selected first image area is identified and selected using a second imaging method that differs from the first (this second imaging method, like the first, can preferably be an optical method). In this second image area, the at least one second image area then visualizes predominantly or even exclusively the target tissue type to be determined. In other words, the target tissue type can thus be located with sufficient selectivity within the surgical field based on the second image area. The result of the second selection step can therefore be an image with an even further reduced amount of information.The image obtained in this way can therefore be, in particular, a spectral image containing the desired information, namely the relative location and extent of the target tissue type within the original at least one image.

[0011] The process can therefore comprise two successive image processing steps, namely the first and the second selection step.

[0012] In a third, subsequent step of the process, the identified second image area can then be highlighted for a user in the first image or in a corresponding image, such as a VIS image (this could be, for example, a still image or a dynamic live image). This can be done using specific false colors or, for example, with the help of an "image overlay".

[0013] This method allows for a pixel-precise and automated determination (without any user input) of whether a given pixel represents the target tissue type or not. In practical application, it is possible that a second image area may have a lower resolution than the underlying image (which could be, for example, a live VIS or IR image) that the surgeon is viewing and / or recording with the image acquisition system.

[0014] The method according to the invention is also fundamentally applicable to both open medical procedures and endoscopic procedures performed inside the body. The image acquisition system used in the method can, for example, comprise an endoscope, a microscope, or a macroscope, since it is irrelevant how the image data is optically acquired.

[0015] The method according to the invention thus enables the rapid (especially in real time based on a live video image data stream), highly reliable, and spatially resolved identification of a specific tissue type within an image recorded by the image acquisition system, particularly within a live video image data stream. The main benefit of this application lies in improved surgical guidance: In particular, injury to nerve tissue can be prevented, thereby effectively preventing consequential damage. Furthermore, the method can support the physician in their decision-making during surgery in certain situations.

[0016] Specifically, when locating peripheral nerves during endoscopic surgery, the surgeon can be supported by the use of this technology. The method according to the invention takes into account the specific constraints that exist during endoscopic procedures, in particular the need to identify the tissue type to be located without contact. The method enables contactless and reliable localization of nerves throughout the entire current and changing field of view of an endoscope.

[0017] As described above, the invention provides for the combination of at least two different imaging (these can also be considered "measurement methods") or measurement methods to detect and locate a specific type of tissue, for example, nerve tissue, within a newly recorded image. This approach is particularly advantageous for the localization of nerves of the peripheral nervous system, as previously explained.

[0018] For example, in the first step of the process, autofluorescence can be used to classify the initial image regions, identifying which tissue types exhibit high autofluorescence activity. Subsequently, in the second step, within these initial image regions, the secondary image regions that represent / contain the target tissue type can be identified and localized. This second step can utilize imaging techniques such as: (a) fluorescence using extrinsically introduced fluorophores; (b) polarization analysis; (c) laser spectral contrast imaging (LSCI); and (d) color ratio analysis.

[0019] The combination of different imaging technologies for localizing specific tissue types is a novel approach that, in particular, enables a precise differentiation between nerve tissue and other tissue types, as observed in a typical surgical situation, especially during endoscopic procedures. One advantage is that localization can be performed across the entire field of view of the endoscope without requiring any contact with the tissue.

[0020] A second possible application of the invention is the localization and visualization of blood vessels. In many surgical procedures, it is extremely disruptive if blood vessels are injured and bleeding must first be stopped before the operation can continue. In particular, the endoscope optics becoming contaminated with blood can be problematic. Such unintentional injuries can also be avoided with the method according to the invention if the second selection step is designed for the identification of blood vessels.

[0021] In contrast to the DRS method mentioned at the outset, the method according to the invention, particularly in the second process step, can provide that only a small number of spectrally narrowly defined wavelength ranges (i.e., specific, narrowly defined color ranges) are used / analyzed to differentiate between different tissue types. It is therefore not necessary to record a complete, high-resolution continuous spectrum, which is generally only possible using a complex and difficult-to-miniaturize spectrometer. The second process step can therefore be carried out, in particular, without the use of a spectrometer, as will be explained in more detail below.

[0022] Since the method according to the invention can be carried out without contact, the measuring optics used, which can be integrated into the endoscope used in the method, do not need to be placed directly on the tissue. Instead, these measuring optics can be held at a distance from the tissue during measurement, as is the case in typical endoscopic procedures. This has the advantage that nerves can be localized within the entire (movable) field of view of the endoscope / microscope / macroscope.

[0023] The target tissue type to be determined can be considered a primary tissue type, which is the focus of attention. The second image area can exclude at least one secondary tissue type (which may belong to a group of tissues to which the target tissue type also belongs), meaning it is either not shown at all or only to a negligible extent. Furthermore, the first image area can exclude at least one further tertiary tissue type that is present / visualized in the overall image, meaning it is either not shown at all or only to a negligible extent. Conversely, both the primary and at least one secondary tissue type can be visualized / present in the first image area.

[0024] In typical application scenarios, an image captured by the imaging system might show, for example, arteries, veins, adipose tissue, muscles, nerve tissue, the liver, or the intestines as different tissue types, but also foreign objects, such as metallic objects like surgical instruments. Using the first (pre-)selection step, a spatial pre-filter can be performed within a newly acquired image, thereby identifying those image areas that show the target tissue type to be located and possibly other tissue types that behave at least similarly to the target tissue type, i.e., that emit an autofluorescence signal like it. The first image area can thus visualize, for example, a specific group of tissue types (this could be, for example, a specific subgroup of certain tissue types).Furthermore, the first image area can already exclude at least one tissue type present / visualized in the (overall) image (for example, a tissue type that does not provide an AF signal).

[0025] Based on the second selection step, the described method can provide a surgeon with spatial information, namely the at least one second image area identified in the recorded image (especially in a live image). This spatial information enables the surgeon to plan and / or perform a subsequent therapeutic procedure. However, the method according to the invention itself is not aimed at performing this therapeutic step, but merely at providing information for preparing the subsequent therapy and / or diagnosis, for example, a specific tissue resection, which is optically observed with the endoscope and performed with a separate surgical instrument.Thus, in the method according to the invention, it can be provided in particular that no therapeutic or diagnostic step is carried out, but only information for the planning and preparation of a subsequent therapeutic and / or diagnostic step is provided.

[0026] The endoscope mentioned can be part of the image acquisition system, which may also include, for example, a monitor and / or a camera control unit.

[0027] The captured image can be a single image (“still image”) or, for example, part of a live video image data stream that is or was continuously recorded by the image capture system.

[0028] The first measurement method can be, for example, imaging based on diffuse reflectance spectroscopy (DRS) or autofluorescence (AF). In the latter AF method, fluorophores, which occur naturally in human tissue, are excited to emit light using (typically invisible) excitation light (e.g., in the UV wavelength range). This light emission can then be detected by a sensor and is typically at 500 nm. After evaluation of such an autofluorescence signal recorded by the imaging system, the first image areas selected in this way can, for example, show only fatty tissue and nerve tissue. The excitation light can be excluded from the image using an excitation light filter.

[0029] As mentioned, the inventive method can, in particular, utilize so-called measured color ratios to differentiate between different tissue types. Such a color ratio can be understood, for example, as the ratio of the respective intensities of a pixel across two different wavelength ranges (especially with a respective bandwidth of, for example, less than 200 nm, preferably less than 100 nm, and / or at least 10 nm). To determine such wavelength ratios (synonyms for which are: color ratios), measured intensity values ​​from the individual wavelength ranges (which may partially overlap) are first required. Such spectrally selective intensity values ​​can be obtained in two fundamental ways: (i) Firstly, wavelength sensitivity can be implemented in the camera head of the system. According to this approach, hyperspectral sensors, for example, can be used, and various technologies exist for implementing such sensors. Among others, so-called "snapshot sensors," which provide only a few (e.g., 16) different color channels, are also suitable for the approach according to the invention. This is because a very high spectral resolution is generally not necessary for the approach according to the invention. However, the acquisition of different wavelength ranges in an image acquisition system according to the invention can also be achieved, for example, by means of a beam splitter and a wavelength-selective (e.g., dichroic) layer; in this case, conventional color and / or monochromatic image sensors can be used.In this way, a spatial separation of different wavelengths can be achieved, so that two different spectral wavelength ranges can be directed to different (spatially separated) image sensors of the system and detected separately there. (ii) On the other hand, wavelength sensitivity (in addition to or as an alternative to approach (i)) can also be based on a (time-modulated) light source that is used together with the image acquisition system during imaging. For example, only a specific wavelength range can be emitted by the light source at any given time. Such time-varying spectral illumination can thus result in a temporal separation of different wavelength ranges, so that – even when using only a single image sensor – different spectral wavelength ranges can be separately detected by the sensor at different times.

[0030] All these approaches can therefore be used to determine the necessary intensity values ​​in different wavelength ranges, which are necessary for calculating color ratios.

[0031] Unlike conventional hyperspectral imaging approaches, this method does not require the acquisition of a complex color fingerprint of a specific tissue type with high spectral resolution. Instead, according to the invention, it is possible and more effective to first empirically determine how certain color ratios manifest in a particular tissue type. Subsequently, when performing the method according to the invention, the tissue type can be directly, much more reliably, and more quickly identified based on such measured color ratios (and especially on such empirical data), without the need for the time-consuming acquisition and evaluation of broadband spectral measurement data (e.g., across the entire visible range plus subsequent non-visible wavelength ranges).The reduction in complexity is achieved through the targeted selection of the respective spectral range, which is taken into account when calculating the respective color ratio. This selection depends on the tissue type to be determined and can be established by preliminary test subjects.

[0032] The challenge in the described method of using color ratios lies in identifying suitable (preferably narrowband) spectral ranges to be used for differentiating between tissue types; that is, in selecting the specific color ratios to be calculated in each selection step. These spectral ranges must, in their entirety, enable the localization of a specific tissue type without interindividual differences in the tissue (from patient to patient) or other interfering factors negatively affecting the localization. However, since the differences between the numerous tissue types found in humans are very large, a significant challenge lies in empirically identifying a sufficiently large number of suitable spectral ranges.However, this can be achieved through preliminary tests and then taken into account when designing the image acquisition system.

[0033] Here, too, the invention offers a significant advantage because the first process / selection step reduces the solution space so drastically that the desired specific differentiation of tissue types can only be achieved in the second step using an optical method such as DRS and / or measured color ratios. The spatial pre-filtering implemented in the first selection step, for example based on AF, allows the image areas to be differentiated to be narrowed down to such an extent that the DRS method or another imaging or measurement technique (such as the aforementioned determination of color ratios) delivers usable results in the second selection step, even with low spectral resolution.

[0034] Accordingly, the problem can also be solved by further advantageous embodiments according to the dependent claims.

[0035] The first and / or second selection step can thus be automated by the image acquisition system, particularly based on an image analysis of at least one image. Image analysis can also include, for example, a pixel-based calculation of color ratios for individual image areas.

[0036] Furthermore, the first and second selection steps can be performed continuously in real time.

[0037] Furthermore, the target tissue type can be visually highlighted, for example, in a live video image stream. This makes it particularly easy for a user to locate the target tissue type within an area observed by the image acquisition system.

[0038] Furthermore, for robust identification of the target tissue type, it is advantageous if the first and / or second selection step is performed based on a partial spectral piece of information that is spatially or temporally separated from the total spectral information of at least one image. In other words, the image processing underlying the first and / or second selection step is applied only to a portion of the total spectral information provided by the respective image.In particular, at least one image can cover the entire visible wavelength spectrum (and thus be a VIS image), while the first and / or second selection step is applied to a spectral image that shows the same scene / image area as the at least one VIS image (in which the target tissue type is ultimately to be visualized for the user), but only represents a portion of this total spectral information or information that differs from it. For example, the first and / or second selection step can be applied to an infrared image, especially a fluorescence image, whose spectral information lies outside the visible range. After the selection of the second image areas within this spectral image, these can then be visualized for the user in the VIS image.

[0039] The identification and selection of the second image areas using the second selection step is particularly reliable and fast if the first selection step reduces the amount of image information (e.g., the number of image pixels to be processed) and / or the amount of spectral information (e.g., the number of color channels to be evaluated per pixel) processed in the second selection step. Thus, the first selection step A reduces the solution space of possible tissue types that need to be distinguished in step B. The fewer tissue types that need to be distinguished, the less information in the form of wavelength ranges is required for this purpose in the second selection step B.In practice, there are also cases where the inventive approach makes it possible to differentiate between tissue types in this way for the first time (since recognition by means of a single selection step is often not possible with available hardware). In the image acquisition system itself, this means that fewer wavelength ranges need to be recorded / sensorially detected. This can result, for example, in a reduction of the number of wavelength ranges generated sequentially by a modulatable light source, which would (actually) be necessary for robust detection of the target tissue, or in a small number of color channels provided by a hyperspectral sensor of the system being sufficient to reliably enable tissue recognition.In other words, the inventive approach can significantly reduce the hardware requirements of the system, thereby opening up new areas of application. For example, with the inventive approach, it may be sufficient to read out only 16 color channels / spectral bands of an image sensor, instead of 100 different channels, as is required by previously known approaches that demand high spectral resolution. This also demonstrates the concrete technical advantages of the invention.

[0040] From another perspective, the method according to the invention can thus be designed such that only a limited spectral range is evaluated in the first selection step and / or in the second selection step. This limited spectral range can differ, in particular be smaller or not overlapping, with a (particularly broader) spectral range that is ultimately displayed to the user in the at least one image (e.g., the entire visible area).

[0041] One possible implementation of the method according to the invention involves calculating so-called color ratios in the second selection step, based on at least two different color channels of the image acquisition system for different pixels of the at least one image. The respective color ratio indicates an intensity ratio of at least two spectral ranges for the respective pixel or image section. It is preferable if the respective spectral ranges on which the color ratios are based have a minimum width of at least 10 nm, preferably at least 15 nm, or even at least 20 nm. This distinguishes this approach from the use of classical spectrometers for tissue analysis, where the spectral resolution is typically a few nm or even less than 1 nm.The lower spectral resolution, while still reliably identifying the target tissue type, is made possible by the pre-selection provided by the first selection step. For example, such an approach can already be implemented with the three color channels of a classic RGB sensor, each of which captures spectral ranges of more than 50 nm, which may also overlap.

[0042] A preferred application of the method involves the target tissue type being nerve tissue and / or the visualization of at least the first image area alongside nerve tissue as well as adipose tissue. In this case, the method is used to identify nerve and adipose tissue in the first selection step, and then to differentiate the nerve tissue from the adipose tissue in the second selection step.

[0043] For example, particularly for the purpose of differentiating nerve tissue as explained above, the first imaging method may be based on autofluorescence, while the second is based on diffuse reflectance spectroscopy. In this approach, the UV excitation light used to excite the autofluorescence can be kept away from the image sensor of the imaging system by means of an excitation light filter.

[0044] Without the use of the initial imaging technique and the resulting preselection, it would be very difficult to directly identify and localize nerve tissue, for example, using the DRS method based on measured reflected light. This is because, firstly, there are many different tissue types, each exhibiting distinct, characteristic colors that can be determined from the diffuse reflection. Furthermore, tissue color can vary from patient to patient; for example, blood flow may be more or less adequate, or certain discolorations may occur, particularly yellowing of specific tissues, due to metabolic disorders. All of this would make it very complex to directly localize the target tissue type using the DRS method. This would require the use of a spectrometer capable of spectrally analyzing diffuse reflection with an accuracy of a few nanometers.However, this is very complex or even impossible to implement, especially in endoscopic applications, because a spectrometer requires an enormous amount of space.

[0045] The presented approach can, for example, be implemented in an image processing algorithm that is deterministic and, due to the pre-reduction of the solution space carried out with the first measurement method, can quickly, efficiently and with high certainty determine a specific tissue type, such as nerve tissue.

[0046] A preferred approach involves modifying diffuse reflection spectroscopy to enable spectrometer-free and / or contactless measurements. This can be implemented, for example, by calculating multiple color ratios based on several color channels (especially one or more color image sensors) of the image acquisition system used in the method. These calculations are based on individual measurements taken by the image acquisition system in specific wavelength ranges (per pixel). For instance, if three beam splitters and three color image sensors, each with three color channels, are used, nine color channels can be generated that can be used to calculate different color ratios.

[0047] Such an approach can be implemented with hardware as follows, for example: System 1: Broadband illumination, for example using a xenon lamp; use of a beam splitter in the image acquisition system so that different wavelength ranges can be captured with at least two image sensors. At least one of the two image sensors can be a color image sensor with multiple color channels. This approach allows for 3, 4, 5, 6, or even more different color channels. Thus, multiple color ratios can be calculated pixel by pixel by combining the signals of the respective color channels. Four parameters must be defined for a specific color ratio: For example, if a ratio is calculated from the wavelength range 470 ± 5 nm and 620 ± 10 nm, four values ​​must be specified for the respective start and end values ​​of the respective wavelength interval (e.g., 465, 475; 610, 630 in the example above).

[0048] In this approach, the target tissue type is identified based on measured color ratios. This approach is purely empirical: Unlike previous methods, the DRS method according to the invention is specifically not intended to be used to determine meaningful physiological parameters, such as fat content, beta-carotene content, or the ratio of oxygenated to unsaturated hemoglobin (this ratio depends on the blood's oxygen saturation). Previously known approaches use spectrometers and DRS to determine such physiological parameters, and only then is the tissue type inferred from these determined physiological parameters.

[0049] In contrast, the approach according to the invention proposes to determine the target tissue type directly (without calculating physiological parameters) from measured color ratios.

[0050] System 2: In the simplest case, an image acquisition system used in the inventive approach can have only a monochrome image sensor, but employ different illumination light sources, for example, three different LEDs. If the light sources are operated discontinuously, for example, in pulsed mode, they emit spectrally different illumination light at different times. In this way, it is also possible to acquire spatially resolved image signals in different spectral ranges, which can then be used to calculate color ratios. However, an RGB sensor or an HSI (hyperspectral imaging) sensor could also be used in such an inventive approach.

[0051] Unlike when using a (potentially single) broadband light source, a beam splitter is not necessary here because the spectral splitting is temporal, not spatial. However, a beam splitter could still be used to improve spectral resolution by further spatially splitting the captured light for each LED using the beam splitter and appropriate dichroic filters.

[0052] Furthermore, the necessary spectral information that needs to be collected to determine the color ratios can also be obtained using a scanning approach, in which spectral image information from a pixel is spatially distributed on an image sensor used in the image acquisition system, for example using a scanning mirror or under time-varying illumination.

[0053] According to a further embodiment of the previously described method according to the invention, it can be provided that the first imaging method is an imaging method based on autofluorescence, while the second imaging method is also based on fluorescence, but on a fluorescence that is generated by introduced fluorophores in the object area (observed with the image acquisition system); i.e., the fluorescence imaging in the second selection step is carried out here using at least one fluorophore which is introduced into an area observed with the image acquisition system.In this specific approach, both fluorescence signals (the autofluorescence signal and the signal generated by the at least one introduced fluorophore) can be distinguished because autofluorescence (AF) is excited with UV light, while, for example, when using ICG (indocyanine green) as a potential fluorophore, an excitation wavelength of 800 nm is used, and the fluorescence signal is therefore not at 500 nm as with autofluorescence, but at approximately 830 nm. In other words, the respective fluorescence signals evaluated in the first and second selection steps can thus have different spectral characteristics. With this approach, a preselection can first be made using AF imaging, and then the second image regions within the first image regions can be identified using fluorophore-based imaging.This is another way a surgeon can perform a resection in such a way that, for example, nerve tissue is not damaged.

[0054] However, a method according to the invention could also be implemented in the reverse case, i.e., if the first selection step is initially carried out on the basis of fluorescence (i.e., on the basis of fluorophores extrinsically introduced into the body) and subsequently the at least one second image area is determined on the basis of a detected AF signal.

[0055] According to a further embodiment of the previously described method according to the invention, the first imaging method can be based on autofluorescence, while the second imaging method is based on laser speckle imaging (LSI). The collective term "laser speckle imaging" can also include, in particular, methods such as laser speckle contrast imaging (LSCI) and laser speckle contrast analysis (LASCA). A characteristic feature of such a method is that coherent illumination is used in the second selection step to generate the desired speckle patterns, while incoherent illumination is used in the first selection step. Based on interference, microscopic movements of the tissue can thus be detected, for example, the expansion of blood vessels or their pulsation.This has shown that nerve tissue often consists of well-vascularized connective tissue permeated by very thin blood vessels, allowing for the selective detection of nerve tissue using LSI. In this specific approach, the differentiation of the target tissue type takes place in the second selection step using interference motion analysis.

[0056] According to yet another embodiment of the previously described method according to the invention, the first imaging method can be based on autofluorescence, while the second imaging method is based on polarization. This specific approach is particularly suitable for determining nerve tissue because nerves typically exist in the form of fine nerve strands, all of which run in the same direction and thus exhibit a high degree of order, which can then be automatically detected in an operating room setting using polarization imaging and corresponding image analysis.

[0057] In summary, the generic approach according to the invention provides that the first and second measurement or imaging methods differ and that either (i) different fluorescence signals, or (ii) an optically detected motion signal (in the case of LSI), or (iii) (optically detected) information about a tissue structure (e.g., in the case of polarization) is used for further tissue differentiation in the second selection step. It should be noted that no tissue-specific (yet to be developed) fluorophores need to be used; instead, conventional fluorophores can be used, or even the use of (extrinsic) fluorophores can be completely dispensed with.

[0058] The invention thus solves the fundamental problem that the aforementioned measurement methods, taken alone, are often not specific enough to locate a particular tissue type, such as nerve tissue, in a surgical setting. It must be taken into account that, especially in endoscopic procedures, there are technical limitations regarding the spectral resolution achievable with the respective imaging system. The two-step approach of the invention solves this problem by significantly reducing the solution space in the first selection step, and then, in the second selection step, reliably differentiating and localizing the tissue type even with limited spectral information.

[0059] The invention will now be described in more detail with reference to exemplary embodiments, but is not limited to these embodiments. Further embodiments of the invention can be derived from the following description of a preferred embodiment in conjunction with the general description, the claims, and the drawings.

[0060] In the following description of various preferred embodiments of the invention, elements that are identical in function are given identical reference numerals even if they differ in design or shape.

[0061] It shows: Fig. 1 an example of an image acquisition system according to the invention with which the method according to the invention can be applied, Fig. 2 a schematic illustration of the essential process steps of the method according to the invention when using a temporally constant illumination and spatial division of the individual spectral image information, Fig. 3 a further schematic flowchart which explains the method according to the invention, Fig. 4 another image acquisition system, but with time-modulated illumination, Fig. 5 a possible embodiment of the method according to the invention when using the in Fig. 4 image acquisition system shown, and finally Fig. 6 another possible implementation of a method according to the invention using an HSI (hyper spectral imaging) camera.

[0062] The Fig. Figure 1 shows an image acquisition system 1, which includes an endoscope 2 with an image sensor 18 and an associated light source 15, with which both illumination light 11 and invisible UV excitation light 10 can be emitted onto an area 9 that is observed with the endoscope 2. The image sensor 18 of the endoscope 2 continuously captures images 3 in the form of a video image data stream, which are processed by a subsequent camera control unit 13 and then displayed in the desired form on a monitor 14.

[0063] How to in Fig. As schematically shown in Figure 2, the endoscope 2 also features a beam splitter 17 with associated spectral optical filters, which make it possible to spatially divide the spectral image information acquired by the optics of the endoscope 2 and assign it to a respective image sensor 18. In the lower imaging path, which is labeled with the letter D, classic image processing of the visible wavelengths takes place in order to generate the visible processed image D shown (VIS image). In addition, the camera control unit 13 also implements the upper path in Fig. The image processing path shown in Figure 2 comprises two image processing steps, A and B. The potentially spectrally selective image information acquired via the endoscope 2, specifically the beam splitter and the upper image sensor 18, is shown in the upper left image. In addition to nerves, the target tissue type 6 to be determined, the image also shows a secondary tissue type 7, namely adipose tissue, as well as other tertiary tissue types 8. The primary tissue type 6 to be localized and the secondary tissue type 7 belong to the same tissue group and generate an approximately comparable autofluorescence signal. Therefore, if the Fig. If the area 9 is irradiated with a suitable excitation wavelength in the UV range by the light source 15 shown in Figure 1, both of these tissue types 6 and 7 emit a corresponding autofluorescence signal, which is then fed into the upper image processing chain of the Fig. 2 reached.

[0064] In order to identify and highlight / visualize the desired target tissue type 6, namely the nerve pathways within the surgical area 9, so that the surgeon can locate or find the nerves in the surgical area 9, the invention provides that in the first selection step A, the first image areas designated with reference numeral 4 are automatically identified and selected by the camera control unit 13 in the current image 3 by means of autofluorescence imaging (and corresponding image processing of the images thus acquired), as can be seen from the highlighting in the upper middle detail image H of the Fig. 2 is illustrated. These first image areas 4 include not only the target tissue type 6 but also the divergent secondary tissue type 7.

[0065] Only in a subsequent second selection step B are several second image areas 5 within the first image area 4 identified and selected, also automatically by the camera control unit 13, using a second imaging method that differs from the autofluorescence method. These second image areas 5 are shown in the upper right detail image I of the Fig. 2 highlighted in black and show predominantly or exclusively nerve tissue, i.e. the desired target tissue type 6.

[0066] As in Fig. As indicated in section 2, in a third process step C, the image information obtained in the lower image processing chain (the VIS image) can be fused with the additional information regarding the relative position of the target tissue type 6 within image 3, obtained using the two selection steps A and B. As a result, as shown in the detail image F, a so-called overlay image F can be presented to the user, in which the identified image areas 6, i.e., the nerves to be localized, are highlighted in the VIS image E as a superposition.

[0067] It should also be mentioned that the approach according to the Fig. 1 and Fig. 2 is operated with a temporally constant illumination, whereby the spectral partial information, on the basis of which the first selection step A and the second selection step B are carried out, is spatially separated from the total spectral information (which is optically captured with the endoscope 2) using the beam splitter 17.

[0068] The previously explained individual steps of the inventive method and the images produced thereby are also shown again in the overview schematic representation of the Fig. Figure 3 illustrates: On the left, the image processing path for the VIS image E is shown, while the right arm shows the spectral image processing path, in which the first image areas 4 are automatically identified using the first selection step A, and within these, the second image areas 5 are automatically identified using the second selection step B. The original spectral image G (taken with the upper image sensor 18 in Fig. 2), to which the two selection steps A and B are applied, contains spectral information that differs from the spectral information captured in the VIS image E. While the VIS image E illustrates visible wavelengths, the image information of the spectral image G / H / I includes fluorescence wavelengths. These can, for example, lie outside the visible wavelength range.

[0069] For example, if the first selection step A is in Fig. 3. Using autofluorescence imaging, the first image areas 4 can show fat and nerve tissue, while other tissue types, such as arteries, veins, muscles, organs like the liver, or metallic objects, which were also still visible in image G, were already filtered out in step A, as indicated schematically in image H. The second selection step B can then be performed, for example, based on diffuse reflectance spectroscopy (DRS), so that image I and the fused subsequent image F contain the location of the nerves as the desired image information to be determined. The further process step C thus represents an image fusion between image I and image E. As a result, image F shows an overlay image consisting of the VIS image E and (as an overlay) the desired tissue information 6 regarding the exact location of the nerves in the surgical field 9.

[0070] Fig. Figure 4 shows another way in which a method according to the invention can be used with an image acquisition system 1. This system 1 differs from that of the Fig. 1 such that the light source 15 is modulated over time, so that different wavelengths are emitted as illumination light 11 and / or as excitation light 10 onto the observed area 9 at different times.

[0071] As the Fig. 5 shows, therefore, it can be used with system 1 of the Fig. 4. A (temporally) sequential image acquisition is performed with only a single image sensor 18, whereby the spectral information of each image 3 differs, with all images 3 being recorded with the image sensor 18. If this spectral information, collected over longer periods, is fused via appropriate image processing, then – as in the lower image processing path of the Fig. Figure 5 shows that a VIS image E can be generated even though only a certain proportion of the visible wavelength range is available for imaging at any given time. However, at a specific time in the upper image processing path, the Fig. 5. A spectral partial piece of information is used to perform the previously explained selection steps A and B and thus to determine the location of the target tissue type 6 within image 3. In other words, in the example of the Fig. 4 and Fig. 5 thus the first selection step A and the second selection step B are carried out on the basis of a spectral partial information which is separated in time from a total spectral information of at least one image 3.

[0072] Fig. Figure 6 finally shows another possible example of an implementation of a method according to the invention, wherein a hyperspectral image sensor 18 is used which, in addition to visible wavelengths, can also record invisible wavelengths (e.g., in the UV and / or IR wavelength range). For this purpose, the HSI image sensor 18 can, for example, have corresponding spectral filters at the pixel level. The entire spectral information that is optically captured by the endoscope 2 and thus transmitted to the HSI image sensor 18 is Fig. 6 is thus spatially subdivided again here, namely on the basis of individual pixels of the image sensor 18. In this way, too, the upper image processing path can then be used. Fig. 6. Based on a spectral partial piece of information, selection steps A and B are executed, while in the lower image processing chain the Fig. 6. A different spectral information is used for imaging to generate the image E, which may in particular be a VIS image.

[0073] In Fig. Figure 6 uses the thickness of the respective block arrow as an indicator of the width of the spectral range used; that is, the wider the arrow, the more spectral information is processed in the respective image processing path. It can thus be seen that a reduction of spectral information also takes place in the first selection step A, so that the second selection step B can be applied to a smaller spectral range (compared to the first selection step A). ​​This makes it possible to utilize much simpler optical measurement and imaging methods and image processing algorithms in the second selection step B, which is particularly advantageous in endoscopic applications where only limited spectral resolution can often be achieved because a spectrometer is typically not available.Furthermore, significantly less information needs to be processed, which facilitates the application of the method in real time, for example with a live video image data stream.

[0074] In summary, a new technical approach is proposed that allows a specific tissue type 6 within a surgical field 9 to be automatically localized using an image acquisition system 1 within a captured image 3. This is achieved by means of two sequential selection steps A and B, in which those image areas 4, 5 within the image 3 that show the desired tissue type 6 are identified and selected. In the first selection step A, both a spatial delimitation of the image areas 5 is performed and, if necessary, a reduction of the spectral information and / or the number of pixels to be processed in the respective image 3 is carried out.Only in the second selection step B is the target tissue type 6 to be determined finally identified within image 3 and selected as the second image area 4, so that a user can finally locate this tissue type 6 within the area 9 observed with the image acquisition system 1 (cf. . Fig. 2). Reference symbol list 1 Image acquisition system 2 Endoscope 3 Image (captured with 2 / 1, especially live image as part of a video image data stream) 4 First image area (automatically identified within 3) 5 Second image area (automatically identified within 4) 6 Target tissue type (= primary tissue type) 7 secondary tissue type (belonging to the same tissue group as 6, but different from 6) 8 Tertiary tissue type (different from 6 and 7; does not belong to the same tissue group as 6) 9 Area (especially operational area, is observed with 1 / 2) 10 Excitation light (especially not visible, e.g. in the UV; to stimulate light emission in 9) 11 Illumination light (to enable classical imaging with visible wavelengths) 12 Fluorescence light 13 controllers, in particular designed as camera control units 14" Monitor 15 Light source (multispectral or temporally varying spectral) 16 Overlay 17 beam splitters (possibly with suitable spectral filters) 18 image sensor

Claims

[1] Method for the automated localization of a specific target tissue type (6) within at least one image (3) recorded with an image recording system (1), in particular during an endoscopic procedure carried out with an endoscope (2) as part of the image recording system (1), characterized by , - that in a first selection step, at least one first image region (4) within the at least one image (3) is automatically identified and selected by means of a first imaging or measuring method, wherein the at least first image region (4) visualizes, in addition to the target tissue type (6), at least one other secondary tissue type (7) that differs from the target tissue type (6), and - that in a second selection step, at least one second image region (5) within the at least one already preselected first image region (4) is identified and selected by means of a second, preferably optical, imaging or measuring method that differs from the first imaging or measuring method, wherein the at least one second image region (5) predominantly or exclusively visualizes the target tissue type (6) to be determined. [2] Method according to claim 1, wherein the first selection step and / or the second selection step is carried out automatically by the image recording system (1) on the basis of an image evaluation of the at least one image (3). [3] Method according to one of the preceding claims, wherein the first and second selection steps are carried out continuously in real time and the target tissue type (6) is visually highlighted in a live video image data stream, in particular to enable a user to localize the target tissue type (6) within an area observed with the image acquisition system (1). [4] Method according to one of the preceding claims, wherein the first selection step and / or the second selection step is carried out on the basis of a spectral partial information which is spatially or temporally separated from an entire spectral information of the at least one image (3). [5] Method according to one of the preceding claims, wherein the first selection step reduces an amount of image information and / or an amount of spectral information processed in the second selection step. [6] Method according to one of the preceding claims, wherein, in the second selection step, color ratios are calculated on the basis of at least two different color channels of the image recording system (1) for different pixels of the at least one image (3), - preferably wherein respective spectral ranges on the basis of which the color ratios are determined have a minimum width of at least 10 nm, preferably of at least 15 nm, or even of at least 20 nm. [7] Method according to one of the preceding claims, wherein the target tissue type (6) is nerve tissue and / or - wherein the at least first image area (4) visualizes not only nerve tissue but also fatty tissue. [8] Method according to one of the preceding claims, wherein the first imaging or measuring method is selected from one of the following methods: - diffuse reflectance spectroscopy (DRS); - autofluorescence imaging, in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with excitation light (10) and / or wherein no dyes, markers or the like are introduced into the area (9) observed with the image recording system (1); - Fluorescence imaging, in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with excitation light (10) and / or dyes or markers are introduced into the surgical area observed with the image recording system (1); - Laser Speckle Imaging (LSI), in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with coherent light; - imaging based on polarization analysis, in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with polarized light and / or wherein the at least one image (3) is recorded with the aid of a polarization filter; [9] Method according to one of the preceding claims, wherein the second imaging or measuring method is selected from one of the following methods: - diffuse reflection spectroscopy (DRS), in particular wherein a distinction is made between nerve tissue and fatty tissue on the basis of the diffuse reflection spectroscopy, so that the at least one second image area (5) contains / visualizes predominantly to exclusively nerve tissue; - Fluorescence imaging using at least one fluorophore which is introduced into an area observed with the image acquisition system; - Laser Speckle Imaging (LSI), in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with coherent light; - imaging based on polarization analysis, in particular wherein for this purpose an area (9) observed with the image recording system (1) is illuminated with polarized light and / or wherein the at least one image (3) is recorded with the aid of a polarization filter; [10] Method according to one of the preceding claims, wherein in the second selection step - a fluorescence signal which differs spectrally from a fluorescence signal used in the first selection step, or - an optically detected movement signal, or - optically acquired information about a tissue structure is used for further tissue differentiation in the second selection step. [11] Method according to one of the preceding claims, - wherein the at least one first image area (4) is determined by computer implementation, - in particular by means of artificial intelligence, - preferably wherein the artificial intelligence has been trained with measurement data which have been measured for different tissue types using the first imaging method, preferably and an image recording system (1) used in the method, in particular an endoscope (2) used here. [12] Medical image acquisition system (1), - in particular comprising an endoscope (2), a microscope or a macroscope, characterized by , - that the image recording system comprises a controller (13) which is configured to carry out a method according to one of the preceding claims on the basis of image data recorded with the image recording system (1), in particular as a live video image data stream, - preferably and as a result of the method, in particular by visualizing the second image areas (5), to output information which enables localization of at least one target tissue type (6) determined with the aid of the method within at least one image (3) recorded with the image recording system (1).

Images