Image acquisition device for improved imaging in low light intensity and associated image acquisition method

Abstract

Two-stage image recording method using an image recording device (1) designed as an endoscope or as an exoscope, which - at least one image sensor (2) whose pixels are based on photodiodes, and - an additional separate single-photon sensitive detector (3), wherein - the image sensor (2) and the single-photon sensitive detector (3) are designed to detect light from a common object area (4), - the image sensor (2) is configured for imaging in the visible wavelength range and the single-photon sensitive detector (3) is configured for fluorescence imaging in a non-visible wavelength range, - the image sensor (2) and the single-photon sensitive detector (3) are arranged in a fixed spatial relationship to each other, and - the image sensor (2) is designed such that it detects not only wavelengths in the visible range but also wavelengths in the ultraviolet or infrared wavelength range for the purpose of fluorescence imaging, and the method is characterized by the following steps: - in order to enable the detection of weak fluorescent light sources in deeper tissue layers, a fluorescence image is captured with the single-photon sensitive detector (3) and superimposed on a white light image recorded with the image sensor (2), the fluorescence image not being visible in the white light image recorded with the image sensor (2) due to its low signal intensity; - Subsequently, a spatially high-resolution fluorescence light imaging is carried out with the image sensor (2) in order to be able to optically examine a fluorescent tissue found in the first step with the help of the image sensor (2) in higher spatial resolution.

Description

[0001] The invention relates to a two-stage image acquisition method using an image acquisition device configured as an endoscope or an exoscope. This image acquisition device has at least one image sensor whose pixels are based on photodiodes and an additional separate single-photon sensitive detector. The image acquisition device can be configured in particular for imaging in medical applications. Furthermore, this image acquisition device can also be configured as a multispectral imaging system. In this method, a first image, preferably a color image, is acquired using the image sensor whose pixels are based on photodiodes.

[0002] Image recording devices as described above are already known in the prior art, for example in the form of an endoscope. Due to the small diameter, the large number of lenses in the endoscope's optical system, and the small pixel size of the miniaturized image sensors used, the available amount of light is a limiting factor. Until now, this has often been compensated for by using very bright light sources or, on the image sensor side, by increasing the electronic gain. Due to the high amounts of light that must be delivered from the light source to the endoscope tip or generated by correspondingly powerful LEDs in the endoscope tip, the endoscope heats up considerably.This self-heating is often problematic, especially in medical applications. Furthermore, like an increase in electronic gain, it negatively impacts the signal-to-noise ratio of the image sensor. Therefore, especially in applications with very low signal responses, such as recording fluorescent light, today's endoscopes often operate at the limits of system performance.

[0003] Previously known image recording devices capable of detecting very low amounts of light, however, still suffer from insufficient image resolution for many applications, which is particularly problematic in the field of endoscopy. The size of the sensors used to date also often makes them unsuitable for endoscopy.

[0004] The scientific article "Super-resolution time-resolved imaging using computational sensor fusion," Sci Rep 11, 1689 (2021) by Callenberg et al. describes an approach for using a conventional camera in addition to a SPAD (silicon photon avalanche diode) array to significantly increase the spatial resolution of the SPAD array using a complex calculation based on the camera data.

[0005] In contrast, WO 2017 174 998 A1 describes a fiber-based endoscope with an additional spectrometer which, based on temporally modulated illumination, is able to provide additional information in addition to the pure imaging by means of the endoscope, for example to enable conclusions to be drawn about materials in the field of view of the endoscope based on the additional information.

[0006] Based on this, the invention aims to provide a novel approach for improved fluorescence imaging. In particular, the invention is intended to enable targeted, high-resolution imaging of very weakly emitting fluorescent light sources.

[0007] To achieve this object, the features of method claim 1 are provided. In particular, to achieve the object in a method as described above, the invention proposes that the image recording device used in the method, as already mentioned at the beginning, also has a single-photon sensitive detector (SPSD) in addition to the aforementioned image sensor, and that the image sensor and the SPSD (each) are configured to detect light from a common object area (which may in particular be located outside the image recording device). For this purpose, the image recording device can, for example, have common imaging optics that guide light from the common object area to the image sensor and to the SPSD. The image sensor and the SPSD can then be arranged corresponding to a common intermediate image plane of the imaging optics used.

[0008] To achieve this objective, the image sensor is configured for imaging in the visible wavelength range and the single-photon sensitive detector for fluorescence imaging in a non-visible wavelength range. Furthermore, the image sensor and the single-photon sensitive detector are arranged in a fixed spatial relationship to one another, and the image sensor is configured such that, in addition to wavelengths in the visible range, it also detects wavelengths in the ultraviolet or infrared wavelength range for the purpose of fluorescence imaging. These device features enable the two method steps described in claim 1 to be carried out: Step 1: To enable the detection of weak fluorescent light sources in deeper tissue layers, a fluorescence image is captured with the single-photon sensitive detector and superimposed on a white light image recorded with the image sensor. Due to its low signal intensity, the fluorescence image is not visible in the white light image recorded with the image sensor. Step 2: Subsequently, spatially high-resolution fluorescence light imaging is carried out with the image sensor in order to be able to optically examine the fluorescent tissue found in the first step with the help of the image sensor in higher spatial resolution.

[0009] It should also be noted that a field of view that can be captured with the SPSD may be different (in particular, smaller) than a second field of view that can be captured with the image sensor. However, in the common object area, the two fields of view overlap, meaning that both the SPSD and the image sensor capture the common object area.

[0010] The image sensor can be designed, for example, as a conventional RGB CMOS sensor or as a monochrome black-and-white image sensor. Furthermore, the image sensor (especially in an otherwise monochrome design) can also be equipped with additional spectral filters, particularly filter arrays, depending on the specific application, in order to obtain spectrally resolved image information.

[0011] This approach can also be applied, for example, to exoscopes, which - unlike endoscopes - are not inserted into body cavities, but are typically placed at a distance of, for example, 25 to 75 cm from the surgical field using a holding arm during open surgical procedures, in order to give the surgeon a large amount of free space in the working area.

[0012] In other words, it is proposed to capture image information from a common object area (which can thus also be located at some distance from the imaging optics) or from a common field of view in parallel and simultaneously using a conventional image sensor and an SPSD. Image information captured with the highly sensitive SPSD can also be used to enhance the high-resolution image of the image sensor in terms of image brightness and contrast.

[0013] While the SPSD thus offers high sensitivity, with currently still comparatively low spatial resolution, the image information captured by the image sensor can be used to improve the resolution of the image to be calculated and / or to provide required color information, for example, for creating a white light image. It is understood that said image recording device can accordingly comprise an image processing unit configured to increase the resolution of an image to be calculated and / or to calculate additional color information based on image information from the SPSD and the image sensor.

[0014] The invention thus recognizes that, for example, information about a specific fluorescent light source is not always required at a very high resolution. For example, it may be sufficient for a surgeon to be able to only approximately locate a very weak fluorescent light source, even hidden beneath additional tissue layers, using the image recording device. The surgeon can then use the higher-resolution visible image to decide how to approach the light source of the fluorescent light source for a more detailed examination (e.g., to examine malignant tissue marked with a fluorescent dye).

[0015] The advantage is that even in applications where very small amounts of light must be detected (such as conventional fluorescence imaging, fluorescence lifetime imaging microscopy (FLIM), or time-of-flight (ToF) measurements, particularly for generating 3D images), high-resolution images with good image quality in terms of signal-to-noise ratio (SNR) and image contrast can be obtained, while even very weak light intensities can still be detected. Single-photon-sensitive detectors (SPSDs), such as photomultipliers, typically have a light detection threshold that is orders of magnitude lower than that of conventional CMOS image sensors, meaning SPSDs are capable of detecting single photons.In recent years, the development of SPSD has progressed significantly and miniaturized SPSD based on CMOS technology are available, which offer a comparatively high spatial resolution.

[0016] The SPSD used in the image acquisition device can thus be designed as a single detector or in the form of an array, fully integrated as an image sensor chip, for example, based on individual detectors such as SPADs (silicon photon avalanche diodes) or SiPMs (silicon photomultipliers). Such technologies are therefore suitable for 2D imaging of single photons. Furthermore, the SPSD can also have on-chip evaluation electronics, typically based on CMOS technology.

[0017] According to the invention, by combining at least one conventional CMOS image sensor with an SPSD array, for example in the form of a SPAD array, the light yield can be drastically increased while simultaneously ensuring high resolution by utilizing the comparatively small pixels of the conventional image sensor. A high-resolution, bright, and high-contrast image can be generated from the measurement data of the at least one conventional CMOS image sensor and the SPSD array.

[0018] Claim 1 describes the corresponding application scenario, namely an “overlay” of fluorescence image data (low signal level) recorded with the SPSD with high-resolution white light image data recorded using the CMOS image sensor.

[0019] According to the invention, the object can also be achieved by further advantageous embodiments according to the subclaims, wherein details of the design of the image recording device used in the method are described below: Since, according to the invention, the image sensor and the SPSD are in a fixed spatial relationship to one another (or are arranged relative to one another), the calculation of the image is simplified. For example, the image sensor and the SPSD can be arranged spatially fixed relative to one another in a distal end region of the image recording device, for example, if the image recording device is designed as a chip-in-tip endoscope. Depending on the size, in particular of the SPSD and / or the image sensor, embodiments in which the image sensor and the SPSD are arranged spatially fixed relative to one another in a proximal end region of the image recording device can also be advantageous.

[0020] Another option, which can be used in addition or as an alternative, is to position the image sensor and the SPSD in a fixed position relative to a beam splitter. This allows the beam splitter to transmit light from the common object area to the image sensor and the SPSD. This makes it possible to use a common imaging optics for the image sensor and the SPSD, as previously explained.

[0021] A particularly preferred embodiment, which results in a compact design, provides that the image sensor and the SPSD are arranged on outer surfaces of the beam splitter and / or corresponding to a common intermediate image plane of an imaging optics.

[0022] The image sensor and the SPSD can be arranged either directly on the beam splitter (i.e. in contact with it) or with some air space from the beam splitter on the outer surfaces of the beam splitter, depending on the optical design of the imaging beam path. The respective imaging beam path is preferably designed so that the image sensor and the SPSD can image / detect a common object area. In this case, the SPSD can also image only a partial area of the image sensor's field of view, or vice versa. Depending on the design, it may also be necessary under certain circumstances for the image sensor and / or the SPSD to each have a slight offset in the direction of the respective optical axis of the imaging beam path. This is because different wavelength ranges to be detected (e.g. VIS vs. NIR range) can cause a shift in the image plane, which can then be compensated for accordingly by the offset.

[0023] As already mentioned, the image recording device can have a common imaging optics (in particular arranged in a distal end region of the image recording device or the endoscope / exoscope) that guides light from the common object area to the image sensor and the SPSD. Alternatively, two (or more) separate light guide channels can be formed in the image recording device / the endoscope / the exoscope, each of which guides light from the common object area to the image sensor or the SPSD.

[0024] According to a further specific preliminary embodiment, a stereo image recording device for generating 3D image data can also be obtained, which follows the inventive concept. This stereo image recording device has one image recording device configured according to the invention, each of which has an SPSD and an image sensor that detects light from a common object region. In this case, for example, only one (larger) image sensor can be used, the image sensor surface of which is divided into two sub-regions, which are then used as image sensors by the respective image recording device. The two image recording devices configured according to the invention of the stereo image recording device are configured for stereoscopic imaging. In other words, the two image recording devices can thus differ in a respective image recording angle (or viewing angle).This allows depth information and thus 3D image data to be obtained from a matching object area that is detected / observed by both image recording devices.

[0025] The image sensor can preferably be designed as an active pixel sensor (APS) with integrated amplifier circuit.

[0026] It can further be provided that the image sensor outputs a black-and-white image or a color image, in particular by using a color filter array (CFA). Such a CFA can be configured, for example, as a classic RGB Bayer pattern or using CMY color filters.

[0027] Furthermore, the image sensor can also be designed such that it detects wavelengths in adjacent wavelength ranges, in particular in the ultraviolet or infrared wavelength range, either in addition to wavelengths in the visible range or alternatively to wavelengths in the visible range.

[0028] To enable sufficient spatial resolution of very weak light signals such as fluorescent light pulses, it is advantageous if the SPSD is configured as a 2D array of individual, preferably fully integrated and / or semiconductor-based (e.g., based on silicon (Si) or gallium arsenide (GaAs)) detectors, each capable of detecting individual photons. As previously mentioned, these detectors can be configured, in particular, as SPADs (silicon photon avalanche diodes) or SiPMs (silicon photomultipliers) and arranged as a 2D detector array. In such configurations, the SPSD can thus provide 2D image data, which can be used, in particular, to enhance the display of 2D image data from the image sensor (image enhancement).

[0029] The SPSD can therefore be realized on the basis of single photon avalanche diodes (SPAD) and / or silicon photomultipliers (SiPM), in particular as a 2D SPAD array or as a 2D SiPM array.

[0030] The image sensor and / or the SPSD can furthermore each be implemented using CMOS technology, preferably as a fully integrated electronic component and / or with integrated signal processing electronics.

[0031] In the current state of the art, a sensible selection of components for the image recording device could be that the image sensor has pixels with a size of less than 5 µm, preferably less than 2 µm and / or wherein the SPSD can have pixels with a size of more than 5 µm, preferably more than 10 µm. This represents a currently common compromise between high resolution of the image sensor and high light sensitivity of the SPSD, while at the same time maintaining a compact design. However, it cannot be ruled out that in the near future, technological developments will further reduce the pixel size of SPSDs in particular, while maintaining the same good sensitivity, which would lead to a higher resolution of this signal component / image information.

[0032] Currently, the resolution of the image sensor can be at least a factor of 5, preferably at least a factor of 10, higher than the resolution of the SPSD; further increases in the near future do not appear to be excluded, which underlines the technical advantage of the concept presented here.

[0033] In certain applications, for example when the image recording device has a light source for emitting excitation light, it may be useful for the image recording device to have at least one optical filter, for example in the form of a bandpass filter or a bandstop filter.

[0034] Such a filter can be used to keep unwanted light from the object area, such as the excitation light required for fluorescence imaging, away from the SPSD and / or the image sensor. For example, a fluorescent marker can be excited with a specific wavelength or wavelength range and then emit fluorescent light with a different, higher wavelength. If this fluorescence response is to be detected, the excitation wavelength can be blocked by designing the filter as a band-stop filter. The filter can then filter out the excitation light from the respective imaging beam path (of the SPSD and / or the image sensor), thus preventing the excitation light from distorting the fluorescence response.

[0035] One or more such filters can, for example, ensure that the image sensor detects light exclusively in a first spectral range (particularly in the visible spectrum), while the SPSD detects light in a second spectral range that differs from the first spectral range (e.g., in the UV or NIR wavelength range). The filter can thus serve, in particular, to keep unwanted light away from the SPSD, preventing it from being flooded or saturated by such stray light. This allows the SPSD to selectively detect even the smallest amounts of light of the desired wavelengths with high sensitivity, thus far exceeding the typical sensitivity limit of conventional image sensors.

[0036] Said at least one optical filter can thus be configured / designed to select a (limited) wavelength range detected by the SPSD and / or the image sensor. In this case, the optical filter can be configured, for example, as a bandpass filter.

[0037] Thus, in particular, one such filter can be provided for the beam path of the SPSD and the beam path of the image sensor. These filters can, of course, have different characteristics, in particular different spectral transmission windows.

[0038] Of course, these approaches can also be combined, so that, for example, a narrow bandpass filter can be used to select the wavelength range captured by the SPSD and, at the same time, to exclude unwanted light, such as excitation light. Multibandpass or multibandstop filters can also be used as filters to allow or block multiple wavelengths.

[0039] According to a preferred embodiment, the image sensor is configured for imaging in the visible wavelength range (VIS). This enables, for example, a surgeon using the image recording device to quickly and easily navigate within a surgical field with the aid of VIS imaging. In such a case, the SPSD can be configured for imaging in a non-visible wavelength range, in particular in the UV or NIR range. This makes it possible, in particular, to capture additional spectral information with the SPSD, for example in the form of a fluorescence image superimposed on a white light image recorded with the image sensor.

[0040] In principle, it is possible to detect light in the NIR wavelength range with both the image sensor and the SPSD. However, the SPSD offers a significantly lower detection threshold, so that, especially at low light intensities in this wavelength range, which is of interest for many applications, only the SPSD is capable of detecting very weak NIR signals, such as NIR fluorescent light emitted by a specific tissue type, which is precisely what the method according to the invention exploits. The only disadvantage here is the lower resolution of the SPSD compared to the image sensor.

[0041] Nevertheless, it may make sense in practice to also design the image sensor so that it can still detect NIR wavelengths. This can be achieved, for example, by increasing the usual cut-off wavelength of a cut-off filter of the image sensor to more than 850 nm or (depending on the application) by omitting a cut-off filter altogether. With such a design of the image recording device, it can be advantageously used within the scope of the method according to the invention, for example, to visualize deeper-lying objects using fluorescence imaging provided by the SPSD, which cannot yet be "seen" / detected by the image sensor due to concealment by superficial tissue layers (and the associated signal attenuation).If the superficial tissue is then removed, the user can benefit from the higher resolution of the image sensor, as it can then also capture the NIR signal due to the now higher signal intensity of the fluorescent light. In other words, a surgeon can first roughly locate a fluorescent light source in the tissue using the SPSD and, after tissue resection, optically examine the fluorescent tissue more closely with the help of the image sensor, as described in claim 1.

[0042] The at least one optical filter can, for example, be implemented particularly simply on a beam splitter, in particular the one mentioned above, as an optical thin film.

[0043] A more advanced embodiment provides for the at least one optical filter to be actively tunable. In this case, the spectral range that passes through the filter and is subsequently detected by the SPSD can be actively selected. Depending on the tuning of the optical filter, different wavelengths (particularly at different times) can be detected by the SPSD. This is useful, for example, in applications where the SPSD is specifically designed to detect a first or a second fluorescence wavelength that is different from the first. This is because tuning the filter allows the sensitivity of the SPSD to be spectrally adjusted.

[0044] According to a particularly simple embodiment, a rotating filter wheel can also be provided, with which specific wavelengths can be selected. Such a configuration of the tunable optical filter may be appropriate, for example, if the image recording device is designed as an exoscope (e.g., a surgical microscope).

[0045] A further embodiment proposes that the image recording device has (at least) two separately arranged SPSDs. In this case, it is preferred if the two SPSDs detect different wavelength ranges, in particular due to two different optical filters (through which light passes to the respective SPSD). These two SPSDs can each be configured as described above.

[0046] The image recording device can also have (at least) two image sensors arranged (spatially) separately from one another. In this case, it is preferred if the two image sensors also detect different wavelength ranges / spectral ranges. For this purpose, the two image sensors can also have different color filters at the pixel level (i.e., different color filter arrays - CFAs).

[0047] Finally, as already mentioned, it is advantageous if the image acquisition device used (i.e., the endoscopic or exoscopic, possibly multispectral, image acquisition device) has an image calculation unit configured to calculate and output a synthetic image from signals of the image sensor and from signals of the SPSD. This facilitates the use of the image acquisition device.

[0048] Further details of the image recording method according to the invention are presented below. It should be noted in advance that the image recording device can, in particular, have the necessary means and can be configured to carry out one of the methods described below or a method according to one of the claims directed to a method, in particular in an automated manner.

[0049] To achieve the aforementioned object, it can be provided, for example, that a first image is recorded using the image sensor of the image recording device, that a second image is recorded using the single-photon sensitive detector (SPSD) of the image recording device, and that a synthetic image is calculated and output from the first image and the second image, preferably by said image calculation unit. This approach can be used to generate an "overlay" of fluorescence image data (typically with a low signal level) recorded using the SPSD with high-resolution white-light image data recorded using the image sensor. As described in claim 1, a fluorescence image that would normally not be visible in the white-light image due to the low signal intensity is thus captured using the SPSD and superimposed on the white-light image recorded using the image sensor.This overlay may, for example, also include a false color representation of the fluorescence image and / or the image components of the fluorescence image and the white light image may be weighted differently.

[0050] The method described above thus results in bright, high-contrast and high-resolution images that can display image information from different (non-overlapping) wavelength ranges, such as the VIS and NIR ranges, in order to offer the user enhanced imaging.

[0051] Since the resolutions of the two images will usually be different, it can be provided that the second image is scaled to the image resolution of the first image before synthesis (in particular, upscaled, namely when the resolution of the first image exceeds that of the second image). In the state of the art, image sensors usually have a significantly higher resolution than the SPSD, i.e., in order to generate an overlay of the images, for example, the resolution must be adjusted. Furthermore - as already described above - the image section recorded by the SPSD can be a subset of the field of view that can be recorded with the image sensor. In this case, the overlay of the image data from the SPSD can also only affect this image section; in this specific case, scaling must therefore only be carried out with regard to this image section, but not with regard to the entire image of the image sensor.

[0052] The synthetic image can also be obtained, for example, by blending, in particular combining, the first image with the second image.

[0053] Another optional variant of the process is to take into account not only color information but also alpha values as a measure of the transparency or opacity of the respective pixels when calculating the synthetic image, preferably using an alpha blending method.

[0054] The method is particularly advantageous when the SPSD detects fluorescent light (particularly autofluorescence light), as described in claim 1. In this case, the excitation light used for fluorescence imaging can be separated and filtered out from the fluorescent light by means of an optical filter, even before the fluorescent light reaches the SPSD. This allows the SPSD to selectively detect the fluorescent light from the excitation light. This is particularly useful when the excitation light is also partially used to illuminate the observed scene and is detected by the image sensor (for the desired white-light imaging).

[0055] Furthermore, in individual applications, it may also be provided that at least one third image, which was recorded with an additional image sensor or an additional SPSD of the image recording device, in particular in an additional wavelength range, is taken into account during the synthesis of the synthetic image. In such an embodiment of the method, in which multiple wavelength ranges are detected separately for individual pixels, the synthetic image can be output as a hyperspectral image.

[0056] In other applications such as FLIM, the SPSD can be used to record and evaluate temporal behavior, in particular the decay behavior of a fluorescent light source.

[0057] Another important application is to use the SPSD of the image capture device to perform a time-of-flight measurement. This can be achieved, for example, using well-known time-of-flight (ToF) methods, in which spatially resolved depth information is determined from an object area outside the image capture device. In this case, a depth map of the object area can be calculated from the depth information, which is of interest for certain applications. For this purpose, the image capture device can also be equipped with a light signal source that emits the light pulses required for the ToF method.

[0058] The invention will now be described in more detail using exemplary embodiments, but is not limited to these embodiments. Further developments of the invention can be derived from the following description of a preferred embodiment in conjunction with the general description, the claims, and the drawings.

[0059] In the following description of various preferred embodiments of the invention, elements which correspond in function are given the same reference numbers even if they have a different design or shape.

[0060] It shows: Fig. 1 an endoscopic image recording device according to the invention, the Fig. 2-4 show different possible optical arrangements that can be used in an image recording device according to the invention, Fig. 5 shows a further possible embodiment of an optical arrangement usable in an image recording device according to the invention with two separate light guide channels, Fig. 6 a flowchart for explaining an image recording method according to the invention and Fig. 7 a stereo image recording device designed according to the invention.

[0061] The Fig. 1 shows an image recording system 11 comprising an image recording device 1 according to the invention, which is designed in the form of an endoscope 1, a camera control unit (CCU) 12 and a monitor 13 on which the images recorded with the endoscope 1 can be viewed. The camera control unit 12 outputs the image data of the endoscope 1 to the monitor 13. The endoscope 1 has an optical arrangement 14 in its interior, which includes an image sensor 2 and a single-photon sensitive detector (SPSD) 3 and which, as in the examples according to the Fig. 2-4 or as in Fig. 5 can be designed.

[0062] In the Fig. 1, the optical arrangement 14 is arranged in a camera head 26 of the endoscope 1, i.e., just at a proximal end of the endoscope 1. Accordingly, the endoscope 1 has an endoscope shaft 18, in which an imaging optics 7 is arranged. With the imaging optics 7, image information from the Fig. 1 to the proximal end of the endoscope 1. The object area 4 can be captured both with the image sensor 2 and with the SPSD 3.

[0063] Since the image sensor 2 is located at the proximal end, the Fig. The endoscope shown in Figure 1 is thus a chip-in-scope endoscope. However, an endoscope 1 according to the invention can also be designed as a chip-in-tip endoscope, in which case the optical arrangement 14 with the image sensor 2 and the SPSD 3 is arranged in a distal end region 5 of the endoscope 1, for example, at the end of the endoscope shaft 18.

[0064] As the Fig. As shown in Figure 1, the light received from the common object area 4 is first transported by the imaging optics 7 (shown only schematically) to a beam splitter 6; this splits the imaging light into two separate beam paths, one leading to the image sensor 2 and the other to the SPSD 3. This ensures that there is a common object area 4 that lies outside the endoscope 1 and from which the light reaches the image sensor 2 and the SPSD 3.

[0065] As in the example of Fig. 4, the optical arrangement 14 or the endoscope 1 may have an optical filter 9 so that the SPSD 3 can detect a spectral range that differs from that detected by the image sensor 2.

[0066] When designing according to the Fig. 4, the optical filter 9 on the beam splitter 6 is realized as an optical thin film 10. Alternatively, such a filter 9 can also be formed in the working layer of the beam splitter 6 indicated by the dotted line, so that the beam splitter 6 is then dichroic.

[0067] If, however, the optical filter 9 is arranged, for example, at a distance from the beam splitter 6, it can in particular be designed to be actively tunable, so that different wavelengths can be detected by the SPSD 3 depending on the tuning of the optical filter 9. A further optical filter 9 can also be provided in the beam path leading to the optical image sensor 2, for example to exclude excitation light, which is used to generate fluorescent light, from the imaging.

[0068] Through the Fig. 2-4 but also in Fig. 5, it is ensured that the image sensor 2 and the SPSD 3 are each arranged in a fixed spatial relationship to each other. In the embodiments according to Fig. 2 and Fig. 3, this is achieved by placing the image sensor 2 and the SPSD 3 on the outer surfaces of the beam splitter 6, corresponding to a common intermediate image plane defined by the respective imaging optics 7. As a result, the beam splitter 6 directs the light from the common object area 4 to the image sensor 2 as well as to the SPSD 3. Unlike the example of Fig. 5, where two separate imaging optics 7 are formed, use the optical arrangements 14 according to the Fig. 2-4 thus only have a common light guide channel that guides light from the common object area 4 to the beam splitter 6.

[0069] When designing according to the Fig. 3, the beam splitter 6 has a total of three outer surfaces and two working layers (each illustrated by dotted lines) that split the incoming light into three different beam paths. As can be seen, the optical arrangement 14 therefore includes, in addition to a color image sensor 16, a black-and-white image sensor 17, with which additional image data from the common object area 4 can be captured.

[0070] Will be in Fig. 3 For example, if the black and white image sensor 17 is equipped with an additional filter 9, this image sensor 17 can capture a different spectral range than the color image sensor 16 (or also than the SPSD 3).

[0071] The Fig. 6 explains how the optical arrangements 14 of a respective endoscope 1 according to the invention, which are shown in the Fig. 2-5, can be used to implement an image recording method according to the invention using the respective endoscope 1. A first image is first recorded using one of the available image sensors 2. More precisely, raw data 19 from the image sensor 2 is captured by an image calculation unit 27 and processed into image data 20. The image calculation unit 27 can, for example, be integrated into the image sensor 2 itself or located in a camera control unit (CCU) 12, i.e., can also be arranged outside the image recording device 1 / the endoscope 1.

[0072] In parallel, raw data 21 acquired with the SPSD 3 is processed by the image calculation unit 27 into further image data 22. The calculation unit 27 then performs image synthesis 24, in which the image data 20, 22 are combined, which can take the form of an overlay, a false color display, or an image combination. The result of this calculation is a synthetic image 25, which can ultimately be viewed on the monitor 13.

[0073] A possible specific application, which may be achieved with the optical arrangement 14 according to Fig. 4 can be implemented, consists in acquiring 2D image data with the SPSD 3, selectively, for example, in an NIR wavelength range that is passed through a filter 9 designed as a bandpass filter. At the same time, high-resolution image data 20 can be acquired with the color image sensor 16. The color image sensor 16 can also have a cut-off filter, for example to filter out excitation light used in this fluorescence light imaging. This approach therefore allows, on the one hand, the fluorescence light to be acquired selectively and with high light sensitivity using the SPSD 3 and used to enhance the 2D image data of the color image sensor 16 or to overlay additional image information regarding the fluorescence light onto this image data. In order to obtain meaningful 2D information about the fluorescent light, the SPSD 3 is designed as a 2D single-photon avalanche diode array (SPAD array).The SPSD 3 is designed as a fully integrated electronic component / chip realized using silicon technology.

[0074] Due to the high light sensitivity of the SPSD 3, the endoscope 1 or the optical arrangement 14 can also be used to detect the decay behavior of a fluorescent light source or to perform a time-of-flight (ToF) measurement.

[0075] The Fig. Finally, Figure 7 shows a stereo imaging device 28 comprising two image recording devices 1a and 1b configured according to the invention, which are configured for stereoscopic vision / stereoscopic image recording with the aid of an additional imaging optics 7. With this approach, 3D image data can be obtained from the illustrated corresponding object area 4 observed by each of the two image recording devices 1a and 1b.

[0076] In summary, to improve imaging, particularly at low light intensities, the use of a specifically configured image recording device 1 is proposed, which is characterized in that, in addition to an image sensor 2, which uses photodiodes as light-sensitive cells, it comprises a single-photon-sensitive detector (SPSD) 3 in order to detect light from a common object area 4. With the help of the SPSD 3, additional image information can be obtained from the object area 4, which can be used to improve the image data recorded with the image sensor 2 or to enhance it with additional image information, in particular with regard to a further spectral range that is detected with the SPSD 3 (see Fig. 4). This can be advantageously used for imaging weak fluorescent light sources, as described in claim 1. List of reference symbols 1 image recording device (particularly designed as an endoscope or exoscope) 2 image sensors 3 SPSD - single photon sensitive detector 4 common object area 5 distal end area (of 1) 6 beam splitters 7 Imaging optics 8 2D imagers (array of photo detectors) 9 optical filters (especially bandpass filters) 10 optical thin film 11 Image acquisition system 12 Camera control unit 13 monitors 14 optical arrangement 15 Imaging beam path 16 color image sensor 17 Black and white image sensor 18 Endoscope shaft 19 raw data (of 2) 20 processed image data (of 2) 21 raw data (of 3) 22 processed image data (out of 3) 23 Image processing 24 Image synthesis (combination / overlay / intensity mapping etc.) 25 synthetic image 26 Camera head 27 Image calculation unit 28 Stereo image recording device

Claims

[1] Two-stage image recording method using an image recording device (1) designed as an endoscope or as an exoscope, which - at least one image sensor (2) whose pixels are based on photodiodes, and - an additional separate single-photon sensitive detector (3), wherein - the image sensor (2) and the single-photon sensitive detector (3) are designed to detect light from a common object area (4), - the image sensor (2) is configured for imaging in the visible wavelength range and the single-photon sensitive detector (3) is configured for fluorescence imaging in a non-visible wavelength range, - the image sensor (2) and the single-photon sensitive detector (3) are arranged in a fixed spatial relationship to each other, and - the image sensor (2) is designed such that it detects not only wavelengths in the visible range but also wavelengths in the ultraviolet or infrared wavelength range for the purpose of fluorescence imaging, and the method is characterized by the following steps: - in order to enable the detection of weak fluorescent light sources in deeper tissue layers, a fluorescence image is captured with the single-photon sensitive detector (3) and superimposed on a white light image recorded with the image sensor (2), the fluorescence image not being visible in the white light image recorded with the image sensor (2) due to its low signal intensity; - Subsequently, a spatially high-resolution fluorescence light imaging is carried out with the image sensor (2) in order to be able to optically examine a fluorescent tissue found in the first step with the help of the image sensor (2) in higher spatial resolution. [2] Method according to claim 1, wherein the image sensor (2) and the single-photon sensitive detector (3) are arranged in a fixed spatial relationship to each other with respect to a beam splitter (6), and the beam splitter (6) transmits light from the common object area (4) to the image sensor (2) and the single-photon sensitive detector (3), - particularly preferably wherein the image sensor (2) and the single-photon sensitive detector (3) are arranged on outer surfaces of the beam splitter (6). [3] Method according to claim 1, - wherein the image recording device (1) has a common imaging optics (7) which guides light from the common object area (4) to the image sensor (2) and to the single-photon sensitive detector (3), in particular wherein the imaging optics (7) is arranged in a distal end area (5) of the image recording device (1), or - wherein the image recording device (1) forms two separate light guide channels, each of which guides light from the common object area (4) to the image sensor (2) or to the single-photon sensitive detector (3). [4] Method according to claim 1, wherein the single-photon sensitive detector (3) is designed as a 2D array of individual, - preferably fully integrated and / or implemented on the basis of semiconductor technology, detectors which can each detect individual photons, so that the single-photon sensitive detector (3) provides 2D image data. [5] Method according to claim 1, - wherein the single-photon sensitive detector (3) is based on - single photon avalanche diodes (SPAD) and / or - silicon photomultiplier (SiPM) is realized, - in particular as a 2D single photon avalanche diode array or as a 2D silicon photomultiplier array. [6] Method according to claim 1, wherein the image sensor (2) and / or the single-photon sensitive detector (3) are realized by means of CMOS technology as fully integrated electronic components, in particular with integrated signal processing electronics. [7] Method according to claim 1, wherein the image recording device (1) has at least one optical filter (9) which is designed to - to keep unwanted light from the object area (4), in particular excitation light used for fluorescence imaging, away from the single-photon sensitive detector (3) and / or from the image sensor (2) and / or - to select a wavelength range which is detected by the single-photon sensitive detector (3) and / or the image sensor (2), - preferably wherein the at least one optical filter (9) on the beam splitter (6) is realized as an optical thin film (10). [8] Method according to claim 1, wherein the at least one optical filter (9) is actively tunable, - in particular so that, depending on the tuning of the optical filter (9), different wavelengths can be detected by the single-photon sensitive detector (3). [9] Method according to claim 1, - wherein the image recording device (1) has at least two separately arranged single-photon sensitive detectors (3), - preferably wherein the at least two single-photon sensitive detectors (3) detect different wavelength ranges, - particularly due to two different optical filters (9). [10] Method according to claim 1, - wherein the image recording device (1) has at least two image sensors (2) arranged separately from one another, - preferably wherein the at least two image sensors (2) detect different wavelength ranges, - in particular wherein the two image sensors (2) have different color filters at pixel level for this purpose. [11] Method according to claim 1, - wherein the image recording device (1) has an image calculation unit (27) which is configured to calculate and output a synthetic image from signals of the image sensor (2) and from signals of the single-photon sensitive detector (3).

Images