Method and system for stereoendoscopic fluorescence measurement and software program product

Abstract

Method for stereoendoscopic fluorescence measurement on tissue mixed with a fluorescent agent, in which a stereo optics (14) of a stereo video endoscope (12) is directed onto a region of the tissue (4) to be examined and the fluorescent agent is excited by means of an excitation light to emit fluorescent light, which is captured by the stereo optics (14) of the stereo video endoscope (12) in a stereo image pair or a sequence of stereo image pairs, characterized in that by means of an optical disparity of at least one pattern occurring in the stereo image pair or the stereo image pairs, using the stereo base and the stereo angle of the stereo optics (14) of the stereo video endoscope, a distance of the tissue (4) from the stereo optics (14) is determined and the fluorescence signal is normalized with a normalization factor dependent on the determined distance.

Description

[0001] The invention relates to a method and a system for stereoendoscopic fluorescence measurement on a tissue mixed with a fluorescent agent, in which a stereo optics of a stereo video endoscope is directed onto a region of the tissue to be examined and the fluorescent agent is excited by means of an excitation light to emit fluorescent light, which is captured by the stereo optics of the stereo video endoscope in a stereo image pair or a sequence of stereo image pairs.

[0002] Endoscopic fluorescence measurements are performed in many medical applications. Typical examples include near-infrared fluorescence imaging (NIRF), which can be used to analyze and evaluate blood vessel perfusion, confirm the anatomy of the hepatobiliary system, locate lymph nodes, or visualize the ureter after administration of an extrinsic contrast agent such as ICG (indocyanine green), Cy5.5, ZW800, or ZW-1. Red dichroic imaging (RDI) can be used to identify the origin of arterial bleeding. Narrow-band imaging (NBI) can help differentiate between benign hyperplasia and cancerous tissues or precancerous tissues, for example, between NICE-1 and NICE-2 colon polyps, to determine whether or not a polyp needs to be resected.

[0003] In certain applications of fluorescence imaging, e.g., in the assessment of tissue perfusion, quantification of the fluorescence signal may be in the interest of the user to determine functional levels at which wound healing or preservation of organ function is unlikely and therefore surgical intervention is necessary.

[0004] The associated concepts, both for requirements and limits and measurement approaches, have been described in the literature. Examples include: • Wada, T., et al. (2017), “ICG fluorescence imaging for quantitative evaluation of colonic perfusion in laparoscopic colorectal surgery”, Surgical endoscopy, 31(10), 4184-4193; • Kim, JC, et al. (2017), “Interpretative guidelines and possible indications for indocyanine green fluorescence imaging in robot-assisted sphincter-saving operations”, Diseases of the Colon & Rectum, 60(4), 376-384; • Hayami, S., et al. (2019), „Visualization and quantification of anastomotic perfusion in colorectal surgery using near-infrared fluorescence“, Techniques in coloproctology, 23(10), 973-980; • Son, G. M., et al. (2019), „Quantitative analysis of colon perfusion pattern using indocyanine green (ICG) angiography in laparoscopic colorectal surgery“, Surgical endoscopy, 33(5), 1640-1649.

[0005] One challenge in quantifying fluorescence is its dependence on various factors, including the dosage of circulating dye, which depends on the patient's blood volume and the amount of injected dye; tissue characteristics, particularly the patient's local anatomy—i.e., the thickness of the blood vessels and the thickness of other tissue layers covering the blood vessels in the observed region; and the distance between the surface of the imaging device, which emits the excitation light and also receives the emitted fluorescent light, and the observed object. The measured fluorescence signal decreases with the square of the distance as the distance increases.

[0006] To control the third factor, the measurement distance, several fluorescence imaging device manufacturers have developed approaches, but these have their drawbacks. These include the use of a fluorescence plate, which can be used as a reference based on known object size and known fluorescence properties (dye concentration) of the plate. However, a reference plate is another disposable item that must be prepared for the procedure, and its introduction into the surgical field can pose the risk of infection or leaving objects in the patient's body.

[0007] Optical guidance can also be used, for example, with two converging laser beams whose intersection marks the desired observation distance to be used for reference measurements. However, optical guidance using intersecting laser beams could limit the versatility of the device, as the observation distances cannot be freely selected. Furthermore, additional components must be integrated into the camera system.

[0008] Another proposed option is the integration of a dedicated proximity sensor to measure the distance between the tissue and the imaging device. However, a proximity sensor also represents an additional component that increases the cost of the device and could make it larger or bulkier.

[0009] The distance problem is circumvented by defining a reference region (RR) and a target region (ROI), and measuring a relative value for the ROI compared to the RR. However, this requires that a reference region is available within the field of view (FOV). And even if a reference region is available, it might be at a different distance from the imaging device than the ROI.

[0010] In contrast, the present invention is based on the object of providing a robust and structurally simple solution for quantitative fluorescence signal measurement.

[0011] The object underlying the invention is achieved by a method for stereoendoscopic fluorescence measurement on a tissue mixed with a fluorescent agent, in which a stereo optics of a stereo video endoscope is directed onto a region of the tissue to be examined and the fluorescent agent is excited by means of an excitation light to emit fluorescent light, which is captured by the stereo optics of the stereo video endoscope in a stereo image pair or a sequence of stereo image pairs, which is further developed in that by means of an optical disparity of at least one pattern occurring in the stereo image pair or the stereo image pairs, using the stereo base and the stereo angle of the stereo optics of the stereo video endoscope, a distance of the tissue from the stereo optics is determined and the fluorescence signal is normalized with a normalization factor dependent on the determined distance.

[0012] In the context of the present invention, a pattern is understood to be a structure that is contained in both images of the stereo image pair and is recognized as the same structure, so that disparity determination is possible.

[0013] The idea underlying the invention follows from the observation that many optical systems used for medical imaging are stereoscopic devices (e.g., WA50082A, LTF-S190-30, OrbEye). In a stereoscopic system, it is possible to determine observation distances based on the optical disparity. For this purpose, one or a plurality of feature points are detected in the image. By referencing these feature points in the left and right channels of the stereoscopic system, the disparity (offset of the two points in the imager coordinate system from each other) can be determined. Based on the known 3D parameters such as stereo base and stereo angle, the distance of the feature points to the imaging device can be determined. The distance determined with a 3D system from the disparity analysis is used to compensate for the fluorescence signal at different observation distances.

[0014] Preferably, the normalization factor is determined as the ratio of the square of the determined distance to the square of a standard distance. This takes into account the principle that, for geometric reasons, the intensity of a signal decreases with the square of the distance from the source. Alternatively, instead of a direct functional dependency, the normalization factor can be calibrated at the factory or during maintenance. In some embodiments, the normalization factor is calculated based on the determined distance or taken from a look-up table.

[0015] In one embodiment of the method, the distance to the stereo optics is determined for several points of the tissue, and a three-dimensional surface is linearly or non-linearly interpolated to the several points, or approximated using splines. In other words, this means that the distance between the stereo optics and the observed field of view is measured, and a cloud of feature points or points of patterns or structures is generated. A 3D surface is interpolated from the feature point cloud. The interpolation can be based on linear interpolation or on a non-linear interpolation method such as spline interpolation.

[0016] In one embodiment, the resulting three-dimensional surface is converted into a distance map containing specific distances for specific pixels or pixel areas of one or both stereo images, thus correlating specific distances for individual pixels or pixel areas of the image or images. Based on the distance, the variable normalization factor or gain value is calculated or retrieved from a lookup table that models the attenuation of the fluorescence signal as a function of the observation distance. The fluorescence signal of each pixel or pixel area is then multiplied or divided by the distance-dependent normalization factor, depending on the definition of the normalization factor. The result is a homogenized fluorescence image that is compensated for different observation distances, even at different observation distances within the field of view.

[0017] In a further embodiment, the distance to the stereo optics is determined for a predefined or adjustable, in particular central, measuring field of the stereo image, wherein the determined distance is used to determine the normalization factor for the entire image. The measuring field preferably has a linear extent of between 1% and 10% of the image height and / or, in the case of multiple recognized patterns in the central area, an average value of the distances determined for the various patterns is used. The measuring field can be predefined or set or selected by the user. There can be several predefined sizes of the measuring area, such as 1% image height, 5% image height, 10% image height or similar, which the user can select.

[0018] Based on this distance value, a variable normalization factor is calculated or retrieved from a lookup table, which models the attenuation of the fluorescence signal as a function of observation distance. The fluorescence signal of the entire image is then multiplied or divided by the distance-dependent normalization factor, depending on the definition of the normalization factor. The fluorescence signal is essentially the signal strength of the fluorescence light in each pixel or pixel area, measured as the difference from a baseline value without fluorescence light. This baseline value can be determined in a dark image for each pixel or pixel area. The difference of the signal from the baseline value is multiplied or divided by the normalization factor, depending on the definition of the normalization factor. The result is a homogenized fluorescence image that is compensated for different observation distances.

[0019] In an alternative embodiment of the method, a distance between at least one predefined or selectable target region (ROI) of the tissue and the stereo optics is determined, and additionally a reference distance of a predefined or selectable reference region (RR) of the tissue and the stereo optics is determined, wherein the normalization factor for the fluorescence signal in the measurement range is determined based on the difference or the ratio of the determined distance to the determined reference distance. Simultaneous measurement in an ROI and a reference region has the advantage over absolute measurement without a reference region that further interfering influences such as individual characteristics of the tissue or fluctuations in the dosage of the fluorescent dye are suppressed. For example, an area of the tissue that is assessed as normal can be used as the reference region, while an altered area is characterized as the ROI.

[0020] The positions and sizes of the measurement fields for ROI and RR can be predefined or user-defined. It is conceivable that there are several predefined area sizes, such as 1% image height, 5% image height, 10% image height, or similar, that the user can select. If different feature points are detected within the measurement region, an average value is calculated.

[0021] Based on the distance values of the RR (reference region) and the ROI (region of interest), the difference in observation distance between the ROI and the RR is calculated. A variable normalization value is calculated from this difference or from a look-up table that models the attenuation of the fluorescence signal as a function of the observation distance. This value is used to calculate the relative fluorescence signal intensity between the ROI and the RR to eliminate differences due to different observation distances between the ROI and the RR. In this case, the normalization value is a function of both observation distances and can be a function of the ratio of the observation distances, in particular the square of the ratio of the observation distances, a function of the difference between the observation distances, or a function approximating calibration measurements.

[0022] In a further embodiment, which can be combined with the previously mentioned embodiments, an illumination intensity distribution is additionally incorporated into the normalization of the fluorescence signals. This distribution is created during a calibration performed before the examination or in a subsequent step. In this embodiment, the previously described embodiments of the method are combined with a reference map that reflects the variable illumination intensity across the field of view. The variable illumination intensity is measured either after the video endoscope is manufactured or during a calibration step at the beginning of the medical procedure and stored in a memory of the endoscopic video system.The brightness of a pixel, pixel region, or measurement region is then multiplied by a correction factor to account for observed fluorescence signal differences resulting from different illumination intensities.

[0023] In embodiments, corrected images or fluorescence values are used to determine a maximum fluorescence, a maximum relative fluorescence, and / or a time to reach a maximum fluorescence, a maximum relative fluorescence, or a fraction thereof.

[0024] The object underlying the invention is also achieved by a system for stereoendoscopic fluorescence measurement, which comprises a stereo video endoscope with stereo optics and an excitation light source as well as an evaluation unit which is designed to carry out a method according to the invention described above.

[0025] Likewise, the object underlying the invention is also achieved by a software program product with program code means which are designed to carry out a previously described method according to the invention when the software program runs on an evaluation unit of a system according to the invention.

[0026] The system and the software program product embody the same features, properties and advantages as the method according to the invention.

[0027] Further features of the invention will become apparent from the description of embodiments of the invention together with the claims and the accompanying drawings. Embodiments of the invention may fulfill individual features or a combination of several features.

[0028] Within the scope of the invention, features marked with “in particular” or “preferably” are to be understood as optional features.

[0029] The invention is described below, without limiting the general inventive concept, using exemplary embodiments with reference to the drawings, whereby express reference is made to the drawings for all details of the invention not explained in more detail in the text. They show: Fig. 1 a schematic representation of a system according to the invention and Fig. 2 a schematic representation of the principle of distance measurement with a stereo video endoscope.

[0030] In the drawings, identical or similar elements and / or parts are provided with the same reference numbers, so that a repeated presentation is omitted.

[0031] Fig. 1 shows a schematic representation of a system 10 according to the invention. The system 10 comprises a stereo video endoscope 12, which is directed at the tissue 4 of a patient and records stereo image pairs with stereoscopically shifted images of the tissue 4 in the field of view of the stereo video endoscope 12 and transmits them to an evaluation unit 18. The tissue 4 is perfused with blood, wherein the blood is mixed with a fluorescent dye, which, when illuminated with an excitation light, emits fluorescent light toward the stereo video endoscope 12.

[0032] The stereo image pairs thus contain a qualitative depiction of the fluorescence activity of the imaged tissue 4. Since the intensity of the fluorescent light decreases with the square of the distance of the stereo video endoscope 12 from the tissue 4, it is possible to eliminate the influence of the distance on the measured intensity of the fluorescent light from the tissue 4 by measuring the distance of the tissue 4 from the stereo video endoscope 12 and using it to correct the fluorescence intensity.

[0033] In Fig. 1 shows two regions of tissue 4, namely a target region (ROI) and a reference region (RR), between which a relative measurement is to be performed. The goal of the measurement is to determine how much more or less intense the fluorescence is in the ROI compared to the RR. Such a relative measurement has the advantage over absolute measurements that further interference, such as individual characteristics of the tissue 4 or fluctuations in the dosage of the fluorescent dye, are suppressed. For example, a region of tissue 4 that is considered normal can be used as the reference region, while an altered region is characterized as the ROI.

[0034] In the example of Fig. 1, the distance of the stereo video endoscope 12 from the reference area RR is 35% greater than the distance to the target area ROI. To make the intensity of the fluorescent light in the target area ROI comparable with that in the reference area RR, the intensity of the fluorescent light in the target area ROI is multiplied by the square of the ratio of the distances d RR and d ROI , i.e. 1.35 -2 = 0.55.

[0035] In Fig. Figure 2 schematically illustrates the principle of stereoscopic distance determination. The distal tip 13 of the stereo optics 14 of the stereo video endoscope 12 has two light entry windows or entry lenses for the left and right stereo channels, which are spaced apart by a base distance B. Furthermore, the distal tip 13 has a light exit opening 16 for an illumination light, from which an excitation light emerges to excite fluorescence in a fluorescent dye that has previously been introduced into the tissue 4 of the organ 2.

[0036] The central rays of the two parallel optics point straight ahead and impinge on the surface of the tissue 4 of an organ 2 being examined at two different points. Alternatively, they can also have a known convergence angle to each other. A point P on the surface of the tissue 4 is perceived in the two channels at different angles to each other and therefore occurs at different locations x P l , x P r through a projection plane 15 or arrives at different locations on the image sensors or image sensor areas (not shown) for the left and right channels. The two values x P l , x P rare offset by various amounts from the image center. This difference depends on the distance z of point P from the stereo optics 14. Knowing the base distance B and the convergence angle between the two central rays of the left and right channels of the stereo optics 14, the distance z can be calculated. The observation distance is determined in the evaluation unit 18, which also performs image processing.

[0037] In this way, the distances to one or more points on the surface of the tissue 4 of the organ 2 can be determined using stereoscopic disparity analysis. The normalization of the fluorescence signals can be achieved in different ways. For example, if the distance of the tissue 4 to the stereo optics 14 in the field of view of the stereo video endoscope 12 varies only insignificantly, a common normalization factor can be used for the entire image. In the case of larger variations, a model of the surface of the tissue 4 can be created, and for each pixel or different pixel areas, it can be determined which part of the surface is imaged therein, and a corresponding normalization factor can be selected that corresponds to the distance of this part of the surface of the tissue 4.

[0038] Alternatively, a relative measurement can be made with respect to a reference range RR, as described in Fig. 1 is shown.

[0039] In addition, an illumination intensity distribution can be taken into account, in which both a known or previously determined illumination profile of the light source of the stereo video endoscope 12 and the attenuation of the intensity of the illumination light as a function of the distance of the tissue 4 from the distal tip 13 of the stereo video endoscope 12 are additionally taken into account.

[0040] Areas of the surface of the tissue 4 further away from the light exit opening 16 are stimulated to emit luminescent light with a lower illumination intensity than closer areas due to the greater distance. This attenuation of the illumination intensity follows the same functional relationship and is proportional to the square of the inverse of the distance. Since both the illumination intensity and the intensity of the luminescent light recorded by the video endoscope decrease quadratically with increasing distance, the luminescence signal, assuming a homogeneous illumination profile, attenuates, to a first approximation, with the inverse of the distance to the fourth power (d -4 ) away.

[0041] In this case, it can also be taken into account that outer areas of the field of view are less illuminated than central areas due to a possibly inhomogeneous illumination profile. In simple terms, by greatly simplifying the geometric conditions, the attenuation can be approximated as a function which, when the illumination profile f(ϑ, φ) is written in polar coordinates with d -4 f(ϑ, φ) when the illumination light exits in the immediate vicinity of the entrance of the stereo optics 14 at the distal tip 13 of the video endoscope 12, as in Fig. 2 shown.

[0042] All mentioned features, including those that can be inferred from the drawings alone, as well as individual features disclosed in combination with other features, are considered essential to the invention, both individually and in combination. Embodiments according to the invention may be fulfilled by individual features or a combination of several features. List of reference symbols 2 organs 4 fabrics 10 systems 12 Stereo video endoscope 13 distal tip 14 Stereo optics 15 Projection plane 16 Light exit opening 18 Evaluation unit

Claims

[1] Method for stereoendoscopic fluorescence measurement on a tissue mixed with a fluorescent agent, in which a stereo optic (14) of a stereo video endoscope (12) is directed onto an area of the tissue (4) to be examined and the fluorescent agent is excited by means of an excitation light to emit fluorescent light, which is captured by the stereo optic (14) of the stereo video endoscope (12) in a stereo image pair or a sequence of stereo image pairs, characterized by that by means of an optical disparity of at least one pattern occurring in the stereo image pair or pairs, using the stereo base and the stereo angle of the stereo optics (14) of the stereo video endoscope, a distance of the tissue (4) from the stereo optics (14) is determined and the fluorescence signal is normalized with a normalization factor dependent on the determined distance. [2] Method according to claim 1, characterized bythat the normalization factor is determined as the ratio of the square of the determined distance to the square of a standard distance. [3] Method according to claim 1 or 2, characterized by that the normalization factor is calculated based on the determined distance or taken from a look-up table. [4] Method according to one of claims 1 to 3, characterized by that the distance to the stereo optics (14) is determined for several points of the tissue (4) and a three-dimensional surface is interpolated linearly or non-linearly to the several points or approximated with splines. [5] Method according to claim 4, characterized by that the three-dimensional surface is converted into a distance map containing specific distances for specific pixels or pixel areas of one of the stereo images or both stereo images. [6] Method according to one of claims 1 to 3, characterized bythat the distance to the stereo optics (14) is determined for a predefined or adjustable, in particular central, measuring field of the stereo image, wherein the determined distance is used to determine the normalization factor for the entire image. [7] Method according to claim 6, characterized by that the measuring field has a linear extension of between 1% and 10% of the image height and / or in the case of several detected patterns in the central area, an average value of the distances determined for the different patterns is used. [8] Method according to one of claims 1 to 3, characterized bythat a distance of at least one predefined or selectable target area (ROI) of the tissue (4) to the stereo optics (14) and additionally a reference distance of a predefined or selectable reference area (RR) of the tissue (4) to the stereo optics (14) is determined, wherein the normalization factor for the fluorescence signal in the measuring range is determined on the basis of the difference or the ratio of the determined distance to the determined reference distance. [9] Method according to one of claims 1 to 8, characterized by that an illumination intensity distribution is additionally included in the normalization of the fluorescence signals, which is created during a calibration carried out before the examination or in a subsequent step. [10] Method according to one of claims 1 to 9, characterized bythat corrected images or fluorescence values are used to determine a maximum fluorescence, a maximum relative fluorescence and / or a time to reach a maximum fluorescence, a maximum relative fluorescence or a fraction thereof. [11] System (10) for stereoendoscopic fluorescence measurement, comprising a stereo video endoscope (12) with a stereo optics (14) and an excitation light source as well as an evaluation unit (18) which is designed to carry out a method according to one of claims 1 to 10. [12] Software program product with program code means which are designed to carry out a method according to one of claims 1 to 10 when the software program runs on an evaluation unit (18) of a system (10) according to claim 11.

Images