Method of measuring fluorescence signal and visible light image, and image capturing and processing device

Abstract

To provide, particularly: an improved method of measuring a fluorescence signal, the method capable of providing an improved fluorescence image output; an improved device for image capturing and processing; and an improved endoscope and laparoscope, and to provide an improved method of diagnosing lymphedema and an improved method of long-term therapy of lymphedema.SOLUTION: An image capturing and processing device (2) is configured to measure a fluorescence signal in a tissue of a body part (4), to which a fluorescent agent (8) has been added, and to image a surface (11) of the body part (4). The device (2) includes: an illumination unit (16); a fluorescence imaging unit (22); and a visible light imaging unit (24). The fluorescence imaging unit (22) and the visible light imaging unit (24) are configured such that a viewing direction and / or a view point of the fluorescence image and the visible light image are linked via a known relationship. A stitching unit (28) applies a stitching algorithm on the visible light images and similarly on the fluorescence images to generate a large visible light image and a large fluorescence image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention involves measuring the fluorescent signal in tissue of a body part that has been administered a fluorescent agent, and imaging the surface of the body part of which the administered tissue forms part. measurement Furthermore, the invention relates to a method adapted to measure a fluorescent signal in tissue of a body part administered with a fluorescent agent and adapted to image a surface of the body part, the tissue administered with the fluorescent agent forming part of the body part. measurement The present invention also relates to such a device. measurement The present invention relates to an endoscope or laparoscope equipped with the device.

[0002] Additionally, the present invention relates to methods for diagnosing lymphedema and methods for long-term treatment of lymphedema. [Background technology]

[0003] Books measurement Method and measurement The device is particularly directed to imaging and measuring lymphatic function, particularly with regard to the diagnosis, treatment and / or prevention of lymphedema.

[0004] Lymphedema is the accumulation of lymphatic fluid in the tissues of the body. While oxygenated blood is pumped from the heart to the tissues through the arteries, deoxygenated blood returns to the heart through the veins. Because the pressure level on the arterial side is much higher than on the venous side, the colorless fluid portion of the blood is forced into the spaces between the cells. Typically, more fluid is pushed out than is reabsorbed on the venous side. The excess fluid is transported by lymphatic capillaries. In addition, the fluid carries away local and foreign materials such as larger proteins and cellular debris. Once in the lymphatic system, this fluid, including the transported materials, is called lymph or lymphatic fluid.

[0005] The lymphatic system includes lymphatic vessels that have one-way valves, similar to those in veins, to transport lymph to the next lymph node. Lymph nodes perform the removal of certain substances and purify the fluid before it is returned to the bloodstream.

[0006] When the lymphatic system becomes obstructed because lymph flow is blocked or not running at the desired level, lymph fluid accumulates in the interstitial spaces between tissue cells. This accumulation is due to impaired lymphatic transport and is called lymphedema. The accumulation of lymph can cause an inflammatory response that damages the cells around the affected area. It can further cause fibrosis that can turn into hardening of the affected tissue.

[0007] Lymphedema is a lifelong condition for which there is no cure or drug therapy, so early diagnosis and appropriate early measures to improve drainage and reduce fluid volume are crucial to the patient's health and recovery. Possible treatments, ranging from lymphatic massage and compression bandaging to surgery, are based on severity, which is a four-tier system defined by the World Health Organization (WHO) as follows: Stage 1: Normal flow in the lymphatic system. No signs or symptoms. Stage 2: Fluid accumulation with swelling. Stage 3: Persistent swelling that does not resolve with elevation of the affected limb or body part. Stage 4: Elephantiasis (largely deformed limbs), thickening of the skin with "warty" growths and extensive scarring.

[0008] For the diagnosis of the functioning of the lymphatic system, a commonly used technique is manual examination of the affected limb or body part by a physician. A known imaging technique is lymphoscintigraphy, in which a radioactive tracer is injected into the tissue of the affected body part, followed by MRI (magnetic resonance imaging), CT (computed tomography), PET-CT scan (positron emission tomography) or ultrasound imaging.

[0009] A relatively new imaging technique is infrared fluorescence imaging using fluorescent dyes, such as ICG (indocyanine green). ICG is a green medical dye that has been used for over 40 years. This dye fluoresces when excited with near-infrared light having a wavelength between 600 nm and 800 nm. This excitation causes ICG to fluoresce between 750 nm and 950 nm. The fluorescence of the ICG dye can be detected using a CCD or CMOS sensor or camera. The fluorescent dye can be administered to tissues in the affected limb or affected part of the body, and based on the detected fluorescence, the concentration and flow of lymphatic fluid can be tracked. Summary of the Invention [Problem to be solved by the invention]

[0010] The object of the present invention is in particular to provide an improved method for measuring fluorescent signals, which is capable of providing an improved fluorescent imaging output. measurement Method, improved measurement The present invention aims to provide an improved endoscope or laparoscope.

[0011] It is a further object of the present invention to provide improved methods for diagnosing lymphedema and for the long-term treatment of lymphedema. [Means for solving the problem]

[0012] The objective is to measure the fluorescence signal in tissue of a body part that has been administered a fluorescent agent and to image the surface of the body part of which the administered tissue forms part. measurement This is solved by measurement The method is: Irradiating the tissue with excitation light having a wavelength suitable for exciting and emitting light from the fluorescent agent. death, capturing a fluorescence image by spatially resolved measurement of the emitted light to provide a fluorescence image; capturing a visible light image of at least a portion of a surface of the body part. P and firefly Optical and visible light images andRepeat the acquisition to obtain a series of fluorescent images and a series of visible light images. and providing a A set of stitching parameters is determined and applied Stitching algorithm A series of visible light images and applying the applied light to generate a large visible light image of the body part. P and , A stitching algorithm is applied to a series of fluorescent images to generate a large fluorescent image. P and , Large visible light image and large fluorescent image and outputting the fruit, the viewing directions and / or viewpoints of the fluorescent image and the visible light image are linked via a known relationship; The stitching algorithm applied to the series of fluorescence images uses a set of stitching parameters determined when generating the large visible light image. .

[0013] Advantageously, two images are obtained: a large visible light image and a large fluorescent image. and is output and, in contrast to conventional methods, these two images are linked to each other. In other words, the two images share a common coordinate system, which means that an object that can be seen in one of the images, for example one of the fluorescence images, can be found at the same position in the visible light image. The visible light image reproduces the surface of the body part that can be observed by the human eye. In the fluorescence image, the intensity of the fluorescence is reproduced. In particular, both images are displayed using the same image scale, image orientation and show the same area of ​​interest of the body part. This is due to the fact that both images are captured in particular with the same viewing direction and the same field of view. Based on the image information provided, the user is set to a position that will spot exactly on the real body part the point or position at which a particular fluorescence phenomenon is observed. This offers unparalleled advantages in terms of diagnosis and therapy.

[0014] Within the context of this specification, the terms "fluorescence image" and "visible light image" are not limited to 2D images. Images within the meaning of this specification may be 2D images as well as 3D images, or 2D images with additional information. 2D images can be captured using dedicated camera equipment. 3D images can be captured using, for example, stereo camera equipment. 2D or 3D images may be line scans, which can be captured using an image line scanner. Images may also be LIDAR scans, which also contain depth information as well as 3D images. Data from the LIDAR scanner can be added to the 2D image as additional information. The combination of 2D image data and LIDAR scan data results in information similar to a 3D image, since the 2D image data can be combined with the corresponding depth information. Also, the term "stitching" is not limited to the combination of two or more 2D images. Stitching can also be performed based on other image data mentioned above, for example 3D image data. Further information, such as depth information captured by a LIDAR scanner, can also be taken into account when performing the stitching algorithm.

[0015] Stitching can be performed based on two or more 2D images, resulting in a larger 2D image. Stitching algorithms can be performed based on multiple 3D images, resulting in a larger 3D image. It is also possible to perform a stitching step of processing multiple 2D images with additional information (e.g., from a LIDAR scanner, time-of-flight sensor, or similar device), resulting in a larger 3D image. In this case, the stitching step includes reconstruction of the 3D image from the additional information 2D image data. Image stitching includes identifying unique special features shown in the two images that correspond to each other and must be stitched together. This requires that the fields of view of the two images to be combined in the stitching process overlap at least slightly.

[0016] Taking into account this prerequisite for the stitching process, according to one embodiment of the present invention, the step of repeatedly capturing fluorescence images and visible light images to provide a series of fluorescence images and a series of visible light images is performed such that each subsequent image in the sequence is captured such that the fields of view of subsequent images at least slightly overlap.

[0017] However, the aforementioned additional information that can be used for the stitching process is not limited to, for example, depth information assigned to every individual pixel in the 2D image. During the image acquisition of the series of fluorescent and visible light images, the image capture device can also acquire as further data, data regarding the spatial orientation of the image capture device. The spatial orientation can be, for example, the position of the image in the coordinate system of the examination room. Import The orientation of the image capture device may be the orientation of the device. This orientation may be characterized by three spatial coordinates x, y, and z together with the viewing direction (e.g. a vector in the coordinate system of the examination room) and the tilt angle of the image capture device with respect to the viewing direction (e.g. a rotation angle of the image capture device with respect to the viewing direction). This information may be captured for each single image of the sequence of visible light and fluorescence images (or an image pair comprising a fluorescence image and a visible light image). This additional information indicating the orientation of the image capture device in space may be used when performing the stitching step. In particular, the orientation of the image capture device in, for example, the coordinate system of the examination room may be recalculated to the orientation of the image capture device with respect to the body part. This information may be used when performing a 3D reconstruction of the body part during the stitching step.

[0018] Furthermore, the stitching algorithm can be applied to any other application to generate large visible light images and large fluorescent images. In other words, the method is not specifically limited to the evaluation of the lymphatic system. The evaluation can be performed, for example, for blood vessels and blood flow, or for perfusion evaluation of organs and tissues. The evaluation also includes visually locating tissues with specific characteristics (such as tumor tissue), locating glands (such as parathyroid glands) and nerves. This list is not exhaustive.

[0019] The stitching algorithm can be applied to images obtained during open surgery or during minimally invasive surgery. The images captured and stitched during minimally invasive surgery can be obtained using an endoscope, such as an endoscope with a rigid shaft or an endoscope with a flexible shaft.

[0020] According to a further aspect of the invention, a method for measuring a fluorescent signal in a body part to which a fluorescent agent has been administered and imaging the surface of the body part is performed. measurement There is a way, this measurement The method is: Irradiating the body part with excitation light having a wavelength suitable for exciting the fluorescent agent to emit light. death, capturing a fluorescence image by spatially resolved measurement of the emitted light to provide a fluorescence image; capturing a visible light image of at least a portion of a surface of the body part. P and , Fluorescence and visible light images and Repeat the acquisition to obtain a series of fluorescent images and a series of visible light images. and providing a A set of stitching parameters is determined and applied Stitching algorithm A series of visible light images and applying the applied light to generate a large visible light image of the body part. P and , A stitching algorithm is applied to a series of fluorescent images to generate a large fluorescent image. P and , including fruit, the viewing directions and / or viewpoints of the fluorescent image and the visible light image are linked via a known relationship; The stitching algorithm applied to the series of fluorescence images uses a set of stitching parameters determined when generating the large visible light image. .

[0021] In particular, this measurement The method is: Large visible light image and large fluorescent image and The method further includes the step of outputting:

[0022] this measurement The method is not limited to the use of specific fluorescent agents and / or dyes. measurementThe method can be performed without the use of dyes by exploiting the effect of autofluorescence of certain tissues, such as parathyroid or intestinal tissue, and the absence of autofluorescence can be exploited to determine, for example, pathology.

[0023] According to this aspect of the invention, measuring an autofluorescence signal in tissue at or in a body part measurement A method is provided that does not require administration or application of a fluorescent agent to a tissue or body part. measurement The method includes imaging a surface of a body part in which tissue from which autofluorescence is measured forms part of the body part. measurement The method further comprises: Illuminating the tissue or body part with excitation light having a wavelength suitable for exciting autofluorescence of the tissue or body part to produce emitted light. death, capturing an autofluorescence image by spatially resolved measurement of the emitted light to provide an autofluorescence image; capturing a visible light image of at least a portion of a surface of the body part. P and , Autofluorescence and visible light images and Repeat the acquisition to obtain a series of autofluorescence images and a series of visible light images. and providing a A set of stitching parameters is determined and applied Stitching algorithm A series of visible light images and applying the applied light to generate a large visible light image of the body part. P and , applying a stitching algorithm to the series of autofluorescence images to generate a large autofluorescence image. fruit, the viewing directions and / or viewpoints of the autofluorescence image and the visible light image are linked via a known relationship; The stitching algorithm applied to the series of autofluorescence images applies a set of stitching parameters determined when generating the large visible light image.

[0024] this measurement The method is particularly well suited for large visible light images and large autofluorescence images. and The method further includes the step of outputting:

[0025] According to yet another aspect of the present invention, measurement The method is: applying a stitching algorithm to the series of fluorescence or autofluorescence images to generate a large fluorescence image or a large autofluorescence image, the stitching algorithm determining and applying a set of stitching parameters; The method includes applying a stitching algorithm to the series of visible light images to generate a large visible light image, the stitching algorithm applying a set of stitching parameters determined when performing stitching of the fluorescence or autofluorescence images.

[0026] In other words, a set of stitching parameters determined when performing stitching of a visible light image can be applied to a fluorescent image, and vice versa.

[0027] Moreover, it is within the scope of the methods described above that the fluorescent image is in fact an image detected in the visible light spectrum, for example when patent blue, methylene blue or isosulfan blue is applied as a fluorescent agent or dye, since the emission of this material is visible to the naked eye.

[0028] According to an advantageous embodiment of the invention, measurement The method includes the steps of superimposing a large visible light image and a large fluorescent image to provide an overlay image of the body part. Further includes , large visible light images and large fluorescent images In the step that outputs Outputting an overlay image do.

[0029] Advantageously, according to an aspect of the present invention measurementThe method provides an overlay image showing a visible light image of a body part affected, for example, with lymphedema, and a corresponding fluorescence image showing the concentration of lymph in individual areas of the affected body part. That is to say, the overlay image is a combination of a visible light image and a fluorescence image. The fluorescence image can be, for example, an image in which the intensity of the fluorescence is shown as a pseudocolor plot. The overlay image improves and simplifies the analysis of the situation of the lymphatic system. In conventional measurement methods for detecting fluorescence signals, typically there is no visible light image combined with the fluorescence image. However, this makes it very difficult to identify the exact location where abnormalities in lymphatic transport are detected, since in the fluorescence image they are visible as intensity spots, for example. This drawback becomes even more pronounced when different measurements are performed at different times or by different operators. Without a link to the visible light image, it is very difficult to compare fluorescence measurements taken at different times or by different people. If the visible light image is not combined with or linked to the fluorescence image, it is difficult to perform a follow-up of the patient or to monitor the progress of the treatment.

[0030] The step of stitching the fluorescence images advantageously comprises a reconstruction of the images into a 3D image. The stitching of the series of fluorescence images is performed based on a set of stitching parameters determined when previously performing the stitching of the visible light images.

[0031] Fluorescence images typically offer valuable and special features that make them suitable for performing the stitching process. Since the visible light images and the fluorescence images are linked by a known fixed relationship with respect to the viewing direction and / or viewpoint, the parameters of the stitching algorithm used for the visible light images can also be applied to the stitching of the fluorescence images. In other words, due to the fact that the visible light images and the fluorescence images are captured via optically aligned sensors, the same stitching parameters can be used for stitching the visible light images and stitching the fluorescence images. This significantly improves the stitching process, resulting in larger fluorescence images of better quality.

[0032] Fluorescent agents are, for example, ICG (indocyanine green) or methylene blue. Within the context of this specification, the term "fluorochrome" or "dye" (also called "fluorochrome" or "fluorophore") refers to the moiety of a molecule that makes the molecule fluorescent. This moiety is a functional group within a molecule that absorbs energy at a specific wavelength and re-emits the energy at another specific wavelength. In various embodiments, the fluorescent agent includes a fluorescent dye, an analog thereof, a derivative thereof, or a combination thereof. Suitable fluorescent dyes include, but are not limited to, indocyanine green (ICG), fluorescein, methylene blue, isosulfan blue, patent blue, cyanine 5 (Cy5), cyanine 5.5 (Cy5.5), cyanine 7 (Cy7), cyanine 7.5 (Cy7.5), cypate, silicon rhodamine, 5-ALA, IRDye700, IRDye800CW, IRDye800RS, IRDye800BK, porphyrin derivatives, Illuminare-1, ALM-488, GCP-002, GCP-003, LUM-015, EMI-137, SGM-101, ASP-1929, AVB-620, OTL-38, VGT-309, BLZ-100, ONM-100, BEVA800.

[0033] In embodiments in which the fluorescence is derived from autofluorescence, one or more of the fluorophores or agents causing the autofluorescence may be tissue endogenous fluorophores (eg, NADH, thyroid, parathyroid, etc.).

[0034] According to an aspect of the present invention measurement The method can be carried out with different fluorescers and dyes. Essentially any combination of suitable dyes and suitable or compatible excitation light sources can be applied. measurement The method can also be performed without the use of dyes, taking advantage of the effects of autofluorescence of certain tissues (eg, the parathyroid gland) or molecules.

[0035] Examples of dyes that are applicable and are primarily used in the lymphatic system include, for example, indocyanine green (ICG), methylene blue, isosulfan blue, or patent blue. Patent blue is visible to the naked eye. Therefore, image stitching can be performed without the need for fluorescence.

[0036] The stitching algorithm is, for example, a panoramic stitching algorithm that analyzes the images to extract special features that stand out. These features are then linked together in the multiple images and an image transformation (e.g., shift, rotation, stretching along one or more axes, or keystone correction) is performed. The positions of the linked features are used to determine image transformation parameters, also called stitching parameters. Following image orientation, the images are merged and thereby "stitched" together. This transformation can be performed similarly for both visible light and fluorescent images. Finally, the two images are output. This output can include, for example, displaying the images on a screen. For example, the two images are shown side by side. According to the above-mentioned embodiment, the two images are shown in an overlay image.

[0037] As mentioned above, the step of stitching a series of images is not limited to stitching 2D images. The step of stitching can also include, for example, the reconstruction of a 3D image based on a series of 3D images or based on a series of 2D images with additional information.

[0038] In many cases, the function of the lymphatic system is affected in the extremities of the body. However, the method of measuring the fluorescent signal is not limited to examination of the extremities. measurement The method generally refers to the examination of a body part, which may be a limb of the patient, but also the trunk, head, neck, back or any other part of the body. It is also possible that the body part is an organ, in which case the fluorescent signal is measured as an indicator of lymphatic flow. measurement The method may be performed during open surgery, as well as when the surgery is a minimally invasive procedure performed using an endoscope or laparoscope.

[0039] Within the context of this specification, a "visible light image" is an image of a real-world situation. It reproduces an image impression similar to what the human eye sees. Unlike the human eye, the visible light image may be a color image, a grayscale image, or a pseudocolor scale plot. The visible light image shows the surface of the body part including the tissue administered with the fluorescent agent. If the tissue is located on the surface of the body part, imaging of the surface of the body part includes imaging of the surface of the tissue.

[0040] According to another advantageous embodiment, a fluorescent image and a visible light image are and The viewing direction and viewpoint are the same. ,firefly Optical and visible light images and teeth ,same The light is captured through one, in particular a single, objective lens.

[0041] Advantageously, the fluorescence and visible light images can be captured with an image capture device comprising a prism assembly and a number of image sensors assigned thereto. The fluorescence and visible light enter the prism assembly as a common beam through one and the same entrance face of the prism assembly. The prism structure comprises a filter for separating the visible wavelength range from the infrared wavelength range in which the excitation emission of the fluorescent agent typically occurs. The different wavelength bands, i.e. visible light (also abbreviated Vis) and infrared light (also abbreviated IR), are directed to different sensors. The capture of the visible light and fluorescence images through only one objective lens allows a perfect alignment of the viewing direction and viewpoint of the two images. In particular, the viewing direction and viewpoint of the visible light and fluorescence images are identical.

[0042] In an advantageous embodiment, the fluorescent image Steps to capture and visible light images The step of taking in are performed simultaneously without time switching between the fluorescence image signal and the visible light image signal.

[0043] Advantageously, this measurement The method eliminates the need for time switching of signals. This is advantageous because the fluorescent, infrared and visible light images are captured at exactly the same time using separate image sensors. Images can therefore also be captured at high frame repetition rates, which makes this method ideal for situations requiring reliable hand-eye coordination. measurement This allows the application of the method. High frame rates of 60 fps and above are possible. High frame rates cannot usually be achieved if time switching is applied. Also, when capturing the fluorescence and visible light images with individual sensors, the sensors can be precisely focused and positioned. Furthermore, the sensor settings can be adjusted to the individual requirements for image acquisition of the visible light and fluorescence images. This concerns, for example, the adjustment of sensor gain, noise reduction, exposure time, etc.

[0044] According to yet another advantageous embodiment, the step of illuminating the tissue with excitation light and capturing a fluorescence image is It is possibleThe step of capturing a visible light image is performed by a single image capture device. When light illumination and image capture are integrated into one device, the overall process of measuring the fluorescent signal and simultaneously capturing a visible image can be improved.

[0045] According to an advantageous aspect of this embodiment, the method comprises the steps of: determining a surface of the body part captured in a visible light image; image The method further includes measuring the distance between the image capture device and, in particular, outputting a signal by the image capture device indicative of the measured distance. Measurements at different distances can be performed to optimize the light illumination and image capture to find the best image acquisition conditions. This best-fit distance can then be stored in the imaging system as the target distance for subsequent measurements.

[0046] According to one embodiment of the present invention, measurement The method can be further improved in that it includes the step of outputting a signal by the image capture device indicative of the measured distance. For example, a visual and / or audio signal can be output by the image capture device. This signal can guide the operator when handling the image capture device so that the image acquisition is performed at an at least approximately constant distance relative to the surface of the body part. The integration of a distance sensor in the image capture device with a user support output (visual or optical signal) allows the operator to capture images with a more uniform light illumination. This improves the quality of the measurement of the fluorescent signal.

[0047] To determine the best fit distance, measurement The method involves capturing fluorescent and visible light images of the same area of ​​the surface of a body part while measuring distance. and The method may further include repeatedly capturing multiple images at different distances. of A set of fluorescent and visible light images are captured. Then, depending on the quality of the images, Multiple A set of analyses is performed to determine the best matching distance that results in the highest quality image.

[0048] The output signal, which may be an audio or optical signal, may also indicate the deviation of the measured distance from the best match distance, thus directly informing the operator whether optimal image capture conditions, especially with regard to light illumination, have been applied during the measurement.

[0049] Furthermore, the image capture step can be performed by a robot or another automatic camera holder. In this embodiment, the output signal is used as an input to the robot or automatic camera holder to adjust to the best match distance. In this case, no signal output is required and the robot or camera holder moves automatically according to the stored best match distance.

[0050] According to yet another embodiment of the present invention, the measurement of the fluorescent signal is performed on tissue administered with at least a first and a second fluorescent agent, and the step of capturing the fluorescent image comprises: capturing a first fluorescence image in a first wavelength range produced by illuminating the tissue with a first excitation light having a first wavelength suitable for producing emission light due to a first excitation emission of a first fluorescent agent; capturing a second fluorescence image in a second wavelength range produced by illuminating the tissue with a second excitation light having a second wavelength suitable for producing emission light due to a second excitation emission of a second fluorescent agent; measurement The method is: Repeating the capture of the first and second fluorescent images to provide a series of first and second fluorescent images; and applying a stitching algorithm to the series of first and second fluorescent images to generate a first and second large fluorescent image. P and , outputting the first and second large fluorescent images; The stitching algorithm applied to the first and second series of fluorescent images employs a set of stitching parameters determined when generating the large visible light image.

[0051] According to the above embodiment, the fluorescent agent includes a first and a second fluorescent dye. The first fluorescent dye may be, for example, methylene blue, and the second dye may be ICG. According to this embodiment, the step of capturing the fluorescent image includes a step of capturing a first fluorescent image of the fluorescence emitted by the first fluorescent dye, and a step of capturing a second fluorescent image of the fluorescence emitted by the second fluorescent dye. The capturing of the two images is performed without any particular time switching. The first fluorescent image may capture an image in a wavelength range of 700 nm to 800 nm when using methylene blue as the first fluorescent dye. The second fluorescent image may capture an image in a wavelength range of 800 nm to 900 nm when using ICG as the second fluorescent dye. Fluorescence imaging based on two different fluorescent agents provides new possibilities for measurement and diagnosis.

[0052] The object is to provide a method for measuring a fluorescence signal in tissue of a body part administered with a fluorescent agent and for imaging a surface of the body part, the tissue of which forms part of the body part. measurement This is further solved by the device, measurement The device is a light irradiation unit configured to irradiate the tissue with excitation light having a wavelength suitable for exciting and emitting light from a fluorescent agent; a fluorescence imaging unit configured to capture a fluorescence image by spatially resolved measurement of the emitted light to provide a fluorescence image; and a visible light imaging unit configured to capture a visible light image of a portion of a surface of the body part. an image capture device including: A processing section; an output unit; Fluorescence imaging section and visible light imaging section and However, the fluorescence and visible light images and configured such that the viewing directions and / or viewpoints of the and Fluorescence and visible light images and The acquisition is repeated to provide a series of fluorescence images and a series of visible light images. In Made R, The processing unit includes: A set of stitching parameters is determined and applied Stitching algorithm A series of visible light images and applying a stitching algorithm to the series of fluorescent images to generate a large visible light image of the body part. In Stitching section completed Including, The stitching algorithm applied to the series of fluorescence images uses a set of stitching parameters determined when generating the large visible light image. The output section is It is configured to output large visible light images and large fluorescent images. do.

[0053] Measure the fluorescent signal measurement The same or similar advantages and advantageous aspects as those mentioned with respect to the method may be applied to the same or similar methods. measurement This applies to the device and will not be repeated here.

[0054] measurement The apparatus is configured such that the stitching unit is configured to apply the same stitching algorithm to stitching the visible light images and stitching the fluorescent images. The stitching algorithm may be configured to apply the same stitching algorithm to stitching the visible light images and stitching the fluorescent images. light Use the same stitching parameters that were determined and used for stitching the images. measurement The apparatus also includes a stitching unit configured to apply a stitching algorithm to stitch the visible light images using the same stitching parameters determined and used for stitching the fluorescent images. do.As mentioned above, stitching is not limited to stitching 2D images. It is also possible to perform stitching based on 3D images. Furthermore, stitching can include image reconstruction of 3D images. This can also be performed by the stitching unit. The stitching unit can be further configured to take into account additional information, for example for performing the 3D reconstruction. This additional information can be, for example, the orientation of the image capture device, or depth information that forms part of the 3D image or is the result of a scanning procedure. According to an embodiment of the present invention, the image capture device comprises a scanner for scanning the surface of the body part, for example a LIDAR scanner, a time-of-flight camera, or any other equivalent device.

[0055] this measurement The device is not specifically limited to evaluating the lymphatic system. measurement The device may be configured to perform, for example, vascular and blood flow assessment, or perfusion assessment of organs and tissues. Assessment may also include visually locating tissue with particular characteristics (such as tumor tissue), locating glands (such as the parathyroid gland) and nerves. This list is not exhaustive.

[0056] moreover, measurement The device is not limited to the use of fluorescent agents and / or dyes. measurement The device can operate without the use of dyes by exploiting the effect of autofluorescence of certain tissues, such as the parathyroid gland. Thus, according to a further aspect of the invention, a device configured to measure autofluorescence signals in tissue of or on a body part is provided. measurement There is a device, this measurement The apparatus is further configured to image a surface of the body part, the tissue forming part of the body part, measurement The device is a light irradiation unit configured to irradiate the tissue with excitation light having a wavelength suitable for generating emitted light by excited self-luminescence; a fluorescence imaging unit configured to capture an autofluorescence image by spatially resolved measurement of the emitted light to provide an autofluorescence image; a visible light imager configured to capture a visible light image of a portion of a surface of the body part; an image capture device including: A processing section; an output unit; the fluorescence imaging unit and the visible light imaging unit are configured such that the viewing directions and / or viewpoints of the autofluorescence images and the visible light images are linked via a known relationship, and configured to repeatedly capture the autofluorescence images and the visible light images to provide a series of autofluorescence images and a series of visible light images; The processing unit includes: A set of stitching parameters is determined and applied Stitching algorithm A series of visible light images and configured to apply the same to generate a large visible light image of the body part. ,and A stitching algorithm is applied to a series of autofluorescence images to generate a large autofluorescence image. In Made Ta Stitching section Including, The output section is Configured to output large visible light images and large autofluorescence images do.

[0057] According to another embodiment, measurement The apparatus further comprises an overlay unit configured to overlay the large visible light image and the large fluorescent image to provide an overlay image of the body part, and the output unit is and Output the overlay image as the output of And It will be further improved in that respect.

[0058] According to yet another advantageous embodiment, measurement The device consists of a fluorescent imaging section and a visible light imaging section. and However, the fluorescence and visible light images and In particular, the fluorescent image and the visible light image are configured so that the viewing direction and the viewpoint are the same. Same as This is further improved in that the image is adapted to be captured through a single objective lens.

[0059] moreover, measurement The device consists of a fluorescent imaging section and a visible light imaging section. and However, the fluorescence image and the visible light image can be obtained without time switching between the fluorescence image signal and the visible light image signal. andThe improvement is that the image processing apparatus is configured to simultaneously capture

[0060] Furthermore, the image capture device having a fluorescence imaging portion and a visible light imaging portion further includes a dichroic prism assembly configured to receive the fluorescence and the visible light via the input surface, the dichroic prism assembly including a first prism, a second prism, and a first compensator prism disposed between the first prism and the second prism; a further dichroic prism assembly for splitting the visible light into three light components; and a second compensator prism disposed between the second prism and the further dichroic prism assembly; The first prism and the second prism each have a cross section with at least five corners, each corner having an included angle of at least 90 degrees. , pieces having separate entrance and exit faces , this A direction parallel to the normal to the incident surface Enter The incident beam incident on the incident surface is No. 1 Prism and each of the second prisms is reflected twice inside the To Through the exit surface No. 1 Prism and from each of the second prisms It is designed to emit No. 1 Prism and each of the second prisms The normal to the incident surface and the normal to the exit surface are perpendicular to each other, When light enters the first prism through the input surface, the light is partially reflected toward the output surface of the first prism. , No. The light travels a first path length from the input face of the first prism to the output face of the first prism, passes through the first compensator prism, partially enters the second prism, and is partially reflected toward the output face of the second prism. , No. traverse a second path length from the input face of the first prism to the output face of the second prism; The first prism is larger than the second prism so that the first and second path lengths are the same.

[0061] Advantageously, the five prism assemblies described above allow for capturing two fluorescence imaging wavelengths and three colors for visible light imaging, e.g., red, blue and green. The five prism assemblies are advantageous in that the optical path of the light going from the entrance face to each one of the sensors has the same length. Thus, all sensors are in focus and furthermore, there are no timing gaps between the signals of the sensors. Advantageously, measurement The device does not require time switching of the received signal, which allows high quality image capture at high frame rates.

[0062] According to yet another aspect, an image capture device including a fluorescence imaging portion and a visible light imaging portion defines first, second and third optical paths for directing the fluorescence and the visible light to the first, second and third sensors, respectively, the image capture device further comprising a dichroic prism assembly configured to receive the fluorescence and the visible light via an entrance surface, the dichroic prism assembly including a first prism, a second prism and a third prism, each prism having a first, second and third exit surface, respectively, the first exit surface being provided with a first sensor, the second exit surface being provided with a second sensor and the third exit surface being provided with a third sensor, in particular, the first optical path being provided with a first filter, the second optical path being provided with a second filter and the third optical path being provided with a third filter; The first, second, and third filters may be, in any order, a green filter, an infrared filter, and a red / blue pattern filter that includes red and blue filters in an alternating pattern such that half of the light received by the red / blue pattern filter passes through the blue filter and half of the light received by the red / blue pattern filter passes through the red filter.

[0063] Furthermore, according to another aspect, the first, second, and third filters are, in any order, a red / green / blue pattern filter (RGB filter), a first infrared filter, and a second infrared filter, in particular, the first and second infrared filters have different light transmission wavelengths.

[0064] In other words, the first and second infrared filters are for filtering IR light in different IR wavelength intervals, for example a first IR band in which a typical fluorescent dye emits a first fluorescence peak and a second IR band in which a typical fluorescent dye emits a second fluorescence peak. Typically, the second IR band is located at a higher wavelength compared to the first IR band. The first and second infrared filters can also be adjusted to the emission bands of different fluorescent agents. Thus, for example, the emission of the first fluorescent agent can pass through the first filter (particularly blocked by the second filter) and be detected by a corresponding first sensor, and the emission of the second fluorescent agent can pass through the second filter (particularly blocked by the first filter) and be detected by a corresponding second sensor. For example, the first filter can be configured to measure the fluorescence emission of methylene blue and the second filter can be configured to measure the fluorescence emission of ICG.

[0065] measurement The device can also be further improved in that the light irradiation unit, the fluorescence imaging unit and the visible light imaging unit are arranged in a single image capture device, and the image capture device further comprises a measurement unit configured to measure the distance to the surface of the body part captured in the visible light image, and in particular the image capture device is configured to output a signal indicative of the measured distance.

[0066] Also, with respect to the embodiment, the fluorescent signal is measured measurement The same or similar advantages and advantageous aspects as mentioned for the method apply.

[0067] This object is related to one or more of the above-mentioned embodiments. measurement It is further solved by an endoscope or laparoscope configured as an image capture device in the device. measurement The device can be used in open surgery as well as during endoscopic or laparoscopic surgery. measurement Method and / or measurementThe same or similar advantages as discussed with respect to the device apply in the same or similar manner to an endoscope or laparoscope.

[0068] This object is further achieved by a method for diagnosing lymphedema, comprising: administering a fluorescent agent to a body part; measuring a fluorescent signal in tissue of the body part administered with a fluorescent agent and imaging a surface of the body part, the tissue administered with the fluorescent agent forming part of the body part; capturing a fluorescence image by illuminating the tissue with excitation light having a wavelength suitable to cause emission light by excitation emission of a fluorescent agent, and by spatially resolved measurement of the emission light to provide a fluorescence image; capturing a visible light image of at least a portion of a surface of the body part; capturing a visible light image, where the viewing directions and / or viewpoints of the fluorescent image and the visible light image are linked via a known relationship; repeating the steps of capturing the fluorescent images and the visible light images to provide a series of fluorescent images and a series of visible light images; applying a stitching algorithm to the series of visible light images to generate a large visible light image of the body part, the stitching algorithm determining and applying a set of stitching parameters; applying a stitching algorithm to the series of fluorescence images to generate a large fluorescence image, the stitching algorithm applying a set of stitching parameters determined when performing stitching of the visible light images; outputting a large visible light image and a large fluorescent image; and deriving a diagnosis related to the severity of lymphedema by analyzing the output image.

[0069] The method for diagnosing lymphedema can be advantageously performed with higher accuracy and reliability, and therefore provides better results. This entirely new approach can advantageously replace the classical method of diagnosing lymphedema. The traditional method of diagnosing lymphedema is the manual examination of the affected body part by a physician. However, this method of performing a diagnosis necessarily includes a non-reproducible and random element due to the physician's personal experience and qualifications. Furthermore, the method for diagnosing lymphedema includes the same or similar advantages as previously described with respect to the method of measuring a fluorescent signal.

[0070] The method of diagnosing lymphedema is further improved in that the fluorescent agent is administered to the patient's arm or leg by injecting the fluorescent agent into the phalanges or interphalangeal tissue of the patient's foot or hand. Typically, the arms and / or legs are affected by lymphedema. Thus, application of the novel and successful method of diagnosing lymphedema is particularly useful when performed on these limbs.

[0071] The object is also solved by a method of long-term therapy of lymphedema, comprising a step of diagnosing the severity of lymphedema by carrying out the steps according to the aforementioned method of diagnosing lymphedema in a patient. administering a therapy to the patient, adjusting the therapy according to the diagnosis of lymphedema severity; and repeating the steps of diagnosing the severity of lymphedema and administering a therapy to the patient, adjusting the therapy in each iteration according to the detected severity of lymphedema.

[0072] The method of long-term therapy according to the present invention is particularly useful because the diagnosis of lymphedema, in contrast to conventional methods, provides an objective result regarding the severity of the disease. The success of the long-term therapy can be analyzed from an objective perspective. Thus, the analysis and diagnosis are more helpful when considering the success of the therapy.

[0073] Further features of the present invention will become apparent from the following detailed description of the embodiments of the present invention taken together with the appended claims and the accompanying drawings, in which: The embodiments of the present invention may embody individual features or a combination of several features.

[0074] The invention is described below on the basis of exemplary embodiments, without restricting the general idea of ​​the invention. Express reference is made to the drawings for the disclosure of all details according to the invention that are not described in detail in the text. [Brief description of the drawings]

[0075] [Figure 1] 1 shows a schematic diagram of a measurement device. [Diagram 2] 1 shows a schematic diagram of the image capture device and processing unit of the measurement device. [Diagram 3] 1 shows an example of a visible light image and a fluorescent image corresponding to the visible light image. [Figure 4] 3(a) and 3(b) show a large overlay image generated in part from the visible light and fluorescent images shown in FIG. 3(a) and FIG. 3(b). [Diagram 5] FIG. 2 shows a schematic diagram illustrating the internal prism assembly of the image capture device. [Figure 6] 1 shows a schematic diagram of an endoscope or laparoscope including an image capture device. [Figure 7] 1 shows a flowchart of the stitching algorithm. [Figure 8] 1 shows a schematic diagram illustrating another internal prism assembly of an image capture device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0076] In the drawings, elements of the same or similar type or corresponding parts are given the same reference numbers so as to avoid the need to reintroduce the items.

[0077] FIG. 1 illustrates a fluorescent light source configured to measure fluorescent signals in tissue of a body part 4 of a patient 6. measurementThe device 2 is shown. Merely by way of example, the body part 4 of the patient 6 to be examined is the arm. The measurement of the fluorescent signal can also be performed on other body parts 4 of the patient 6, e.g. the leg, parts of the head, the neck, the back or any other part of the body. This measurement can also be performed during open surgery. In this application scenario, the body part 4 can for example be an internal organ of the patient 6. The measurement of the fluorescent signal can also be performed during minimally invasive surgery. In this application scenario, measurement The device 2 is at least partially integrated into, for example, an endoscope or laparoscope, which for example comprises an image capture device 10.

[0078] Before starting a measurement for the first time, a fluorescent agent 8 is administered, ie injected, into the tissue of the patient's body part 4 . measurement As may be described with reference to a diagram illustrating an apparatus 2, the method for measuring a fluorescent signal in tissue of a body part 4 excludes the step of administering a fluorescent agent 8.

[0079] The fluorescent agent 8 is, for example, ICG. ICG (indocyanine green) is a green medical dye that has been used for over 40 years. ICG emits fluorescence when excited with near-infrared light having a wavelength of 600 nm to 800 nm. The emitted fluorescence is 750 nm to 950 nm. It is also possible that the fluorescent agent 8 contains two different medical dyes. For example, the fluorescent agent 8 may be a mixture of methylene blue and ICG.

[0080] After administration of fluorescent agent 8, as shown by the arrow in FIG. measurement An image capture device 10 forming part of the apparatus 2 is used to examine a body part 4 of a patient.

[0081] The image capturing device 10 is configured to image the surface 11 of the body part 4 and to detect a fluorescent signal resulting from the irradiation of the fluorescent agent 8 with excitation light. When the image capturing device 10 is applied in surgery, the surface 11 of the body part 4 is, for example, the surface of an internal organ. In this case, the surface 11 of the body part 4 is identical to the surface of the tissue administered with the fluorescent agent 8. To emit light having a suitable excitation wavelength, the image capturing device 10 comprises a light irradiating unit 16 (not shown in FIG. 1 ).

[0082] The captured image is similarly measurement The results of the analysis are output, for example for display on a display 14 of the processing unit 12. The image capture device 10 can be operated by a physician 3.

[0083] Figure 2 shows measurement 1 is a schematic diagram showing in more detail the image capture device 10 and the processing unit 12 of the apparatus 2. The image capture device 10 comprises a light irradiation unit 16 configured to irradiate the tissue with excitation light having a wavelength suitable for exciting the emission of the fluorescent agent 8, thereby causing fluorescence. For example, the light irradiation unit 16 comprises a number of LEDs.

[0084] The image capture device 10 further comprises an objective lens 18 for capturing visible light and fluorescent light. The light is directed through the objective lens 18 to a prism assembly 20. The prism assembly 20 is configured to separate the fluorescent light, particularly in the wavelength range of 750 nm to 950 nm, from the visible light resulting in a visible light image. The fluorescent light is directed to a fluorescence imager 22, for example a CCD or CMOS sensor with additional wavelength filters and electronics as required. The fluorescence imager 22 is configured to capture the fluorescent image by spatially resolved measurement of the emitted light, i.e. the excited emission of the fluorescent agent 8, to provide a fluorescent image. In addition, there is a visible light imager 24, which may be another CCD or CMOS sensor with additional wavelength filters and electronics as required. The prism assembly 20 is configured to separate the fluorescent light from the visible light imager 24, so as to enable the visible light imager 24 to capture a visible light image of a portion of the surface 11 of the body part 4 of the patient.24 Prism assembly 20 is configured to direct visible light to a fluorescent light imaging portion 22. Similarly, prism assembly 20 is configured to direct fluorescent light to a fluorescent light imaging portion 22. Prism assembly 20, fluorescent light imaging portion 22 and visible light imaging portion 24 are described in further detail below.

[0085] In one embodiment, image capture device 10 is a scanning unit, such as an image line scan unit or a LIDAR scanning unit. Image capture device 10 may also be a 3D camera suitable for capturing a pair of stereoscopic images from which a 3D image including depth information can be computed. Of course, image capture device 10 may also be a combination of these.

[0086] Image data is communicated from image capture device 10 to processing unit 12 via a suitable data link 26, which may be a wireless data link or a wired data link, such as a data cable.

[0087] Image capture device 10 is configured to operate fluorescence imaging unit 22 and visible light imaging unit 24 to capture visible light images and fluorescence images simultaneously. In particular, image capture device 10 does not perform time switching between the fluorescence image signal and the visible light image signal. In other words, the sensors of fluorescence imaging unit 22 and visible light imaging unit 24 are used exclusively to capture images in their respective wavelength ranges, i.e., the sensors of imaging units 22, 24 are used to capture fluorescence images in the IR spectrum or are used to capture visible light images in the visible spectrum. Imaging unit 22, 24 are not used to capture images in both wavelength ranges. This brings great advantages. For example, the sensor can be positioned with precise focus, which is not possible if an image sensor is used for both purposes, i.e. to capture visible and infrared light. This is because the position of the focus for these different wavelengths is usually different. Furthermore, the sensor parameters can be adjusted individually, for example with regard to the required exposure time or the sensor gain. Since the IR signal is usually lower than the visible light signal, an individual setting is advantageous.

[0088] The fluorescent imaging section 22 and the visible light imaging section 24 have a fixed spatial relationship to each other since they are located within a single mounting structure or frame of the image capture device 10. Additionally, the fluorescent imaging section 22 and the visible light imaging section 24 use the same objective lens 18 and prism assembly 20 for capturing the fluorescent and visible light images, respectively. With these provisions, the fluorescent imaging section 22 and the visible light imaging section 24 are configured such that the viewing directions and viewpoints of the fluorescent and visible light images are linked in a known and fixed relationship. In a given embodiment, Imaging As both sections 22, 24 are imaged through the same objective lens 18, the viewing direction of the two images is identical.

[0089] The image capture device 10 is further configured to operate the fluorescence imager 22 and the visible light imager 24 to repeatedly capture the fluorescence images and the visible light images to provide a series of fluorescence images and a series of visible light images. This operation can be performed by the processing unit 12 operating the image sensor of the fluorescence imager 22 and the image sensor of the visible light imager 24. The series of images is typically captured while an operator or physician 3 (see FIG. 1 ) moves the image capture device 10 along the longitudinal direction L of the body part 4 of the patient 6. This movement can be performed such that subsequent images of the series of images include overlapping portions. In other words, details shown in a first image of the series of images are also shown in a subsequent second image of the series. This is important for the subsequent stitching process. The frequency of image acquisition can be set to a sufficiently high value to ensure that corresponding features can be found in the subsequent images. The image capture can be initiated manually, for example by the physician 3, or the image capture can be controlled by the image capture device 10, since the described prerequisites are met.

[0090] The image capture device 10 may be further configured to acquire a position and orientation of the image capture device 10 during this movement. For example, the position and orientation of the image capture device 10 in the frame of reference of the examination room or the frame of reference of the patient 6 may be determined for each captured image or image pair. This information may be stored and communicated along with the image or image pair including a visible image and a fluorescent image. This information may aid in subsequent image reconstruction to generate a 3D image from a series of 2D images.

[0091] Once the doublet of images (i.e. a first series of visible light images and a second series of fluorescent images) or series of image pairs (each image pair including a fluorescent image and a visible light image) have been captured by the capture device 10 and received by the processing unit 12, the series of visible light images are processed by the stitching unit 28 (see FIG. 2). The stitching unit 28 is configured to apply a stitching algorithm to the series of visible light images to generate a large visible light image of the body part 4. The large image is "larger" in that it shows in a single image a larger portion of the body part 4 of the patient 6 that is analyzed by the image capture device 10.

[0092] Within the context of this specification, the term "stitching" should not be understood as limiting the stitching process to the combination of two or more 2D images. Stitching can also be performed based on 3D images, which results in a larger 3D image. Stitching can also be performed based on 2D images with additional information about the viewing direction from which the 2D images are captured. Further information about the position of the image capture device 10 can also be taken into account. As mentioned above, based on these data sets, a larger 3D image can be generated, i.e., from a series of 2D images with information about the position and orientation of the image capture device 10, stitched together into a larger 3D image. It is also possible to combine 3D scan data, for example from a LIDAR sensor, with 2D image information. In this case, too, the result of the stitching process is a larger 3D image.

[0093] In all applications, the stitching algorithm starts with stitching of visible light images. The stitching algorithm generates and applies a set of stitching parameters when performing the stitching operation. The detailed operation of the stitching unit 28 is further described below. The stitching unit 28 is configured to apply the stitching algorithm not only to a series of visible light images, but also to a series of fluorescence images to generate a large fluorescence image. Again, the stitching process is not limited to the combination of two or more 2D images. It is also possible to generate a 3D fluorescence image in a similar manner as described above for the visible light images.

[0094] The stitching algorithm applied to stitch the fluorescence images is the same as that used for stitching the visible light images. Moreover, the stitching of a series of fluorescence images is performed based on the same set of stitching parameters determined when performing the stitching of the visible light images. This is possible because there is a fixed relationship between the viewing directions and viewpoints of the visible light images and the fluorescence images. Of course, if the viewing directions and viewpoints of the visible light images and the fluorescence images are not identical, a fixed offset or shift of the stitching parameters needs to be applied. This takes into account the known and fixed spatial relationship between the IR and Vis image sensors and the corresponding optical systems.

[0095] Following stitching, a large visible light image and a large fluorescence image are output. The images are displayed, for example, side-by-side on the display 14. Unlike conventional examination systems, the display 14 shows the visible light image and the fluorescence image, which correspond to each other. In other words, details visible on the fluorescence image, for example high fluorescence intensity indicating lymphatic fluid accumulation, can be found exactly at the corresponding location of the patient's body part 4, shown in the visible light image. This allows the physician 3 to pinpoint exactly the area where lymphatic fluid accumulation is present. This is highly valuable information, for example, for a customized specific therapy of the patient 6.

[0096] It is also possible to superimpose the visible light image and the fluorescence image, in particular a large visible light image and a large fluorescence image, to provide an overlay image of the body part 4, in particular in a large overlay image. This is performed by a superimposition unit 30 of the processing unit 12. The overlay image can also be output via the display 14.

[0097] Figure 3 of( a) shows an example of a visible light image 5 in which a portion of the surface 11 of a body part 4 of a patient 6 is visible. 6 A portion of the leg of the of( b) Patient 6 3 shows a corresponding fluorescence image 7 determined by measuring the fluorescence signal of a fluorescent agent 8 administered to the leg tissue of a patient. Spots or areas of high intensity of the fluorescence signal are visible. 6 This is highly suggestive of lymphatic fluid accumulation due to slow lymphatic transport in the leg and possible lymphedema. Advantageously, the physician 3 can now pinpoint the location of the area where slow lymphatic transport occurs by comparing the fluorescent image 7 with the visible light image 5.

[0098] In Fig. 4, there is an overlay image 9, and in Fig. 3 of( a) and Figure 3 of( In addition to the image shown in b), stitching of a visible light image 5 and a fluorescent image 7 was performed.

[0099] 4, each projected between the dashed lines shown in a larger overlay image 9. By stitching the visible light image 5 and the fluorescent image 7 together, a large overlay image 9 can be provided that shows substantially the entire body part 4 of the patient 6. The fluorescent signal can be shown in pseudocolor to clearly distinguish it from features in the visible light image 5.

[0100] FIG. 5 is an embodiment of the prism assembly 20 of the image capture device 10. The first prism P1 is a pentagonal prism. The incident light beam A, which is visible light and fluorescent light, enters the first prism P1 through the incident surface S1 and is partially reflected at the surface S2, which is one of the two surfaces not adjacent to the incident surface S1. The reflected beam B is then reflected to the first surface adjacent to the incident surface S1. The angle of reflection can be less than the critical angle so that the reflection is not internal (the adjacent surface can be coated to avoid light leakage and reflect the required wavelengths of interest). The reflected beam C then intersects with the incident light beam A and exits the first prism P1 from the second surface adjacent to the incident surface S1 toward the sensor D1. A portion of the beam A passes through the surface S2 and enters the compensation prism P2. Two non-internal reflections can be used to direct the incident beam A to the sensor D1 via beams B and C. Additionally, there may be no air gap between prisms P1 and P2, no air gap between prisms P3 and P4, and no air gap between prisms P2 and P3. Prism P2 is a compensator prism for adjusting the individual lengths of the optical paths from input face S1 to sensors D1-D5.

[0101] Beam D is prism From P2, the beam enters a second pentagonal prism P3. As in prism P1, internal reflections are used to make the beam cross itself. For brevity, the description of the beams will not be repeated except to state that in prism P3, beam portions E, F, and G correspond to beam portions A, B, and C, respectively, in prism P1. Prism P3 can also not use internal reflections to reflect the incident beam towards sensor D2. Two non-internal reflections can be used to direct incident beam E towards sensor D2 via beams F and G.

[0102] After prism P3 there is another compensation prism P4. Finally, beam H enters a dichroic prism assembly comprising prisms P5, P6 and P7 with sensors D3, D4 and D5 respectively. The dichroic prism assembly is for splitting the visible light into red, green and blue components for the respective sensors D3, D4 and D5. Light enters the prism assembly via beam I. prism An optical coating C1 is disposed between prisms P5 and P6, and another optical coating C2 is disposed between prisms P6 and P7. Each optical coating C1 and C2 has a different reflectance and wavelength sensitivity. Optical Coatings At C1, the incident beam I is partially reflected back (beam J) to the same facet of the prism where the light entered, where again a beam, now labelled K, is reflected towards sensor D3. beam From J beam The reflection to K is an internal reflection. Therefore, sensor D3 receives light reflected by coating C1, and sensor D4 receives light reflected by coating C2. C2 prism (beams M and N), and sensor D5 receives light from beam O that traverses the prism unimpeded.

[0103] There is an air gap between prisms P4 and P5. In prism assembly 20, the following total path lengths can be defined for each end channel (defined with respect to the sensor at the end of the channel): Sensor D1 (e.g., first near infrared) path: A+B+C Sensor D2 (e.g., second near infrared) path: A+D+E+F+G Sensor D3 (e.g., red) path: A+D+E+H+I+J+K Sensor D4 (e.g., blue) Path: A+D+E+H+I+ L+M+N Sensor D5 (e.g., green) Path: A+D+E+H+I+ L + O

[0104] Since the path lengths are consistent, A+B+C=A+D+E+F+G=A+D+E+H+l+J+K=A+D+E+H+ I+L+M+N =A+D+E+H+I+ L + O It is.

[0105] Matching the path lengths can include adjusting for differences in focal positions of the focal planes of the wavelengths detected at the sensors D1-D5. That is, for example, the path length towards a sensor for blue (B) light may not be exactly the same as the path length towards a sensor for red (R) light, because the ideal distance for forming a sharply focused image depends to some extent on the wavelength of the light. The prisms can be configured to allow for these dependencies. The length of D+H can be adjusted as a result of wavelength shifts due to lateral displacement of the compensator prisms P2, P4 to act as a focal compensator.

[0106] For focus shift and compensation, the larger air gap in path I can be used for additional filters or filled with glass compensators. Due to internal reflections in the path from beam J to beam K, an air gap must exist at that particular base face of the red prism. Space must be provided between the prism output face and each of the sensors D1-D5 to provide additional filters or fill with glass compensators accordingly.

[0107] The sensors D1 and D2 are IR sensors configured for capturing the fluorescence image 7. As an example, the sensors D1 and D2 with suitable electronics are part of the fluorescence imaging section 22. The sensors D3, D4 and D5 are for capturing the three components of the visible light image 5. As an example, the sensors D3, D4 and D5 with suitable electronics are part of the visible light imaging section 24. One can also consider corresponding prisms directing light beams to the sensors that are part of each section, i.e., the fluorescence imaging section 22 and the visible light imaging section 24, respectively.

[0108] 6 shows a schematic diagram of an endoscope 50 or laparoscope according to an embodiment of the present invention. When considering aspects of the present invention, the difference between a laparoscope and an endoscope is relatively small. Thus, where the description refers to an endoscope, a laparoscopic configuration is generally also possible. By way of example only, in the following, reference is made to an endoscope 50.

[0109] Endoscope 50 includes image capture device 10, which has been described in greater detail above. Image capture device 10 includes objective lens 18 that captures fluorescent image 7 and visible light image 5. Objective lens 18 focuses light incident through entrance face S1 of prism assembly 20 onto sensors D1-D5. Objective lens 18 may also be integrated into the last portion of the endoscope section to match the prism back focal length.

[0110] The endoscope 50 comprises an optical fiber 52 connected to a light source 54 that couples light into the endoscope 50. The light source 54 can provide white light for illumination of the surface 11 of the body part 4 and for capturing the visible light image 5. Furthermore, the light source 54 can be configured to emit excitation light suitable for exciting the fluorescent dye administered as a fluorescent agent to emit fluorescence. In other words, the light source 54 can be configured to emit both visible light and light in the IR spectrum.

[0111] Inside the shaft 56 of the endoscope 50, the optical fiber 52 is split into several fibers 51. The endoscope 50 can have a flexible shaft 56 or a rigid shaft 56. In the rigid shaft 56, a lens system consisting of lens elements and / or relay rod lenses can be used to guide the light through the shaft 56. If the endoscope 50 has a flexible shaft 56, the fiber bundle 51 can be used to guide the light of the light source 54 to the tip of the endoscope shaft 56. A fiber bundle 58 is placed in the shaft 56 of the endoscope 50 to guide the light from the distal end of the endoscope shaft 56 (not shown in FIG. 6) coming from the examination site to the image capture device 10 at the proximal end of the shaft 56. In another embodiment not shown, the entire image capture device 10 can be miniaturized and placed at the distal or end of the endoscope shaft 56.

[0112] FIG. 7 shows a flow chart of a stitching algorithm that can be used for stitching visible light and fluorescent images. The flow chart is more or less self-explanatory and will be explained rather briefly. First, the acquired sequence of images (S1) is transferred to the stitching unit 28 of the processing unit 12. Then, the algorithm performs a frame pre-selection (step S2). In this pre-selection step, frames suitable for stitching are selected. S3 represents the selected images to be stitched, which are then subjected to pre-processing (step S4). Within the pre-processed images (S5), feature extraction is performed (step S6). Once the image features have been extracted (S7), image matching (step S8) is performed using the known image from S3 and the features extracted from step S7. Based on the selected images (S9), a transformation of the image is estimated (step S10). This estimate of the image transformation, also called stitching parameters (S11), is applied (step S12). Applying the transformation transforms the image (S13). Further image corrections, such as exposure correction, can be performed (step S14). The transformed and corrected images (S15) are stitched together by locating the seams, i.e. the lines where the images are joined together (step S16). Data indicating the location of the seams (S17) are used together with the transformed and corrected images (S12) to create a composite of the images (step S18). In a given embodiment, this results in a large visible light image or a large fluorescent image as the stitching result (S19).

[0113] The image capture device 10 applied for capturing the visible light image 5 and the fluorescence image 7 may further comprise a measurement unit 32 configured to measure a distance d (see FIG. 1 ) between the surface 11 of the patient's body part 4 captured in the visible light image 5 and the image capture device 10. A distance sensor 33 in communication with the measurement unit 32 is, for example, an ultrasonic sensor, a laser distance sensor or any other suitable distance measuring device. Furthermore, the image capture device 10 is configured to output a signal indicative of the measured distance d. For example, the image capture device 10 outputs an optical or acoustic signal that gives the operator of the device 10 information about the best distance d for performing the measurement. Performing the measurement at a constant distance d significantly improves the measurement result, especially due to the presence of a uniform illumination.

[0114] In addition to the distance sensor 33, the image capture device 10 can include an internal measurement unit (IMU) that can be used to collect data regarding pitch, yaw, and roll rotations, as well as acceleration data in the three spatial axes (x, y, and z). This information from the IMU can be used as additional data to improve the performance of the stitching algorithm, or can be used to provide feedback to the operator to position and rotate the camera to provide a better image to the stitching algorithm.

[0115] 8 shows an embodiment of another prism assembly 20 of the image capture device 10. The prism assembly 20 includes, for example, prisms P5, P6, and P7 configured to split the light into red, green, and blue components for respective sensors D3, D4, and D5. According to a further embodiment, the prism assembly 20 is configured to split the incident light into a green component, a red / blue component, and an infrared component and direct them to the respective sensors D3, D4, and D5. According to yet another embodiment, the prism assembly 20 is configured to split the incident light into a visible light component that is directed to a red / green / blue sensor (RGB sensor), a first infrared component of a first wavelength or wavelength interval, and a second infrared component of a second wavelength or wavelength interval and direct them to the respective sensors D3, D4, and D5.

[0116] Light enters prism assembly 20 as indicated by the arrows. prism An optical coating C1 is disposed between prisms P5 and P6, and an optical coating C2 is disposed between prisms P6 and P7, with each optical coating C1 and C2 having a different reflectance and wavelength sensitivity. Optical Coatings At C1, the incident beam I is partially reflected back (beam J) to the same facet of prism P5 where the light entered, where again a beam, now labelled K, is reflected towards filter F3 and sensor D3. beam From J beam The reflection to K is an internal reflection. Thus, filter F3 and sensor D3 receive light reflected by coating C1, and filter F4 and sensor D4 likewise receive light reflected by coating C2. C2 Filter F5 and sensor D5 receive light from beam L that is reflected by the prism (beams M and N). Filter F5 and sensor D5 receive light from beam O that traverses the prism unimpeded.

[0117] With reference to the embodiment in which the incident light is split into red, green and blue components, the coatings and filters are selected accordingly.

[0118] In an embodiment where the incoming light is separated into green, red / blue and infrared components, filter F3 can be a pattern filter (red / blue). In particular, red and blue filters are arranged in an alternating pattern. The pattern can consist of groups of 2x2 pixels that are filtered to obtain one particular color. Filter F4 can be a green filter, meaning a filter that contains only green filters. There is a single pixel grid where the light received at each pixel is filtered with a green filter. Filter F5 can be an IR filter. Each pixel is filtered with an IR filter.

[0119] Coatings C1, C2 should typically be matched with filters F3, F4, F5. For example, the first coating C1 may reflect IR light while transmitting visible light so that IR light is directed towards IR filter F3. The second coating C2 may reflect red and blue light while being transparent to green light such that filter F4 is a red / blue pattern filter and F5 is a green filter 23.

[0120] According to a further embodiment in which the incident light is split into a visible light component (RGB), a first infrared component and a second infrared component, the coatings C1, C2 and the filters F3, F4, F5 are configured such that for example the sensor D4 is a color sensor (RGB sensor) that detects all three color visible light images. Furthermore, the sensor D3 can be configured to detect fluorescence at a first wavelength and the sensor D5 is configured to detect fluorescence at a second wavelength.

[0121] Similarly, referring to the prism assembly 20 of FIG. 5, the coatings S1, S2, S3, S4, C1 and C2, and filters F1, F2, F3, F4 and F5, which are disposed in front of each one of the sensors D1, D2, D3, D4 and D5, can be configured to detect up to four fluorescence wavelengths. For example, sensor D4 is a color sensor for detecting visible light images of all three colors. Sensor D3 is for detecting fluorescence of a first wavelength or wavelength interval, sensor D5 is for detecting fluorescence of a second wavelength or wavelength interval, sensor D1 is for detecting fluorescence of a third wavelength or wavelength interval, and sensor D2 is for detecting fluorescence of a fourth wavelength or wavelength interval.

[0122] All specified features, including those obtained solely from the drawings, and individual features disclosed in combination with other features, are considered to be important to the invention both alone and in combination. The embodiments according to the invention can be realized through individual features or combinations of several features. Features combined with the expressions "especially" or "specially" should be treated as preferred embodiments. [Explanation of symbols]

[0123] 2. Image capture and processing equipment 3. Doctor 4 body parts 5 Visible light images 6 patients 7. Fluorescence Images 8 Fluorescent Agents 9. Overlay Images 10 Image capture device 11 Surface 12 Processing section 14 Display 16 Light irradiation unit 18 Objective Lens 20 Prism Assembly 22 Fluorescence imaging unit 24 Visible light imaging unit 26 Data Links 28 Stitching Section 30 Superimposed section 32 Measuring part 33 Distance Sensor 50 Endoscopy 52 Optical Fiber 51 Fiber 54 Light source 56 Shaft 58 Fiber Bundle P1 First pentagonal prism P2, P4 compensation prism P3 Second Pentagonal Prism P5, P6, P7 Dichroic Prism Assemblies A Incident light beam B ~ O Light beam S1 entrance plane D1 ~ D5 Sensor C1, C2 coating F1 ~ F5 Filter L Longitudinal d Distance

Claims

1. 1. A method for measuring a fluorescent signal in tissue of a body part administered with a fluorescent agent and imaging a surface of the body part, said tissue administered with the fluorescent agent forming part of said body part, comprising: illuminating the tissue with excitation light having a wavelength suitable to cause emission light by excitation emission of the fluorescent agent, and capturing the fluorescence image with an image capture device by spatially resolved measurement of the emission light to provide a fluorescence image; capturing a visible light image of at least a portion of a surface of the body part with the image capture device; repeating the capturing of the fluorescent images and the visible light images to provide a series of fluorescent images and a series of visible light images; applying a stitching algorithm to the series of visible light images, wherein a set of stitching parameters is determined and applied to generate a large visible light image of the body part; applying the stitching algorithm to the series of fluorescence images to generate a large fluorescence image; and outputting the large visible light image and the large fluorescent light image; the viewing directions and / or viewpoints of the fluorescent image and the visible light image are linked via a known relationship; A measurement method, wherein the stitching algorithm applied to the series of fluorescence images applies the set of stitching parameters determined when generating the large visible light image.

2. and superimposing the large visible light image and the large fluorescent light image to provide an overlay image of the body part; The measurement method according to claim 1 , wherein the step of outputting the large visible light image and the large fluorescent image includes outputting the overlay image.

3. The measurement method according to claim 1 , wherein the viewing direction and the viewpoint of the fluorescent image and the visible light image are the same, and the fluorescent image and the visible light image are captured via a same objective lens.

4. The measurement method of claim 1 , wherein the steps of capturing a fluorescent image and capturing a visible light image are performed simultaneously without time switching between the signals of the fluorescent image and the visible light image.

5. The measurement method described in claim 4, wherein the step of irradiating the tissue with excitation light and capturing the fluorescence image and the step of capturing the visible light image are performed by a single image capturing device.

6. The method of claim 5 further comprising the step of measuring a distance between a surface of the body part captured in the visible light image and the image capture device.

7. The method of claim 6 further comprising the step of outputting, by the image capture device, a signal indicative of the measured distance.

8. repeatedly capturing the fluorescent images and the visible light images of the same portion of the surface of the body part while measuring the distance, wherein multiple sets of the fluorescent images and the visible light images are captured at different distances; The method of claim 6 further comprising: analyzing the plurality of sets according to image quality and determining a best matching distance that results in the highest quality of the image.

9. The method of claim 8 , wherein the image capture device outputs a signal indicative of a deviation of the distance from the best match distance.

10. said measuring said fluorescent signal is performed on tissue administered with at least a first and a second fluorescent agent; The step of capturing a fluorescent image further comprises: capturing a first fluorescence image in a first wavelength range produced by illuminating the tissue with a first excitation light having a first wavelength suitable for causing emission light due to a first excitation emission of the first fluorescent agent; capturing a second fluorescence image in a second wavelength range produced by illuminating the tissue with second excitation light having a second wavelength suitable for producing emission light due to a second excitation emission of the second fluorescent agent; The measuring method comprises: repeating said capturing of said first and second fluorescent images to provide first and second series of fluorescent images; applying the stitching algorithm to the first and second series of fluorescence images to generate first and second large fluorescence images; and outputting the first and second large fluorescent images. The measurement method of claim 1 , wherein the stitching algorithm applied to the first and second series of fluorescent images applies the set of stitching parameters determined when generating the large visible light image.

11. 1. A measurement device configured to measure a fluorescent signal in tissue of a body part administered with a fluorescent agent and configured to image a surface of the body part, the tissue administered with the fluorescent agent forming part of the body part, comprising: a light irradiation unit configured to irradiate the tissue with excitation light having a wavelength suitable for exciting and emitting light from the fluorescent agent; a fluorescence imaging unit configured to capture the fluorescence image by spatially resolved measurement of the emitted light to provide a fluorescence image; a visible light imager configured to capture a visible light image of a portion of a surface of the body part; A processing section; an output unit; the fluorescence imaging unit and the visible light imaging unit are configured such that the viewing directions and / or viewpoints of the fluorescence images and the visible light images are linked via a known relationship, and the capturing of the fluorescence images and the visible light images is repeated to provide a series of fluorescence images and a series of visible light images; The processing unit includes: a stitching unit configured to apply a stitching algorithm to the series of visible light images, wherein a set of stitching parameters is determined and applied, to generate a large visible light image of the body part, and configured to apply the stitching algorithm to the series of fluorescence images to generate a large fluorescence image, the stitching algorithm applied to the series of fluorescence images applies the set of stitching parameters determined when generating the large visible light image; The output unit is A measurement device configured to output the large visible light image and the large fluorescent image.

12. 12. The measurement device of claim 11, further comprising an overlay unit configured to overlay the large visible light image and the large fluorescent image to provide an overlay image of the body part, and wherein the output unit outputs the overlay image as an output of the large visible light image and the large fluorescent image.

13. 12. The measurement device according to claim 11, wherein the fluorescence imaging section and the visible light imaging section are configured so that the viewing direction and the viewpoint of the fluorescence image and the visible light image are the same, and the fluorescence image and the visible light image are captured via a same objective lens.

14. The measurement device according to claim 11 , wherein the fluorescence imaging section and the visible light imaging section are configured to simultaneously capture the fluorescence image and the visible light image without time switching between the fluorescence image signal and the visible light image signal.

15. the image capture device comprising a dichroic prism assembly configured to receive fluorescent light and visible light via an input surface; the dichroic prism assembly includes a first prism, a second prism, and a first compensator prism disposed between the first prism and the second prism; a further dichroic prism assembly for splitting the visible light into three light components; and a second compensator prism disposed between the second prism and the further dichroic prism assembly; the first prism and the second prism each have a cross section with at least five corners, each corner having an internal angle of at least 90 degrees, each have a respective entrance face and a respective exit face, and are designed such that an incident beam incident on the entrance face in a direction parallel to a normal to the entrance face is reflected twice within each of the first prism and the second prism, and exits each of the first prism and the second prism through the exit face parallel to the normal to the exit face; the normal to the entrance surface of each of the first prism and the second prism and the normal to the exit surface of each of the first prism and the second prism are perpendicular to each other; When light enters the first prism through the entrance surface, the light is partially reflected towards the exit surface of the first prism and travels a first path length from the entrance surface of the first prism to the exit surface of the first prism, the light partially enters the second prism through the first compensator prism and is partially reflected towards the exit surface of the second prism and travels a second path length from the entrance surface of the first prism to the exit surface of the second prism, The measurement device of claim 11 , wherein the first prism is larger than the second prism such that the first and second path lengths are the same.

16. The measurement device of claim 11, wherein the light irradiation unit, the fluorescence imaging unit, and the visible light imaging unit are arranged within a single image capture device further comprising a measurement unit configured to measure a distance to the surface of the body part captured in the visible light image.

17. The measurement device of claim 16 , wherein the image capture device is further configured to output a signal indicative of the measured distance.

18. An endoscope configured as the image capture device in the measurement device of claim 11.

19. A laparoscope configured as the image capture device within the measurement device described in claim 11.

Images