Motion-corrected background subtraction for fluorescence imaging

Abstract

a step (315) of applying the transformation matrix to one of the first and second fluorescence channel images to align them; and a step (316) of generating a background subtraction image by performing a subtraction operation between the fluorescence channel image to which the transformation matrix has been applied and the other fluorescence channel image.

Description

[Technical Field]

[0001] The present disclosure relates to background subtraction applied to fluorescence imaging. [Background technology]

[0002] Fluorescence imaging is a specific imaging technique used to obtain images that provide a visual representation of an object even when the object cannot be directly viewed. This technique is based on the phenomenon of fluorescence, in which fluorescent substances emit light when irradiated with excitation light (light having a wavelength and energy sufficient to excite the fluorescent dye). For this purpose, fluorescence images are typically obtained and displayed as a representation of the fluorescence emitted from various locations corresponding to the fluorescent dyes present in the object.

[0003] Fluorescence imaging is routinely used in fluorescence-guided surgery (FGS) (especially in tumor-related cases, fluorescence-guided resection (FGR)), endoscopy, and other medical procedures (surgical, diagnostic, and therapeutic) to identify desired targets (e.g., lesions such as tumors) within the patient's body.

[0004] For this purpose, a fluorescent agent is typically administered to a patient, and the fluorescent agent is directed to a desired biological target within the body part to be observed and allowed to bind to the target. For example, in the application of Fluorescence Molecular Imaging (FMI), the fluorescent agent is directed to a specific molecule of the desired target, typically a lesion such as a tumor, and is adjusted to remain immobilized on the specific molecule.

[0005] Next, an excitation light source is used to illuminate the target body part, causing the fluorescent substance administered to the patient to emit fluorescence. This fluorescence is then captured using a fluorescence camera, and a series of fluorescence images (called "light images" acquired using the excitation light source) are obtained. This allows the location of the fluorescent substance emitting fluorescence in the target body part, i.e., the target, to be visually identified.

[0006] The series of optical images acquired using the fluorescence camera can be displayed on a monitor along with a corresponding series of color images acquired separately by a color camera of the same target body part, allowing the operator to view the desired object and its surrounding area side-by-side. For example, in a surgical procedure, the displayed images can assist the surgeon in recognizing the boundaries of a lesion to be removed, in diagnostic applications, they can assist the physician in detecting or monitoring a lesion, and in therapeutic applications, they can assist the physician in defining the lesion to be treated.

[0007] However, optical images acquired using a fluorescence camera may contain unnecessary pixel information in addition to pixel information corresponding to the fluorescence emitted by the fluorescent agent in response to the light irradiated from the excitation light source. Specifically, even if the ambient light is neither excitation light nor fluorescence, pixel information resulting from such ambient light may be mixed into the optical image if some of its spectral components are included in the passband of the fluorescence camera.

[0008] A known technique for removing such unwanted pixel information is background subtraction, also known as “dark image” subtraction, as disclosed, for example, in Themelis et al., Journal of Biomedical Optics (2009).

[0009] According to this technique, a fluorescence camera is used to acquire a series of images ("dark images") without using an excitation light unit to induce fluorescence in fluorescent materials. The acquisition of these dark images is alternated with the acquisition of light images acquired using excitation light to induce fluorescence in fluorescent materials. As a result, the acquired dark images contain background image components due to ambient light within the bandpass of the fluorescence camera. Therefore, a background-subtracted fluorescence image can be obtained by subtracting the acquired dark image from a corresponding light image acquired before or after the dark image (ideally removing the unwanted background component due to ambient light in the light image). In fact, assuming that the background image component of the acquired dark image perfectly matches the unwanted background image component in the acquired light image, subtracting the dark image from the light image can remove this unwanted component from the resulting background-subtracted fluorescence image (which is identical to the background-subtracted fluorescence image).

[0010] The fluorescent image obtained in this manner after background subtraction can be displayed as, for example, a live video of the fluorescent image.

[0011] However, this background subtraction method has several practical drawbacks. In particular, movement of one or more objects (e.g., the target body part or other objects, such as a surgical instrument or a hand) within the field of view of the fluorescence camera, or movement of the fluorescence camera itself relative to the scene, can occur between the time of acquisition of the dark and light images. This can result in a mismatch (misalignment) between the background image components in the dark image and the corresponding background image components in the light image. This misalignment (even a small value, such as a one-pixel shift) can cause significant artifacts in the image after background subtraction.

[0012] For example, if an object in the field of view reflects ambient light (e.g., white light used to acquire color images, artificial light illuminating a room where a medical procedure is being performed, or sunlight shining into a room) and moves laterally between the time of acquisition of a dark image and the subsequent light image, the edge of the object will be represented by a high brightness value in the background subtracted image. Thus, ambient light with a non-negligible intensity relative to the intensity of the emitted fluorescence, combined with object and camera motion, can produce non-negligible ambient light artifacts in the background subtracted image. The greater the ambient light intensity, the greater the brightness value of the artifact.

[0013] Furthermore, ambient light can cause spurious (background) fluorescence emissions from endogenous fluorophores other than those of the fluorophore that are not generated by the use of the excitation light unit. Such spurious fluorescence emissions, combined with object and camera motion, can cause artifacts in some of the images after background subtraction due to shifts in background fluorescence caused by movement of the target body part.

[0014] Another problem associated with strong ambient light is that the image sensor of a fluorescence camera has a limited dynamic range. Therefore, excessive incident light can saturate (overexpose) the sensor. In particular, when ambient light from artificial sources, such as room lighting, or sunlight, falls within the fluorescence detection passband, its intensity can be much greater than the fluorescent emission intensity the sensor is intended to detect. However, if the user is only presented with the background-subtracted fluorescence image, the overexposure in the acquired dark and light images may not be visually noticeable. This is because, when generating the background-subtracted fluorescence image, the high pixel values ​​of corresponding pixel regions affected by overexposure are subtracted from each other to generate the background-subtracted image, resulting in pixel values ​​of low intensity (ideally zero).

[0015] A common feature of both of these drawbacks is that they are somewhat obscured from the user's view because the user is only shown the image after background subtraction, and overexposure (high pixel values ​​are subtracted) and movement are not directly indicated. These drawbacks pose practical challenges to the practical application of background subtraction in live fluorescence imaging.

[0016] U.S. Patent No. 11,322,245 discloses a medical image processing device connected to an observational imaging device. The observational imaging device is configured to capture light reflected from a subject at a first timing to generate a subject image, and capture fluorescence emitted from the subject at a second different timing to generate a fluorescence image. In particular, the medical image processing device is configured to determine whether at least one of the subject and the observational imaging device has moved between the capture of the previous subject image and the first timing, or between the capture of the previous fluorescence image and the second timing, and to prohibit superimposition of the subject image and the fluorescence image if it is determined that at least one of the subject and the observational imaging device has moved between the capture of the subject image and the fluorescence image.

[0017] U.S. Patent No. 11,276,148 discloses a method for generating a plurality of exposure frames including red, green, blue, and fluorescent frames by activating an emitter to emit a plurality of electromagnetic radiation pulses including red, green, blue, and fluorescent pulses, and detecting reflected electromagnetic radiation resulting from the plurality of pulses with a pixel array of an image sensor, and further includes detecting motion between two or more consecutive exposure frames of the plurality of exposure frames, correcting for the detected motion, and combining the two or more consecutive exposure frames to generate an image frame.

[0018] Chinese Patent No. 102361583 discloses an image processing device including a motion vector calculation unit and a registration processing unit. The motion vector calculation unit is configured to calculate information about a motion vector between a fluorescence image based on fluorescence generated from an observation area irradiated with excitation light and a reflected light image based on light reflected from the observation area. The registration processing unit is configured to correct a misalignment of the subject between the fluorescence image and the reflected light image of the observation area based on the information about the motion vector.

[0019] U.S. Patent No. 10,869,645 discloses an adaptive imaging method for generating low-light video of an object for medical visualization, including the steps of: acquiring, with an image acquisition unit, a series of reference frames and / or a series of low-light video frames showing the object; assessing relative motion between the image acquisition unit and the object based at least in part on the acquired sequence of reference or low-light video frames; adjusting a level of image processing of the low-light video frames based at least in part on the relative motion; and generating a characteristic low-light video output from a predetermined amount of low-light video frames determined based on the adjusted image processing level.

[0020] U.S. Patent No. 10,108,844 discloses an imaging subsystem configured to image particles disposed within the subsystem at different wavelength bands, the imaging subsystem including the steps of selecting a first filter set of one or more optical filters corresponding to a first wavelength band, illuminating the particle through the first filter set, selecting a second filter set of one or more optical filters corresponding to a second wavelength band, illuminating the particle through the second filter set, storing acquired data for a plurality of images of the particle, generating a first composite image based on the plurality of images, the first composite image including a first composite spot corresponding to the particle, the first composite spot being displaced from the spot in the stored plurality of images by a first offset, and generating a second composite image based on the corrected coordinates, and correcting the coordinates of at least one of the plurality of images so that the second composite image includes a second composite spot that is displaced a second, smaller offset from the spot in the plurality of images. [Prior art documents] [Patent documents]

[0021] [Patent Document 1] U.S. Patent No. 11,322,245 [Patent Document 2] U.S. Patent No. 11,276,148 [Patent Document 3] Chinese Patent No. 102361583 [Patent Document 4] U.S. Patent No. 10,869,645 [Patent Document 5] U.S. Patent No. 10,108,844 Summary of the Invention

[0022] According to a first aspect of the present invention, there is provided a method as set forth in claim 1.

[0023] Also provided are related imaging systems and computer program products as set forth in claims 17-18, and related surgical, diagnostic and therapeutic procedures as set forth in claims 19-21.

[0024] Embodiments of the present invention are based on the recognition that two consecutive dark and light images (frames) acquired on a fluorescence channel by an imaging system cannot be compared to extract a transformation matrix that estimates the amount of motion causing misregistration between the two images (because the images are different and contain only a small amount of image content; specifically, the light image contains image content corresponding to the target body part that emits fluorescence in response to excitation light, while the dark image contains only a background component due to the portion of ambient light that passes through the passband of the fluorescence camera), but that by performing relative registration between reference reflectance images (e.g., color images) associated with the dark and light images, it is instead possible to extract a transformation matrix that estimates the amount of motion causing misregistration between the dark and light images.

[0025] By spatially aligning the dark and light image frames before pixel-by-pixel subtraction of the dark image from the light image, a transformation matrix can be advantageously applied for motion correction, resulting in a motion-corrected, background-subtracted fluorescence image (hereafter referred to as the background-subtracted image) that avoids motion artifacts while minimizing background effects due to ambient light, which is different from the fluorescence emitted in response to excitation light.

[0026] In some embodiments in which the fluorescence channel images and the reflectance images are acquired asynchronously, first and second interpolated transformation matrices are determined that are suitable for aligning the reference reflectance image with the corresponding dark and light images, respectively. In some of these embodiments, the transformation matrices for aligning the dark and light images are determined as a function of the reference reflectance image and the first and second interpolated transformation matrices. In other embodiments, the first and second interpolated transformation matrices are applied to the reference reflectance image to obtain a first virtual reflectance image that is approximately aligned with the dark image and a second virtual reflectance image that is approximately aligned with the light image, respectively. The transformation matrix for aligning the dark and light images is then determined as the transformation matrix for aligning the first and second virtual reflectance images. In yet another embodiment, the first and second interpolated transformation matrices are applied to align the dark and light images with the respective reference reflectance images, and then the transformation matrix for aligning the dark and light images is determined as the transformation matrix for aligning the reference reflectance image.

[0027] In another embodiment, when the fluorescence channel image and the reflectance image are acquired asynchronously, the reference reflectance image associated with the dark image and the light image is the reflectance image acquired at a time closest to the reference time of each dark image and light image, and the transformation matrix for aligning the dark image and the light image is determined as the transformation matrix for aligning these selected reflectance images.

[0028] In embodiments where the fluorescence channel images and the reflectance images are acquired synchronously, the reference reflectance images associated with the dark and light images are the reflectance images acquired at the reference times of the respective dark and light images, and the transformation matrices for aligning the dark and light images are determined as the transformation matrices for aligning these selected reflectance images.

[0029] In some embodiments, after capturing either a dark image or a light image, a check is performed to determine whether the captured image is overexposed, and if so, a warning is output indicating excessive ambient light.

[0030] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 illustrates an imaging system for fluorescence-guided open surgery configured to perform fluorescence background subtraction with motion compensation, according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of the imaging system shown in FIG. [Figure 3] 7 illustrates a method that can be performed by the imaging system of FIG. 1 or FIG. 6. [Figure 4] 4A-4C show a series of color, dark, and light images acquired asynchronously over time with reference to a virtual reference color image associated with the dark or light image, according to operation of one embodiment of the method of FIG. [Figure 5] 1A-1C show a series of color, dark, and light images acquired synchronously over time with reference to a reference color image associated with the dark or light image, in accordance with one embodiment of the present invention; [Figure 6] FIG. 1 illustrates an imaging system for an endoscope configured to perform motion-compensated fluorescence background subtraction in accordance with an embodiment of the present invention. [Figure 7] FIG. 7 is a block diagram of the imaging system shown in FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0032] FIG. 1 shows an imaging system 100 for assisting a surgeon in fluorescence-guided surgery (eg, fluorescence-guided resection when the surgery is for a tumor).

[0033] In particular, the illustrated imaging system 100 comprises an apparatus 105 that includes a trolley 1150 housing a supply unit and a control unit (not shown in FIG. 1 ) that supply and control the operation of the system 100. A column 125 extends upward from the rear of the trolley 1150. The column 125 is provided with a handle bar 130 that allows an operator to move the apparatus 105. A cantilever 135 projects from the column 125 above the trolley 1150. The cantilever 135 carries a primary monitor 140 for displaying images to the operator and a keyboard 145 with a pointing device, such as a mouse or trackball, for the operator to input information and commands. Attached to the top end of the column 125 (above the cantilever 135) is a pivot arm 150. The pivot arm 150 carries a secondary monitor 155 for displaying images to a medical practitioner (e.g., a surgeon), and the secondary monitor 155 can be rotated left or right. An articulated arm 160 is also attached to the upper end of the column 125, adjacent to the pivot arm 150. An imaging head 165 is suspended from the articulated arm 160. The imaging head 165 includes two handle bars 170 that allow an operator to operate the imaging head 165.

[0034] Referring now to FIG. 2, a functional configuration of the imaging head 165 and a control unit (denoted by reference numeral 205) is shown in schematic form.

[0035] The imaging head 165 includes at least an image capture unit for capturing an image of a scene within a corresponding field of view 104 (e.g., an area within a solid angle that the imaging unit can sense) and an illumination unit for illuminating the scene.

[0036] In particular, as shown in Figure 2, the imaging head 165 is positioned relative to a patient's body 295 undergoing a surgical procedure such that a target body part 290 (e.g., a surgical cavity exposed by a small skin incision in minimally invasive surgery) is within the image acquisition unit's field of view 104. During operation, the field of view 104 may include one or more extraneous objects (not shown in Figure 2) separate from the surgical cavity, such as surgical instruments, hands, surgical tools, surrounding body parts, background objects, etc. (surrounding or overlying the surgical cavity).

[0037] The target body part 290 contains a fluorescent substance. To this end, a fluorescent agent is administered to the patient (e.g., intravenously or topically). The fluorescent agent may be a target-specific fluorescent agent designed to bind to a specific biological target (e.g., tumor tissue, nerves, blood vessels, lymph nodes, lymphatic vessels, etc.). The fluorescent agent is administered to the patient 295 in advance so that it can circulate within the vascular system of the patient 295 until it reaches the target body part 290 and binds to the desired target.

[0038] The illumination and image acquisition units of imaging head 165 are disclosed in more detail below.

[0039] First, the illumination unit is described. The illumination unit includes a near-infrared (NIR) excitation light source 210 (e.g., a laser or LED light source) configured to illuminate a scene within the field of view 104 of the image acquisition unit with excitation light. The excitation light is guided to the scene by corresponding light-guiding optics 220. The NIR excitation light has a wavelength and energy suitable for exciting fluorescent dyes of fluorescent materials within the target body part 290, causing them to emit fluorescence. Specifically, when a fluorescent dye absorbs the excitation light, the dye transitions to an excited, unstable (electronic) state and then briefly returns to its ground (electronic) state. This process emits fluorescence, the intensity of which depends on the amount of illuminated fluorescent dye (as well as other factors, such as the fluorescent dye's location within the field of view 104 and the body part 290), and the emitted fluorescence has a characteristic NIR radiation spectrum with a longer wavelength than the NIR excitation light (due to the loss of energy as heat in the excited state).

[0040] The illumination unit further comprises a white light source 215 (e.g., an LED, a halogen lamp, or a xenon lamp) configured to emit white light (i.e., light that contains all wavelengths in the color spectrum visible to the human eye at equal intensity and appears substantially colorless), which is directed by corresponding light-guiding optics 225 to a scene in the field of view 104 of the imaging system.

[0041] Turning now to the image acquisition unit, the image acquisition unit comprises collection optics 230 configured to capture light present within the field of view 104. The light collected within the scene may include: fluorescence emitted by fluorescent dyes present within the target body part 290 (by absorbing the excitation light provided by the excitation light source 210); unwanted fluorescence (excited by absorbing components of the white light provided by the light source 215 or other ambient light) that may be emitted by fluorescent dyes other than the target fluorescent reagent (e.g., endogenous fluorescent dyes present within the inspected target body part 290); excitation light reflected by objects present within the field of view 104; and visible light reflected by objects within the field of view 104 (illuminated by the white light emitted by the light source 215 or other visible light present in the room in which the imaging system 100 operates, e.g., artificial lighting, light from the monitors 140, 150, sunlight shining into the room, etc.).

[0042] The image acquisition unit further comprises a beam splitter 235 configured to separate the light captured within the field of view 104 into two channels: a first channel consisting of light in the near-infrared (NIR) spectrum (including the fluorescent emission spectrum) and a second channel consisting of light in the visible light spectrum. For example, the beam splitter 235 may be a dichroic mirror that transmits light with wavelengths above and reflects light with wavelengths below a threshold wavelength between the NIR and visible light spectra.

[0043] The image acquisition unit further includes a fluorescence camera 245 having a fluorescence emission detection passband. In particular, the fluorescence camera 245 includes a fluorescence imaging element 241 and an associated emission filter 240 (e.g., located on or near the light-receiving surface of the sensor 241).

[0044] Filter 240 is a passband filter having a fluorescence emission detection passband corresponding to the fluorescence emission spectrum of the fluorescent dye in target body part 290. Filter 240 may be comprised of a single filter or a combination of multiple filters. In this manner, filter 240 filters the light provided by the first channel of beam splitter 235, passing only light within the fluorescence emission spectrum of the target fluorescent dye (toward the light receiving surface of sensor 241), while (under ideal operation) blocking components of excitation light reflected by objects in field of view 104 and components of reflected ambient light other than fluorescence and excitation light (e.g., a portion of the white light emitted and reflected from white light source 215, other artificial light illuminating the room in which system 100 operates, or a portion of sunlight entering the room) that are in the near-infrared spectrum or that are inadvertently deflected toward filter 240 due to non-ideal operation of beam splitter 235.

[0045] Imager 241 may be any sensor with suitable sensitivity to detect incident light emitted from fluorescent dyes in target body part 290 (light provided by the first channel of beam splitter 235 and passing through filter 240). In response to detecting this light, fluorescence camera 245 is configured to generate a fluorescence image representing the distribution of fluorescent dyes within field of view 104. For example, imager 241 may be an EMCCD sensor, an intensified charge coupled device (ICCD) sensor, a CMOS sensor, an InGaAs sensor, a photomultiplier tube (PMT), or other highly sensitive sensor for low-light imaging in the near-infrared range.

[0046] Referring again to the second channel of the beam splitter 235, in connection with the second channel that collects the reflected visible light within the field of view 104, the image acquisition unit of the imaging head 165 further comprises a reflected camera, in particular a color camera 250, which includes an imaging element 250a (e.g., of the CCD or CMOS type) that can individually detect the intensity of the color components of the reflected visible light.

[0047] For example, imager 250a may include RGB filters (e.g., Bayer filters) on its light-receiving surface (so that the red, green, and blue components of collected white light pass through filters for each corresponding color component before reaching corresponding pixel areas on the imager surface). In some variations, the imager may include, in addition to or instead of RGB pixels, pixels sensitive only to specific light bands or pixels with broadband sensitivity to, for example, the entire visible light spectrum (referred to herein as white (W) pixels). (In yet another variation, the imager may include a hyperspectral sensor with multiple planes each sensitive to a subband of red, green, or blue light.) Other imagers may include sensors with color filter arrays (CFAs) different from RGB filters, for example, RGBE filters (similar to a Bayer filter with the addition of an "emerald" filter) or CYYM filters (an array of cyan, yellow, and magenta filters). Furthermore, instead of using a CFA, the image sensor may be fabricated to include regions with different sensitivities to different color components of the incident light, or the incident light may be split into multiple color components and each directed to a corresponding sensor surface.

[0048] Turning now to the control unit 205 of the imaging system 100, the control unit 205 includes multiple units, which are connected to each other via a bus structure 255, as shown schematically in Figure 2. In particular, the control unit 205 includes one or more microprocessors (μP) 260 configured to provide the processing and control functions of the control unit 205. A non-volatile memory (ROM) 265 stores basic code for bootstrapping the control unit 205, and a volatile memory (RAM) 270 is used by the microprocessor(s) 260 as working memory. The control unit 205 also includes a mass memory 275 (e.g., a solid-state disk (SSD)) for storing programs and data.

[0049] Furthermore, the control unit 205 comprises a number of controllers 280 for peripheral devices or input / output (I / O) units. In particular, one or more of the controllers 280 are configured to control peripheral devices 285, namely the primary monitor 140 and secondary monitor 155, the keyboard 145, a pointing device, a drive for reading and writing removable storage media (e.g., USB type), a network interface card (NIC) for connecting to a communication network (e.g., a LAN and the Internet thereafter), etc.

[0050] Furthermore, the controller 280 includes a fluorescence imaging control module 281 and a color imaging control module 282, which are configured to control the operation of the excitation light source 210, the white light source 215, the fluorescence camera 245, and the color camera 250 of the imaging head 165 to acquire a series of fluorescence images and color images.

[0051] In particular, the fluorescence imaging control module 281 is configured to coordinate the operation of the fluorescence camera 245 and the on / off operation of the excitation light source 210 so that the following images can be alternately acquired via the fluorescence acquisition channel: a series of light images (including, for example, light image 602 shown in FIG. 4 ) taken using an excitation light source 210 to cause fluorescent dyes in the target body part 209 to emit fluorescence; and A series of dark images (including, for example, dark images 601 and 603 shown in FIG. 4) captured without using the excitation light source 210.

[0052] The acquired dark and light images may have the same size and shape, and each image may include a matrix of cells that store intensity values ​​at picture elements (pixels) corresponding to positions within the field of view 104. Each pixel value thus defines the brightness of the pixel as a function of the intensity of light emitted from that position (which in the light image corresponds to the amount of fluorescent dye present at that position), e.g., varying from black to white as the amount of fluorescent dye increases.

[0053] Preferably, the acquired dark and light images are monochrome images containing grayscale values. Alternatively, the acquired light and dark images may be monochrome images containing grayscale values ​​in RGB, YcBcr, HSL, CIE-L, or other formats in pixels or voxels. * a * b, or LAB color.

[0054] Furthermore, in order to acquire the optical image at an appropriate exposure level, a predetermined exposure time T exp To enable acquisition of a light image during a desired exposure time, the fluorescence imaging control module 281 is configured to control the light source 210 to emit a series of excitation light pulses (each pulse causing a fluorescent emission from a fluorescent dye in the target body part 290, the fluorescent lifetime of which is shorter than the time between pulses) over a period of time corresponding to a desired exposure time, e.g., to achieve a predetermined average brightness level or brightness range within the exposure range of the image sensor 241 at a certain number of pixels in a predetermined region or region of interest of the light image.

[0055] Between the end of acquisition of each light image and the start of acquisition of the next light image, the control module 281 is configured to keep the excitation light unit 210 in an off state, allowing dark images to be acquired in the meantime, preferably with the same exposure time as the corresponding alternating light image.

[0056] During acquisition, the alternating acquired light and dark images are read by module 281 and associated with corresponding timestamps indicating the reference time of their acquisition and stored in a light / dark image repository 283 held within control unit 205 (which may be included in mass memory 275 or provided separately from mass memory 275) or accessible to control unit 205. For example, each acquired dark or light image is stored in association with at least one timestamp indicating the acquisition time of either the beginning, middle, or end of the acquisition period.

[0057] The color imaging control module 282 will now be described. The module 282 is configured to illuminate the white light source 215, which preferably remains illuminated to continuously illuminate the field of view 104 of the imaging head 165 while the system 100 is being used to observe the target body part 209.

[0058] While the white light source 215 illuminates the target body part 290, the color imaging control module 282 is further configured to control the color camera 250 to acquire a series of color images (e.g., including the six color images 401-406 shown in FIG. 4) in the color acquisition channel.

[0059] For example, if the color imager 250a used in the color camera 250 is an RGB sensor, the color camera 250 is configured to combine (e.g., by demosaicing) the raw R, G, and B images acquired via the R, G, and B pixels of the imager 241 during image acquisition to generate a corresponding RGB image (in which each pixel is associated with three intensity values ​​representing brightness in each of the R, G, and B image planes). Furthermore, the resulting RGB image can be converted (in the camera 245 or the control unit 205) into a YUV image including a luminance plane (Y) and two color difference planes (U, V) (or may be converted into any other image format associated with a color space different from RGB).

[0060] The color imaging control module 282 is configured to control the color camera 250 to capture the color images 401-406 at a higher capture frequency than the alternating series of dark and light images 601-603 (e.g., in FIG. 4, the color images are captured at an average frame rate of 36 fps, while the light images 601-603 are captured at an average frame rate of 18 fps, twice the frequency of the former). In this way, a single dark or light image is captured between two corresponding color images (e.g., in FIG. 4, dark image 601 is captured between color images 401 and 402, light image 602 is captured between color images 403 and 404, and dark image 603 is captured between color images 405 and 406).

[0061] Furthermore, control modules 281 and 282 are preferably configured to control image acquisition in the fluorescence and color channels independently of each other (though fluorescence camera 245 and color camera 250 share the same field of view 104, each camera is independently controllable by its corresponding control module 281, 282). As a result, a series of color and dark / light images are acquired asynchronously in each channel. Therefore, the timing of color image acquisition is generally asynchronous with the timing of dark / light image acquisition, and the start times of acquisition of dark / light images 601-603 do not coincide with the start times of color image acquisition (e.g., as shown in FIG. 4). Furthermore, the interval between the start times of fluorescence and color image acquisition and the acquisition frequency in the color and fluorescence channels may vary between different operations of system 100 or even during the same operation.

[0062] The color images 401-402 may be acquired with the same (or similar) size and shape as the fluorescence channel images 601-603.

[0063] Furthermore, it should be noted that the exposure time for acquiring a single color image is typically much shorter than the exposure time for acquiring the light / dark images 601-603 (because more light is typically available for color imaging compared to fluorescence imaging). Therefore, to improve comparability between the color images 401-406 acquired in the color channels and the light / dark images 601-603 acquired in the fluorescence channels, the control module 282 may optionally be configured to control the color camera as follows to acquire each of two color images acquired before and after a given fluorescence channel image: Acquire multiple color image frames 407 over a time period identical (or similar) to the exposure time used to acquire a given light / dark image (e.g., as shown in relation to the first color image 401 in FIG. 4). These color frames 407 are combined (added) to form a single color image acquired over a period of time that is the same as or similar to the exposure period of the given light / dark image (note that this image combination may be performed by the control module 282 or by another module of the control unit 205 after receiving the acquired color image frames 407).

[0064] The acquired color images 401-406 are acquired by the control module 282 and stored in a color image repository 284 maintained within the control unit 205 in association with a corresponding timestamp indicating the time of each acquisition (e.g., indicating the start, middle, or end of the acquisition period).

[0065] Thus, the control unit 205 stores timing information for each frame, whether it is a color image or a light / dark image, indicating the capture time of that frame relative to frames of other channels (even if the two frames are captured asynchronously, as shown in Figure 4).

[0066] Successive acquired dark and light images may contain the same (or similar) background image content, which may be caused by reflected ambient light, such as white light emitted by light source 215, artificial light illuminating the room in which system 100 operates, sunlight shining into the room, or even spurious fluorescence excited by the white / artificial / sunlight, passing through emission filter 240 (e.g., due to non-ideal operation of emission filter 240) and reaching the light-receiving surface of fluorescence imager 241. However, such common background image content may be spatially shifted in the successive acquired dark and light images due to movement of objects in the imaged scene and / or movement of fluorescence camera 245 relative to the imaged scene.

[0067] To address this issue, further configuration and functionality of system 100 for performing motion-compensated background subtraction on a fluorescence channel according to the present invention will now be described with reference to method 300 shown in Figure 3. In particular, the illustrated method 300 is disclosed with respect to the use of system 100 during a surgical procedure (e.g., observing target body part 290 as shown in Figure 2) and operates based on the asynchronous acquisition of a series of color images and a series of dark / light images (which for purposes of illustration will include color images 401-406 and dark / light images 601-603, respectively, as shown in Figure 4).

[0068] Prior to a surgical procedure, medical personnel administer a fluorescent dye (e.g., indocyanine green, methylene blue, etc.) to the patient. The fluorescent dye reaches the target biological body site 109 (e.g., the tumor to be resected). This result can be achieved either by using a non-targeted fluorescent dye, which is designed to accumulate without specific interaction with the target (e.g., passively), or by using a targeted fluorescent dye using a formulation containing a target-specific ligand, which is designed to bind to the target through specific interactions by utilizing its physical structure based on chemical binding properties or tissue, vascular, or metabolic properties. The fluorescent dye can be administered intravenously as a bolus via syringe. As a result, the fluorescent dye circulates within the patient's vasculature, reaching and binding to the tumor. Meanwhile, the remaining (unbound) fluorescent dye is cleared from the blood pool according to a fixed half-life. After a waiting period (e.g., from a few minutes to 24-72 hours) to allow the fluorescent dye to accumulate in the tumor and wash away from other body parts, the operator creates an opening in the patient's body to expose the target body part 290 where the fluorescent dye was administered. At this point, the operator powers on the imaging system 100 and positions the imaging head 165 of the system 100 adjacent the surgical field area that was opened in the patient's body. The operator then enters a start command to the imaging system 100 (e.g., using the keyboard 145).

[0069] In response to the start command, color imaging control module 282 turns on white light source 215 (which preferably remains on continuously throughout the procedure) and begins controlling the acquisition of a series of color images on the color channels using color camera 250, including color images 401-406 shown in FIG. 4. Control module 282 stores the series of color images in repository 284 along with their corresponding timestamps as the series progresses.

[0070] While the color imaging control module 282 controls the acquisition of a series of color images on the color channels, the method 300 shown in FIG. 3 begins, in which the fluorescence imaging control module 281 controls the fluorescence camera 245 to acquire an exposure time T exp A first dark image 601 (shown in FIG. 4 ) is acquired over a period of time (step 301). In particular, as shown in FIG. 4 , the first dark image 601 is acquired between two consecutive color images 401 and 402 acquired on the color channels. After acquisition, the control module 281 stores the first dark image 601 (along with a corresponding timestamp) in the repository 283. Note that the method 300 may also start in step 301 with the acquisition of a first light image (e.g., light image 602 shown in FIG. 4 ) rather than the dark image 601.

[0071] Preferably, method 300 proceeds to step 302, where control module 281 (or other suitable module within control unit 205, or fluorescence camera 245 itself) checks whether first dark image 601 is within the linear detection range of fluorescence imager 241 (i.e., whether first dark image 601 is not overexposed).

[0072] To determine whether the first dark image 601 is within the detection range of the image sensor, a first pixel intensity threshold can be set (during sensor development) to correspond to the maximum value within the linear range of the sensor 241. A second threshold is also set, corresponding to the maximum allowable number of pixels exceeding the first threshold. The number of pixels in the first dark image 601 with intensities exceeding the first threshold is then counted. If the counted number exceeds the second threshold, the first dark image 601 is deemed overexposed (out of the detection range). These thresholds may be fixed, preset values, depending on the device hardware (e.g., the image sensor). The pixels compared to these thresholds may be limited to a predetermined region (e.g., a region of interest) within the image, in which case irrelevant regions are ignored in the overexposure determination. Any other method for determining whether an image is within the detection range of the image sensor 241 may be used, provided that the method has a computational speed that can be performed for each newly acquired image.

[0073] If the first dark image 601 is determined to be overexposed (e.g., due to strong ambient light components, such as artificial lighting illuminating the room where the surgery is being performed or sunlight shining into the room, passing through the emission filter 240), the system 100 outputs a warning (e.g., a visual and / or audio warning) to the operator indicating that there is excessive ambient light (optional step 303). Method 300 may then be restarted from step 301.

[0074] If it is determined that the first dark image is not overexposed, method 300 proceeds to step 304. Before describing the operations of step 304, it should be noted that the operations of step 302 are optional, and thus method 300 can proceed directly to step 304 after step 301.

[0075] In step 304, the control module 281 (or the control module 282, or another appropriate module in the control unit 205) determines a first reference (auxiliary) color image associated with the first dark image 601 based on the reference time of the dark image 601 (indicated by the timestamp stored in association with the dark image 601). For example, if the reference time is assumed to be the start time of acquisition of the dark image 601, the color image 401 shown in FIG. 4 is selected as the first reference color image associated with the first dark image 601 because it is a color image acquired before the reference time. Note that the first reference color image associated with the first dark image 601 may be selected in step 304 as the first color image acquired after the reference time, and the reference time itself may be a time point other than the start time of acquisition of the first dark image 601 (e.g., an intermediate time or an end time of acquisition).

[0076] The method 300 proceeds to step 305 where the imaging control module 281 (or any other module in the control unit 205) determines whether there is a next fluorescence channel image (i.e., a light image) acquired on the fluorescence channel after the first dark image 601.

[0077] If it is determined that the first light image 602 should be acquired on the fluorescence channel after the first dark image 601, the method 300 proceeds by causing the fluorescence imaging control module 281 to turn on the excitation light source 210 to emit fluorescence from the target body part 290 containing a fluorescent material, and controlling the fluorescence camera 245 to capture an exposure time T exp 4, the first light image 602 is acquired between two color images 402 and 403 that are acquired consecutively on the color channels. After acquisition, the control module 281 stores the first light image 602 (along with a corresponding timestamp) in the repository 283.

[0078] Preferably, method 300 proceeds to optional step 307, in which control module 281 (or other suitable module within control unit 205, or fluorescence camera 245) determines whether first light image 602 is within the linear detection range of fluorescence imager 241. This determination is performed in a manner similar to optional step 302 described above.

[0079] If the first light image 602 is determined to be overexposed, then in step 308, the system 100 outputs a warning to the operator in a manner similar to optional step 303 described above. The method 300 is then restarted (optional step 309, restarting from step 301). Alternatively, the method 300 can re-execute step 306 until a first light image that is not overexposed is acquired.

[0080] Method 300 then proceeds at step 310, where control module 281 (or control module 282, or other suitable module within control unit 205) determines a second reference color image to be associated with first light image 602 (based on the reference time indicated by the timestamp associated with light image 602). For example, color image 403 shown in FIG. 4 is associated with first light image 602 (similar to how first reference color image 401 is associated with first dark image 601 in step 304).

[0081] At this stage, method 300 proceeds to step 311, where color imaging control module 282 (or fluorescence imaging control module 281, or other suitable module within control unit 205) determines a transformation matrix suitable for aligning the first dark image 601 and the first light image 602 based on the relative alignment of the corresponding first and second reference color images 401, 403.

[0082] It should be noted that the transformation matrix calculated in step 311 can accurately estimate the motion occurring between the first dark image 601 and the first light image 602 based on information obtained from the reference color images 401, 403 for at least the following reasons: The color camera 250 has significantly higher resolution than the fluorescence camera 245 (so the transformation matrix can be effectively extrapolated). Color images must be stable in terms of pixel intensity.

[0083] More specifically, in some embodiments, a first interpolation coefficient φ1 is calculated using stored timestamps indicating the capture times of the color images 401 and 402 and the reference time of the first dark image 601. The interpolation coefficient φ1 is a coefficient indicating where the capture times of the color images 401 and 402 are located relative to the reference time of the first dark image 601. For example, the timestamps of the color images 401 and 402 and the first dark image 601 are respectively set to t C1 ,t C2 ,t D1 Then, the interpolation coefficient φ1 can be calculated by the following equation: φ1=(t D1 -t C1 ) / (t C2 -t C1 ) where φ1∈[0,1].

[0084] Two exemplary methods for obtaining a first interpolation transformation matrix suitable for aligning the first reference color image 401 with the first dark image 601 using the interpolation coefficient φ1 are presented below. Both methods are based on calculating a transformation matrix that represents a transformation that estimates the underlying motion (i.e., motion that causes a proportional spatial shift between the color images) that occurred between the acquisition of the color images 401 and 402. As specific examples, we refer to affine warping and perspective warping, which are commonly used for motion estimation and compensation. These transformation algorithms are widely used in video stabilization techniques to estimate and compensate for camera motion and are implemented in the OpenCV library (https: / / docs.opencv.org / 3.4 / ) (see Bradski, "The OpenCV Library" and Dr. Dobb's Journal of Software Tools). They are also disclosed in Shi, J., "Good features to track," a paper presented at the 1994 IEEE Computer Vision and Pattern Recognition conference. It should be noted that other transformation algorithms known from video stabilization and other imaging applications can also be used to extract the motion transformation matrices associated with color images 401 and 402.

[0085] In a first exemplary technique, affine warping is used to calculate a transformation matrix between the color images 401 and 402. Specifically, at least three feature points (landmarks) are extracted from one color image and tracked in the other color image to estimate an affine transformation matrix that represents the movement between the color images 401 and 402 using three basic elements: rotation (linear transformation), translation (vector addition), and scaling transformation (linear transformation). The affine transformation matrix can be a 2x3 matrix with the following structure:

[0086]

number

[0087] where θ is the rotation angle, s is the scaling factor, and t x and t y are the translations along the x and y axes, respectively.

[0088] After the affine transformation matrix is ​​calculated, the affine transformation matrix is ​​decomposed into dimensional motion components, and the scaling factor s and rotation angle θ are calculated as follows:

[0089]

number

[0090] The calculated interpolation coefficient φ 1 between color images 401 and 402 is then used to obtain an interpolated version of the dimensional motion component as follows:

[0091]

number

[0092] A new interpolated affine transformation matrix is ​​then reconstructed from the interpolated components.

[0093]

number

[0094] According to a second exemplary approach, transformation matrices for color images 401 and 402 are calculated using affine warping or perspective warping (or any other motion transformation algorithm), and then an interpolated (fractional) transformation matrix is ​​directly obtained using a fractional power of a matrix algorithm (for example, as disclosed in "A Schur-Pade Algorithm for Fractional Powers of a Matrix" by Nicholas J. Higham and Lijing Lin, SIAM Journal on Matrix Analysis and Applications, 2011). In this regard, since fractional power of a matrix is ​​only applicable to square matrices, when an affine transformation matrix is ​​obtained using affine warping, for example, a last row [0,0,1] is added to a 2x3 affine transformation matrix to make it a 3x3 matrix.

[0095] Let T be the transformation matrix to which the fractional power operation is applied. Assuming that this matrix is ​​diagonalizable, there exists the following orthogonal matrix P and diagonal matrix D, and the following relationship holds:

[0096]

number

[0097] Each element of matrix D is an eigenvalue of matrix T, and matrix P contains the respective eigenvectors.

[0098] Using the calculated interpolation coefficient φ1 between the color images 401 and 402, the φ1 power (φ1-th power) of the matrix T is calculated as follows.

[0099]

number

[0100] (Here, when φ1=0, T φ corresponds to the identity matrix, and when φ1=1, T φcorresponds to the matrix T.)

[0101] Based on the above-described principles, a second interpolation coefficient φ2 is calculated using stored timestamps indicating the acquisition times of the two color images 403 and 404 and the reference time of the first light image 602. The interpolation coefficient φ2 is then used to determine a second interpolation transformation matrix for aligning the second reference color image 403 with the first light image 602.

[0102] In some embodiments, in step 311, a transformation matrix representing a transformation that estimates the underlying motion that occurred between the acquisition of the first dark image 601 and the acquisition of the first light image 602 (which results in a corresponding spatial (pixel-wise) shift between the first dark image 601 and the light image 602) is determined based on the first and second reference color images 401, 403 and the calculated first and second interpolation transformation matrices.

[0103] According to some alternative embodiments, the calculated first and second interpolation transformation matrices are applied to the first and second reference color images 401 and 403, respectively, to obtain a first virtual color image 408 that is generally aligned with the first dark image 601 and a second virtual color image 409 that is generally aligned with the first light image 602 (e.g., as shown in FIG. 4 ). The first virtual color image 408 has a field of view that substantially overlaps the field of view (framing view) of the first dark image 601, and the second virtual color image 409 has a field of view that substantially overlaps the field of view (framing view) of the first light image 602. In effect, the first and second virtual color images 408 and 409 approximate the field of view (imaged content and associated location within the frame) that would have been obtained if the first dark image 601 and the first light image 602, respectively, had they been acquired at the reference time. The resulting first and second virtual color images 408, 409 may be associated with timestamps midway between the timestamps of the color images 401-402 and 403-404, respectively, and stored in the color image repository 284 (or in a dedicated repository within the central unit 205 for storing virtual color images).

[0104] At this stage, in step 311, a transformation matrix is ​​determined to align the first and second virtual color images 408, 409. For example, the aforementioned affine warping or perspective warping (or other transformation algorithms used in video correction processes and other imaging applications) can be used to determine this transformation matrix. Because the first and second virtual color images 408, 409 are approximately aligned with the first dark image 601 and the first light image 602, respectively, this transformation matrix also represents a transformation that estimates the underlying motion that occurred between the acquisition of the first dark image 601 and the acquisition of the first light image 602.

[0105] According to another alternative embodiment, the aforementioned first and second interpolation transformation matrices are applied to align the first dark image 601 to the first reference color image 401 and the first light image 602 to the second reference color image 403, respectively. Then, in step 311, a transformation matrix suitable for aligning the first and second reference color images 401, 403 is determined (and therefore also a transformation matrix suitable for aligning the first dark image 601 and the first light image 602).

[0106] Referring back to step 304, according to some alternative embodiments, one of the color images 401, 402 acquired at a time closest to the reference time of the first dark image 601 is determined as the first reference color image to be assigned to the dark image 601. Specifically, if the calculated interpolation coefficient φ1 is less than or equal to 0.5, the first color image 401 is assigned as the first reference color image for the first dark image 601. On the other hand, if φ1 is greater than 0.5, the second color image 402 is assigned as the first reference color image. Similarly, referring back to method step 310, one of the color images 403, 404 acquired at a time closest to the reference time of the first light image 602 is determined as the second reference color image to be assigned to the first light image 602.

[0107] At this stage, in step 311, a transformation matrix is ​​determined for aligning the selected first and second reference color images, where the transformation matrix also represents a transformation that estimates the underlying motion that occurred between the acquisition of the first dark image 601 and the acquisition of the first light image 602, based on the fact that the acquisition time of the selected reference color image is sufficiently close to the reference time of the associated first dark image 601 and first light image 602.

[0108] Preferably, step 311 is followed by optional step 312, which determines whether the underlying motion estimated by the determined transformation matrix is ​​excessive (e.g., exceeds a predetermined threshold). If excessive estimated motion is determined, system 100 outputs a warning to the operator indicating excessive motion of either or both of the objects in the illuminated scene and / or the imaging head 165, including the fluorescent camera 245 and color camera 250. Method 300 then resumes from step 301 (or, alternatively, can return to step 305).

[0109] If excessive movement is determined not to exist, method 300 proceeds to step 315. Before disclosing the operation of step 315, it should be noted that processing of step 312 is optional, and thus method 300 may proceed directly to step 315 after processing of step 311.

[0110] At this stage, the fluorescence imaging control module 281 (or color image processing control module 282, or any other module in the control unit 205) applies the transformation matrix determined in step 311 to the first dark image 601 to spatially align (pixel-by-pixel) the first dark image 601 with the first light image 602, thereby correcting for the estimated movement.

[0111] In this way, background image content that is common to the first dark image 601 and the first light image 602 (due to ambient light) and that was originally misaligned between the first dark image 601 and the first light image 602 (due to motion) can be advantageously (at least approximately) realigned by applying the transformation matrix to the first dark image 601 before background subtraction processing.

[0112] The fluorescence imaging control module 281 (or the color imaging control module 282, or another module in the control unit 205) then performs motion-compensated background subtraction in step 316. That is, background subtraction is performed by subtracting (pixel-by-pixel) the first light image 602 from the corrected first dark image 601. This improves the effect of removing background components compared to performing direct subtraction between the first dark image 601 and the first light image 602 (because the first dark image 601 is aligned with the first light image 602).

[0113] The fluorescence image obtained in this manner as the background subtracted image (background subtraction image) is stored in the image repository 283 (or in a dedicated repository within or accessible by the control unit 205 for storing background subtraction images), associated with a corresponding timestamp if necessary.

[0114] Additionally, in step 317, the background subtraction image is typically overlaid on the color image (e.g., forming a composite image) and displayed as part of a real-time video stream (which depicts the fluorescent target body part 290 in context with surrounding anatomical parts of the patient's body). In embodiments in which the aforementioned virtual color images 408, 409 are generated, the color image on which the background subtraction image is overlaid may be the virtual color image 409 (or either of the color images 403, 404) assigned to the light image 602 used to generate the background subtraction image.

[0115] In this embodiment, a background subtraction image is generated after acquisition of each image (dark or light) on the fluorescence channel. However, it should be understood that in other embodiments, a background subtraction image may be generated only for each new pair of dark and light images acquired on the fluorescence channel. Thus, in this embodiment, method 300 returns to step 305, and if it is determined that a next image (i.e., a new dark image) needs to be acquired on the fluorescence channel, in step 306, a second dark image 603, shown in FIG. 4, is captured for an exposure time T 1 during which excitation light source 210 is not illuminated and fluorescent materials in target body part 290 do not fluoresce. exp is obtained between

[0116] If the second dark image 603 is determined not to be overexposed, step 310 is performed to determine a third reference color image 405 associated with the second dark image 603 based on the reference time of the second dark image 603 indicated by the timestamp stored in association with the dark image 603 (following the same principles as those disclosed above for the association of the first and second reference color images 401, 403 with the first dark image 601 and light image 602). The resulting third virtual color image 410 is stored in association with a timestamp midway between the timestamps of the color images 405, 406.

[0117] At this stage, in step 311, method 300 can proceed to determine a transformation matrix that estimates the underlying motion that occurred during acquisition between the first light image 602 and the second dark image 603, following similar principles as previously disclosed in determining a transformation matrix for aligning the first dark image 601 and the first light image 602.

[0118] If, according to optional step 312, it is determined that there is no excessive motion, then in step 315, the determined transformation matrix is ​​applied to the first light image 602 to correct for motion between the first light image 602 and the second dark image 603.

[0119] In step 316, background subtraction with motion compensation is performed by subtracting the second dark image 603 from the corrected first light image 602.

[0120] In step 317, the resulting background-subtracted fluorescence image is saved and displayed as part of the real-time video stream overlaid on the corresponding color image. Again, according to an embodiment in which a virtual color image 410 is generated and assigned to the second dark image 603, as illustrated in FIG. 4, the corresponding color image may be the virtual color image 410 (or either color image 405 or 406).

[0121] Method 300 then returns to step 305 and iteratively performs steps 306-317 on newly acquired alternating light and dark images. This generates sequential motion-corrected, background-subtracted fluorescent images, which are then displayed as part of a real-time video stream overlaid on their corresponding color images. Specifically, in each iteration, a new reference color image is assigned to the newly acquired fluorescent channel image (step 310), a transformation matrix is ​​determined based on the relative alignment with the reference color image assigned to the previous fluorescent channel image (step 311), and the transformation matrix is ​​applied to the previous fluorescent channel image (step 315). If the newly acquired fluorescent channel image is a light image, the corrected previous dark image is subtracted from the new light image (step 316). On the other hand, if the newly acquired fluorescent channel image is a dark image, the new dark image is subtracted from the corrected light image (step 316).

[0122] Method 300 proceeds in this manner and repeats until it is determined at step 305 that there are no more images to acquire on the fluorescence channel (e.g., the surgical procedure is complete and the operator has entered a stop command), at which point method 300 ends at step 318.

[0123] While the operation of the motion-compensated background subtraction process of system 100 disclosed above was based on asynchronous control of the color and fluorescence channels, the operation of the motion-compensated background subtraction process of system 100 will be disclosed below in connection with an alternative channel control scheme. In this scheme, imaging control modules 281 and 282 control the acquisition of alternating dark / light images 601-603 and their corresponding color images 401-406 to maintain synchronization. This ensures that the reference time of each fluorescence channel image (dark or light) is aligned with the acquisition time of the corresponding color image. For example, in FIG. 5, dark image 601 is acquired between the acquisition of color images 401 and 402, and its acquisition start time approximately coincides with the acquisition start time of corresponding color image 401. Light image 602 is acquired between the acquisition of color images 403 and 404, and its acquisition start time approximately coincides with the acquisition start time of corresponding color image 403. Furthermore, the dark image 603 is acquired between the acquisition of the color images 405 and 406, and the acquisition start time thereof substantially coincides with the acquisition start time of the corresponding color image 405.

[0124] To perform motion-compensated background subtraction when a series of color images and a series of fluorescence channel images are acquired synchronously, system 100 may be configured to perform a process similar to that of method 300 described above. However, the difference is that the determination of a reference color image (steps 304 and 310) corresponding to an acquired fluorescence channel image (dark image or light image) is performed by simply selecting a color image that is temporally aligned with the reference time of the fluorescence channel image (based on its closer framing view to the fluorescence channel image). For example, referring to FIG. 5 , the operation of this modified method 300 assigns color image 401, color image 403, and color image 405 as reference color images for dark image 601, light image 602, and dark image 603, respectively.

[0125] Referring to FIG. 6, an exemplary embodiment of an endoscopic imaging system, i.e., an endoscope system 500, configured to perform motion-compensated fluorescence background subtraction according to the present invention is shown.

[0126] In particular, the endoscope system 500 is used in a medical procedure to image an internal body part 103 (a part defining a body cavity that is not normally visible) within the body of a patient 106. The body cavity is accessible through an opening 112, which may be a natural orifice or a small incision made in the skin of the patient 106. The body part 103 includes a target body part 109 (e.g., a lesion such as a tumor) that is the subject of the medical procedure and that includes a fluorescent substance (e.g., a fluorescent dye adapted to accumulate in the tumor and previously administered to the patient 106).

[0127] For example, in diagnostic applications, the endoscopy system 500 allows for the detection and monitoring of lesions, and in (minimally invasive) surgical procedures, it allows for the identification of lesions to be removed. In therapeutic applications, it allows for the definition of the boundaries of lesions to be treated. Examples of these medical procedures include diagnostic applications such as gastroscopy, colonoscopy, and esophagoscopy, and surgical applications such as arthroscopy, laparoscopy, and thoracoscopy. Furthermore, therapeutic applications include ablation, dilation, and stent placement.

[0128] The endoscopy system 500 is intended to use a combination of fluorescent (endoscopic) technology for displaying fluorescent substances contained in a target site 109 within a body cavity and standard (endoscopic) technology for displaying a visible image visible to the human eye. For this purpose, the endoscopy system 500 is composed of two endoscopy units: a primary endoscopy unit (hereinafter referred to as the "motherscope" 115m) and an auxiliary endoscopy unit (hereinafter referred to as the "babyscope" 115b).

[0129] The mother scope 115m includes the following components: A central unit of the mother scope 115m is used to manage its operation and is implemented, for example, as a trolley 118m. A monitor 121m (e.g., attached to the top of the trolley 118m) is used to display images of the body part 103 during the medical procedure. A video interface 124m (e.g., an SDI (Serial Digital Interface) port on the back of the trolley 118m) is used to exchange video information with the outside. A probe 127m (e.g., connected to the trolley 118m via a cable) is used to perform the procedure on the patient 106. For example, the probe 127m is configured as an elongated shaft for insertion into the body cavity of the body part 103. The shaft may be rigid, but is preferably flexible so that it can smoothly navigate through the body cavity (even if the path is curved). A distal end (or tip) 130m of the probe 127m is used to reach the target site 109 and perform illumination and color image acquisition (described in more detail below).

[0130] The probe 127m includes a handle 133m at its proximal end (outside the body cavity 103) that controls the tip 130m (driven via a control cable, not shown). The probe 127m includes one or more working channels accessible through corresponding working ports near the probe's proximal end (only one of which is shown in FIG. 6 as 136m). Various tools used during a medical procedure (e.g., snares, forceps, knives, clip appliers, etc.) can be inserted through the working channels. Dedicated working channels can also be connected to a fluid injection / aspiration device (not shown) to flush the tip 130m and the body cavity of the body part 103 during a medical procedure.

[0131] The baby scope 115b includes the following components: Like the mother scope 115m, the baby scope 115b also includes a central unit for managing its operation. For example, this central unit is configured as a trolley 118b. A monitor 121b (for example, attached to the top of the trolley 118b) is used to display (additional) images of the body cavity of the body part 103 during the medical procedure. A video interface 124b is used to send and receive video information to and from the outside. A probe 127b (for example, connected to the trolley 118b via a cable) is used to perform operations on the patient 106. For example, this probe is configured as a (preferably flexible) elongated shaft. A distal end (tip) 130b of the probe 127b is used to reach the target area of ​​the medical procedure within the body cavity of the body part 103 and to illuminate and acquire fluorescent and color images (as will be described in detail below).

[0132] In certain embodiments, probe 127b is configured to be thinner than probe 127m, and probe 127b is inserted into working channel 136m such that its tip 130b reaches tip 130m of probe 127m (its size and flexibility allow it to be inserted without compromising the maneuverability of the latter).

[0133] Next, the mother scope 115m will be described in more detail with reference to the functional block diagram of the endoscope system 500 shown in Fig. 7. In particular, the mother scope 115m includes at least an image acquisition unit for imaging a region of interest 109 of the body part 103 within a corresponding field of view 221m, and an illumination unit for illuminating the region of interest 109.

[0134] Starting with the illumination unit, the unit includes a white light source 209m (e.g., an LED, a halogen lamp, or a xenon lamp, which may be mounted in the trolley 118m) configured to emit white light. This white light is transmitted to the illumination optics 212m via a fiber optic bundle 218m (a non-coherent fiber optic bundle extending along the probe). The illumination optics 212m (mounted at the tip of the probe of the motherscope 115m) is configured to receive the white light and illuminate the body part 103 (including the region of interest 109) within the field of view 221m.

[0135] Turning to the description of the image acquisition unit of the Motherscope 115m, said unit comprises a collection optics 224m configured to collect visible light reflected by objects present in a field of view 221m illuminated by white light, and a color camera 239m including at least a color imager 240m (for example of the CCD type) suitable for detecting the collected visible light and for generating accordingly a color image representative of what is visible to the human eye in the illuminated field of view 221m.

[0136] In a videoscope configuration, color camera 239m is located at the tip of the probe, with a digital connection 241m transmitting the captured color image to trolley 115m (alternatively, color camera 239m could be located within trolley 115, in which case a coherent fiber optic bundle could be used to transmit visible light from collection optics 224m to color camera 239m).

[0137] 7, the baby scope 115b includes at least an image acquisition unit for capturing an image of a region of interest 109 of a body part 103 within a corresponding field of view 221b, and an illumination unit for illuminating the region of interest 109. In general, the field of view 221b of the baby scope 115b is different from the field of view 221m of the mother scope 115m. For example, the field of view 221b is narrower than the field of view 221m and overlaps (at least partially) with the field of view 221m, and both fields of view 221b and 221m include the target body part that is the object of interest 109 (as shown in the example of FIG. 7).

[0138] Starting with the illumination unit, the unit comprises a near-infrared excitation light source 203b (e.g. a laser light source or an LED, for example located inside the trolley of the baby scope 115b) configured to emit excitation light having a wavelength and energy suitable to excite the fluorescent substances of the fluorescent dyes present in the region of interest 109 of the body part, thereby causing them to emit fluorescence in the near-infrared spectrum of the corresponding fluorescence emission.

[0139] The lighting unit of the baby scope 115b further includes a white light source 209b (eg, an LED, a halogen lamp, or a xenon lamp inside the trolley 118b) configured to emit white light.

[0140] An incoherent fiber optic bundle 218b along the fiber optic probe is configured to transmit the emitted excitation light and white light (mixed together if emitted simultaneously) to illumination optics 212b (not shown in FIG. 7) located at the tip of the probe. The illumination optics 212b is configured to illuminate the body part 103 within a field of view 221b of the image acquisition system with the received excitation light and white light, illuminating the target body part 109. Alternatively, two separate illumination optics, each with a corresponding incoherent fiber optic bundle, can be used to independently illuminate the body part 103 within the field of view 221b with excitation light and white light from each light source 203b, 209b.

[0141] Turning to the image acquisition unit of the baby scope 115b, the unit includes a collection optical system 224b configured to collect light present within the field of view 221b. The light collected within the field of view 221b may include: fluorescence emitted by fluorescent dyes present within the subject's region of interest 109 (by absorbing the excitation light provided by the excitation light source 203b); spurious fluorescence that may be emitted by fluorescent dyes other than the fluorescent dye of the target fluorescent agent, such as endogenous fluorescent dyes present within the observed body region of interest 109, by absorbing components of the white light provided by the light source 215 or the light source 209m of the mother scope 115m; and excitation light reflected by objects present within the field of view 221b. Furthermore, the captured light may include visible light reflected by objects within the field of view 221b illuminated by the white light emitted from the light source 209b or the light source 209m of the mother scope 115m.

[0142] The image acquisition unit further includes a coherent fiber optic bundle 230b positioned along the probe and configured to transmit light captured by collection optics 224b to beam splitter 227b. Similar to beam splitter 235 described above with respect to system 100 shown in Figure 2, beam splitter 227b (e.g., a dichroic mirror) is configured to separate the captured light into two channels: a first channel of captured light in the near-infrared spectrum (including the fluorescent emission spectrum) and a second channel of captured light in the visible light spectrum.

[0143] The image acquisition unit further includes a fluorescence camera 236b including at least a fluorescence imager 234b and a corresponding emission filter 233b. Similar to the emission filter 240 described above with respect to the system 100 shown in Figure 2, the filter 233b is a bandpass filter having a fluorescence detection passband corresponding to the fluorescence emission spectrum of fluorescent substances present in the targeted body region of interest 109, while being configured (in ideal operation) to block components of the excitation light reflected by objects within the field of view 221b and reflected ambient light other than the fluorescence and excitation light (e.g., components of the white light emitted from the white light source 209m or white light source 209b of the Motherscope 115m that are in the near-infrared spectrum or that are inadvertently directed toward the filter 233b due to non-ideal operation of the beam splitter 227b).

[0144] As with the image sensor 241 in the system 100 of FIG. 2, the image sensor 234b can be any high-sensitivity sensor that is capable of capturing low-illumination images in the near-infrared region.

[0145] Referring again to the second channel of the beam splitter 227b, the image acquisition unit of the baby scope 115b further comprises a color camera 239b. Similar to the color camera 250 of the system 100 described above, the color camera 239b comprises an image sensor 240b (e.g., of the CCD or CMOS type) capable of separately detecting the intensity of each color component contained in the reflected light (e.g., using an RGB filter or other color filter array).

[0146] Continuing with FIG. 7, central unit 242b and central unit 242m are used to control the operation of baby scope 115b and mother scope 115m, respectively. Each central unit 242b, 242m includes multiple units interconnected via respective bus structures 245b, 245m. In particular, each central unit 242b, 242m includes at least one microprocessor (μP) 248b, 248m that provides the logic processing functions for the respective unit. ROM 251b, 251m stores basic code for bootstrapping the central units 242b, 242m, and RAM 254b, 254m is used as working memory by the microprocessors 248b, 248m. The central units 242b, 242m are equipped with mass storage devices 257b, 257m for storing programs and data. Additionally, each central unit 242b, 242m includes multiple controllers 260b, 260m for peripheral devices (I / O units). In particular, the controller 260m of the motherscope 115m includes at least one color imaging control module configured to control the operation of the white light source 209m and the color camera 239m, and to acquire and store a series of color images in a color image repository maintained in the central unit 242m (e.g., a repository in the mass storage device 275m or a dedicated repository within or accessible to the central unit 242m).

[0147] Controller 260 of baby scope 115b includes at least a fluorescence imaging control module and a color imaging control module, and is configured to control the operation of excitation light source 203b, fluorescence camera 236b, white light source 209b, and color camera 239b to acquire and store a series of color images and alternating dark / light images, similar to the process described above for acquiring and storing color images and alternating dark / light images according to the configuration of control modules 281-282 in imaging head 165 of system 100.

[0148] In particular, similar to the fluorescent imaging control module 281 in the imaging head 165 of the system 100, the fluorescent imaging control module of the baby scope 115b is configured as follows. In the fluorescence acquisition channel, the operation of the fluorescence camera 236b and the on / off of the excitation light source 203b are coordinated to alternately acquire light images acquired using the excitation light source 203b and dark images of the same shape and size acquired without using the excitation light source 203b (see, for example, fluorescence channel images 601 to 603 shown in FIG. 4). The acquired alternating dark / light images 601-603 are stored in a corresponding fluorescence channel image repository (e.g., a repository in mass memory 275b or in central unit 242b or other dedicated repository accessible thereto) in association with a timestamp indicating the time of their acquisition.

[0149] To acquire light images with appropriate exposure times, the fluorescence imaging control module is configured to control and turn on the light source 203b so as to continuously emit excitation light pulses over a time interval corresponding to the desired exposure time. To acquire dark images, the fluorescence imaging control module is configured to keep the excitation light source 203b off between the end of acquisition of each light image and the start of acquisition of the next light image.

[0150] Similar to the color imaging control module 282 in the imaging head 165 of the system 100, the color imaging control module of the baby scope 115b is configured as follows. The white light source 209b is turned on (while the baby scope 115b is positioned near and observing the target body region of interest 109), and preferably remains on to constantly illuminate the field of view 221b. Controlling color image acquisition 239b to acquire a series of color images (e.g., including color images 401-406 shown in FIG. 4) via the color acquisition channel while white light source 209b illuminates the targeted body region of interest 109, the acquisition frequency of which is higher than the acquisition frequency of the sequence of alternating dark image / light image 601-603.

[0151] Furthermore, similar to the imaging control modules 281, 282 in the imaging head 165 of the system 100, the fluorescent imaging control module and the color imaging control module of the baby scope 115b preferably control image acquisition in the fluorescent channel and the color channel, respectively, independently of one another, so that a series of color images and fluorescent channel images are acquired asynchronously in their respective channels (e.g., as shown in FIG. 4).

[0152] The color images 401-406 may be acquired with the same (or approximately the same) size and shape as the fluorescence channel images 601-603 (and, optionally, each color image 401-406 may be acquired with the exposure time T of the corresponding alternating dark / light images 601, 602, 603 by combining multiple color images acquired successively with shorter exposure times, e.g., as shown in FIG. 4 ). exp The signal can be acquired over an acquisition time corresponding to

[0153] The color images 401 are stored by the color imaging control module in a corresponding color image repository (e.g., a repository in mass storage device 275b, or in central unit 242b, or any dedicated repository accessible thereto) along with a corresponding timestamp indicating the time of acquisition.

[0154] Successive acquired dark and light images may contain the same (or similar) background image content due to reflected ambient light (e.g., reflections of white light emitted from light source 209b or light source 209m of Motherscope 115m, or unwanted fluorescent emissions due to white light from light sources 209m, 209b). This background image content may pass through emission filter 233b (e.g., due to non-ideal operation of the filter) and reach the light receiving surface of fluorescence imager 234b. However, this common background image content may be spatially shifted in the successive acquired dark and light images due to movement of objects in the imaged scene or movement of fluorescence camera 236b relative to the imaged scene.

[0155] To solve this problem, the central unit 242b of the baby scope 115b is configured to perform background subtraction with motion compensation on a series of alternating dark / light images acquired asynchronously with the series of color images by performing the previously disclosed method 300 shown in Figure 3. This is done while the endoscope system 500 is being used during an endoscopic procedure (e.g., to observe the target body part 109 as shown in Figure 7).

[0156] In particular, after administering a fluorescent substance to the patient, the endoscopic procedure begins. After administering general or local anesthesia as needed, the physician inserts the probe of the Motherscope 115m, activated by the (medical) operator, into the patient's body cavity until its tip reaches the target area 109 in the patient's body part where a tumor (or other lesion) may be present. During the procedure, the white light source 209m of the Motherscope 115m constantly illuminates the scene within the field of view 221m, and the color camera 239m is continuously used to acquire Motherscope color images of the illuminated scene in real time and display them on the Motherscope 115m's monitor 121m.

[0157] At some point during an endoscopic procedure, the physician inserts the probe of the baby scope 115b into the working channel of the mother scope 115m and advances it until its tip reaches the same target area 109 of the patient's body part as the tip of the mother scope's probe. While the probe of the baby scope 115b is being inserted along the working channel of the mother scope 115m, or when the tip of the probe of the baby scope 115b reaches the tip of the probe of the mother scope near the target area 109, the medical operator can input a start command to the baby scope 115b. In response, the color imaging control module of the central unit 242b turns on the white light source 209b (which preferably remains on at all times while the baby scope 115b is in operation) and begins controlling the acquisition of a series of color images (e.g., including color images 401-406 shown in FIG. 4) in the color acquisition channel using the color camera 239b. The color imaging control module proceeds with the acquisition of a series of images, storing these color images with corresponding timestamps in a dedicated repository.

[0158] Operation of method 300 shown in FIG. 3 begins at step 301, where the color imaging control module of baby scope 115b controls the acquisition of a series of color images on the color channels, and the fluorescent imaging control module of baby scope 115b controls the fluorescent camera 236b to acquire a first fluorescent channel image (e.g., dark image 601 shown in FIG. 4) to control acquisition on the fluorescent channels.

[0159] Each subsequent step of method 300 is performed by the fluorescent imaging control module or the color imaging control module of central unit 242b (or another module within central unit 242b or another unit of baby scope 115b configured to perform that particular step), respectively. This configuration is similar to the operations disclosed above, in which each step is performed by the fluorescent imaging control module 281 or the color imaging control module 282 of central unit 205 of system 100 (or another module within central unit 205 or another unit of system 100 configured to perform that particular step).

[0160] These steps are performed iteratively in conjunction with a series of acquired alternating light / dark fluorescence channel images, resulting in a series of (motion-compensated) background-subtracted fluorescence images on the fluorescence channels that are displayed sequentially on monitor 121b of BabyScope 115b (overlaid with their associated color images acquired on the color channels of BabyScope 115b, and possibly with color images simultaneously acquired by MotherScope 115m), as part of the displayed real-time video stream.

[0161] Method 300 continues in this manner until it is determined in step 305 that there are no more fluorescence channel images to acquire on the fluorescence channel (e.g., when operation of baby scope 115b is complete and the operator has entered a stop command into baby scope 115b), at which point method 300 ends in step 318.

[0162] Similar to system 100, when a series of color images and alternating dark and light images are acquired synchronously (e.g., as shown in FIG. 5), endoscopic system 500 may be configured to perform method 300 shown in FIG. 3, except that determining a reference color image for an acquired fluorescence channel image (steps 304 and 310) can be simply accomplished by selecting a color image acquired at a time that coincides in time with the reference time of the fluorescence channel image.

[0163] Another embodiment of the present disclosure provides a computer program. The computer program, when executed, causes a computing device of a medical fluorescence imaging system, such as system 100 or endoscopy system 500, to perform the above-described method 300. Yet another embodiment provides a computer program product, including a computer-readable storage medium embodied with the computer program. When the computer program is loaded into the working memory of a computing device of a medical fluorescence imaging system, the computing device is configured to perform the same method. However, the computer program may also be implemented as a stand-alone module, as a plug-in to, or directly incorporated within, existing software (e.g., an imaging system manager). In either case, similar considerations apply whether the program is configured differently or includes additional modules or functions. Other types of storage structures may also be used, and equivalent structures may be substituted, without being limited to physical storage media. The computer program may be in a format usable by any computing device, configuring the device to perform the desired operations. In particular, the program may be in the form of external or embedded software, firmware, or microcode (object code or source code (e.g., compiled or interpreted)). The program may also be provided on any computer-readable storage medium. By storage medium, we mean a tangible medium, other than a transferable signal itself, that can hold or record instructions for use by a computing device. For example, the storage medium may be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, and examples include a fixed disk (if the program is pre-installed), a removable disk, or a memory key (e.g., USB type).The computer program may be downloaded to the computing device from the storage medium or via a network (e.g., the Internet, a wide area network, and / or a local network including fiber optics, wireless connections, and network devices). One or more network adapters in the computing device receive the program from the network and store it in one or more storage devices. In any case, the operations of the above-described method 300 may also be implemented by a hardware configuration (e.g., electronic circuits integrated in one or more chips made of semiconductor material, e.g., FPGAs or ASICs) or by a suitable combination of software and hardware.

[0164] [Variations] Although the present disclosure has been described with reference to particular embodiments of the present invention, it will be apparent to those skilled in the art that various omissions, substitutions, and changes can be made in the arrangement and details of the disclosed embodiments, and that other embodiments may also be made.

[0165] For example, while the disclosed systems 100 and 500 use color cameras 250 and 239b to cooperate with each of white light sources 215 and 209b to acquire a series of color images 401-406 in addition to the fluorescence channel images acquired by fluorescence cameras 245 and 236b, it should be noted that other reflectance cameras configured to acquire images in response to reflected light can be used in place of or in combination with color cameras 250 and 230b to acquire a series of reflected images (which can be used to provide motion information for motion-compensated background subtraction between fluorescence channel images in accordance with embodiments of the present invention). For example, the reflectance camera may include a simple imager that cannot individually detect the intensity of each color component of the incident reflected light, i.e., a CCD or CMOS sensor without a CFA. In this case, the reflectance camera can generate monochrome (grayscale) images in response to the reflected light detected by the imager. In another variation, a reflectance camera may be used in combination with a light source configured to emit reflected light other than visible light (but light that does not induce significant fluorescence in an illuminated target body part containing a fluorescent substance and that is different from the fluorescence emitted by that substance when excited), and a reflected image (e.g., an IR image or a UV image) may be acquired in response to detection of the non-fluorescent reflected light, such as far-infrared or ultraviolet light when a fluorescent substance that can be excited by near-infrared light is used.

[0166] For example, while the disclosed systems 100, 500 use two separate cameras 245, 250 and 236b, 239b with nearly identical fields of view 104, 221b (via beam splitters 235, 227b) to acquire the fluorescence and color images, respectively, cameras with only overlapping (or different) fields of view may be used for the fluorescence and reflected images. In this case, post-image acquisition processing steps can be performed to align the fields of view of the fluorescence channel images and the reflected images.

[0167] Alternatively, a single camera may be used to acquire both color (or other reflectance) and fluorescence channel images, either asynchronously or synchronously. For example, a single reflectance+fluorescence camera may be used, including an imager with RGB+NIR filters, allowing the red, green, and blue components of the incident reflected visible light and emitted fluorescence to reach corresponding filtered pixel areas on the imager surface. In yet another variation, the imager may be a hyperspectral sensor with multiple surfaces sensitive to red, green, and blue subbands, or indeed individual bands of near-infrared light.

[0168] It should further be noted that the imaging system according to the present invention may not have a white light source used to acquire the reflected images (particularly in the case of imaging systems used in medical procedures where the body part being examined is exposed to artificial light and / or sunlight in the room where the medical procedure is performed).

[0169] Furthermore, an imaging system according to the present invention may not include a reflectance camera for acquiring reflectance images used to perform motion-compensated image subtraction on images acquired by the fluorescence camera, in which case the system may acquire reflectance images from an external source configured to acquire reflectance images of the same scene as imaged by the fluorescence camera.

[0170] Although the disclosed embodiments refer to the use of fluorescent materials excited by near-infrared light, and corresponding near-infrared excitation light sources 210, 203b and near-infrared-sensitive imagers 241, 234b used in systems 100, 500, the fluorescent materials used to mark target body parts may be excited by light outside the near-infrared spectrum, for example, the ultraviolet (UV) spectrum, the far-infrared (IR) spectrum, or portions of the visible spectrum, in which case appropriate excitation light (e.g., ultraviolet or infrared light) would be used in combination with an imager capable of detecting fluorescence emitted from the fluorescent object in response to the excitation light (e.g., in the ultraviolet or infrared spectrum).

[0171] For example, in the disclosed embodiment, the white light sources 215, 209m, and 209b are maintained in a constantly lit state during operation of the system 100 and during operation of the mother scope 115m and the baby scope 115b in the endoscope system 500, respectively, but these white light sources may also be controlled to be in an off state during acquisition of dark or light images of at least the fluorescence channel.

[0172] Furthermore, in the endoscopic system 500 shown in FIG. 6, two separate central units 242m and 242b corresponding to the mother scope 115m and the baby scope 115b, respectively, are provided, but a single central unit may also be used (or at least some of the components of the central units 242m and 242b may be shared between the mother scope 115m and the baby scope 115b).

[0173] Referring to FIG. 4, the series of color images 401-406 are shown being acquired at twice the acquisition frequency of the asynchronously acquired alternating dark / light images 601-603, but the acquisition frequency of the color images may be more than twice the acquisition frequency of the fluorescence channel images (so that more color images are acquired between the color image acquired after acquisition of a fluorescence channel image and the color image acquired immediately before acquisition of the next fluorescence channel image) or may be equal to the acquisition frequency of the fluorescence channel images (so that the color image acquired after acquisition of a fluorescence channel image coincides with the color image acquired immediately before acquisition of the next fluorescence channel image).

[0174] Similarly, referring to FIG. 5, although the series of color images 401-406 are shown being acquired at twice the acquisition frequency of the synchronously acquired alternating dark / light images 601-603, the acquisition frequency of the color images may be more than twice the acquisition frequency of the fluorescence channel images (so that more color images are acquired between the color image acquired during the acquisition of one fluorescence channel image and the color image acquired immediately before the acquisition of the next fluorescence channel image) or may be equal to the acquisition frequency of the fluorescence channel images (in which case each color image is acquired at the acquisition time of the corresponding fluorescence channel image).

[0175] Furthermore, the operating principles of the method 300 described above also apply to embodiments in which the reflectance images are acquired less frequently than the fluorescence channel images.

[0176] It should also be noted that the image acquisition frequency in the color and fluorescence channels may vary during different operations of a fluorescence imaging system according to the present invention, or even during the same operation.

[0177] For example, although the operation of method 300 is disclosed with reference to the example of FIGS. 4-5 (where the acquisition time of color images 401-406 is substantially the same length as the exposure time used to acquire light / dark images 601-603), the acquisition time of the reflectance images may be (significantly) shorter than the acquisition time of light / dark images 601-603 (in such cases, at least two reflectance images are acquired during the acquisition time of a corresponding light or dark fluorescence channel image). In this case, the reference time used to determine a reference reflectance image associated with a given fluorescence channel image may be set to a midpoint in the acquisition of that fluorescence channel image.

[0178] 3 , particularly with respect to the two solutions disclosed above for determining and assigning a reference color image for a given fluorescence channel image to be analyzed (i.e., interpolating between two acquired color images that sandwich the fluorescence channel image, or selecting one of the two color images that is closest in time to the fluorescence channel image), it should be noted that while these solutions are disclosed as alternatives, they may also be combined. For example, one of two surrounding color images may be selected that is sufficiently close to the fluorescence channel image (e.g., within a predetermined time interval), and otherwise a virtual reference color image may be determined by interpolating between the two surrounding color images.

[0179] Additionally, although the display process of step 371 of method 300 is disclosed in connection with displaying the resulting sequence of motion-compensated background subtracted images on monitors 155, 140, 121b of systems 100, 500, the resulting sequence of images may alternatively or additionally be output (e.g., wirelessly) to and displayed on an external device (e.g., a virtual reality head-mounted display (VR goggles), a smart TV, an in-room screen, etc.) connected to these systems 100, 500.

[0180] In connection with the operation of method 300 by endoscopic system 500 shown in Figures 6 and 7, the series of color images acquired by mother scope 115m may also be used in place of or in combination with the series of color images acquired by baby scope 115b in accordance with the operation of method 300 disclosed above to perform motion-compensated background subtraction on the series of fluorescence channel images including alternating dark / light images acquired by mother scope 115m.

[0181] Furthermore, although the endoscopic system 500 shown in FIG. 6 includes a mother scope 115m and a baby scope 115b, it should be noted that the method 300 can also be performed by an endoscopic system having a single scope with at least a fluorescence camera.

[0182] Additionally, while method 300 is disclosed as one example for performing motion-compensated background subtraction, other modifications to the disclosed method 300 are possible, such as using functionally equivalent steps or portions thereof, omitting non-essential steps, or adding additional optional steps. Additionally, the steps of method 300, or modifications thereof, may be performed in a different order, in parallel, or with partial interleaving.

[0183] Although the operation of the motion-compensated background subtraction method of the present invention is disclosed in relation to systems 100 and 500 used in surgical and endoscopic procedures, it is also advantageously applicable to systems / apparatuses that require viewing of target body regions in other medical (e.g., surgical / diagnostic / therapeutic) or cosmetic applications using fluorescence imaging.

[0184] In one embodiment, a surgical procedure is provided, comprising the steps of: imaging a patient's body part using the method described above in accordance with the present disclosure to display a series of motion-corrected, background-subtracted fluorescence images; and administering a procedure to the body part based on the displayed series of images. However, the proposed method is applicable to any surgical procedure in the broadest sense (e.g., therapeutic, preventative, cosmetic / aesthetic, etc.) and to any body part in any patient.

[0185] In another embodiment, a diagnostic method is provided, comprising the steps of: imaging a patient's body part using the method described above in accordance with the present disclosure to display a series of motion-corrected, background-subtracted fluorescence images; and analyzing the body part based on the displayed image sequence. However, the proposed method is applicable to any diagnostic application in the broadest sense (e.g., health status assessment, detection of new lesions, monitoring of known lesions, etc.) and to the analysis of any body part in any patient.

[0186] In one embodiment, a therapeutic method is provided, comprising the steps of: imaging a patient's body part using the method described above in accordance with the present disclosure, thereby displaying a series of motion-corrected, background-subtracted fluorescence images; and administering a therapy to the body part based on the displayed image sequence. However, the proposed method is applicable to any therapeutic treatment in the broadest sense (e.g., curing, preventing the progression of, or preventing the onset of a pathological condition, or simply improving the patient's comfort) and to any body part of any patient.

[0187] In either case, the method aids the physician in his work, but only provides intermediate results, and medical procedures in the narrow sense are always performed by the physician himself. The target body part can be of any type (e.g., organs, regions, tissues, etc., such as the liver, prostate, or heart), and can relate to any condition (e.g., in vivo, cadaveric, or ex vivo specimens (e.g., biopsy samples), etc.), or to any patient (e.g., human, animal, etc.).

[0188] The subject of the treatment is defined by a "target condition" in a body part, however, the target condition can be of any type (e.g., pathological tissue such as a tumor or inflammation, healthy tissue, etc.).

[0189] The fluorescent substance used to image the target body region may be either exogenous / endogenous or foreign / endogenous (e.g., any fluorescent reagent, naturally occurring fluorescent component, etc.). The fluorescent substance may be administered by any method (e.g., non-invasive means, such as oral administration for gastrointestinal imaging, administration to the airway via a nebulizer, or topical spray application or local introduction during a surgical procedure) without requiring specialized medical treatment or posing health risks to the patient (e.g., intramuscular injection), and at any timing (e.g., before, immediately before, continuously during, etc.). Furthermore, if the fluorescent substance is endogenous, this administration step itself may be omitted. Furthermore, the fluorescent substance may circulate within the target body region and may not necessarily bind to biological components.

[0190] Furthermore, it is intended that specific components and / or method steps described in connection with any embodiment of the present disclosure may be incorporated into any other embodiment as part of a common design choice. In addition, items shown in the same group or in different embodiments, examples, or alternatives should not be considered to be de facto equivalents of one another, but rather are separate and distinct entities.

Claims

1. A method (300) for motion-compensated background image subtraction in an imaging system (100, 500) for imaging a scene including at least one target body part (290, 109) containing fluorescent material, the imaging system comprising: a light source (210, 203b) configured to emit excitation light for illuminating the target body part, the excitation light being suitable for causing fluorescence emission from the target body part; a fluorescence camera (245, 236b) having a fluorescence emission detection passband configured to detect the fluorescence emitted from the target body part; The method comprises: acquiring a series of reflected images (401-406) acquired by a reflected camera (250, 239b) configured to detect light reflected due to illumination conditions different from the excitation light; While acquiring the series of reflectance images, using the fluorescence camera, acquiring (301, 306) a series of fluorescence channel images (601-603) that alternate between light images (602) acquired while causing the target body part to emit fluorescence using the light source and dark images (601, 603) acquired without using the light source and causing the target body part to emit fluorescence; determining (304) a first reflectance image (401) of the series of reflectance images associated with a first fluorescence channel image (601) having a first reference time; determining (310) a second reflection image (403) of the series of reflection images associated with a second fluorescence channel image (602) acquired by the fluorescence camera consecutively to the first fluorescence channel image (601) and having a second reference time; determining (311) a transformation matrix based on the relative alignment of the first and second reflected images (401, 403); applying the transformation matrix to one of the first and second fluorescence channel images (601, 602) to align the first and second fluorescence channel images (315); performing a subtraction operation (316) between one of the first and second fluorescence channel images to which the transformation matrix has been applied and the other of the first and second fluorescence channel images to generate a background subtraction image; and outputting (317) the background subtraction image for display.

2. The first reference time is between the acquisition of the first reflection image (401) and the acquisition of an adjacent third reflection image (402), and the second reference time is between the acquisition of the second reflection image (403) and the acquisition of an adjacent fourth reflection image (404), and the step (311) of determining the transformation matrix includes: using the third reflectance image (402) to determine a first interpolation transformation matrix for aligning the first reflectance image (401) with the first fluorescence channel image (601); and using the fourth reflection image (404) to determine a second interpolation transformation matrix for aligning the second reflection image (403) with the second fluorescence channel image (602).

3. The step of determining the first interpolation transformation matrix comprises: calculating a first interpolation coefficient between the first and third reflection images (401, 402) as a function of their relative acquisition times and the first reference time; determining a first transformation matrix for aligning the first and third reflected images (401, 402); applying the first interpolation coefficients to the first transformation matrix to obtain the first interpolated transformation matrix; The step of determining the second interpolation transformation matrix comprises: calculating a second interpolation coefficient between the second and fourth reflection images (403, 404) as a function of their relative acquisition times and the second reference time; determining a second transformation matrix for aligning the second and fourth reflected images (403, 404); and applying the second interpolation coefficients to the second transformation matrix to obtain the second interpolated transformation matrix.

4. applying the first interpolation coefficients to the first transformation matrix, decomposing the first transformation matrix into motion components; applying the first interpolation coefficient to the motion component; reconstructing the first interpolated transformation matrix from the interpolated motion components to obtain the first interpolated transformation matrix; applying the second interpolation coefficients to the second transformation matrix, decomposing the second transformation matrix into motion components; applying the second interpolation coefficients to the motion components; and reconstructing the second interpolated transformation matrix from the interpolated motion components to obtain the second interpolated transformation matrix.

5. applying the first interpolation coefficients to the first transformation matrix, determining a fractional power of the first transformation matrix corresponding to the first interpolation coefficient; applying the second interpolation coefficients to the second transformation matrix, 4. The method of claim 3, further comprising determining a fractional power of the second transformation matrix corresponding to the second interpolation coefficient.

6. 6. The method of claim 2, further comprising determining (311) the transformation matrix as a step performed by: determining the transformation matrix as a function of the first and second reflection images (401, 403) and the first and second interpolated transformation matrices.

7. The step of determining the transformation matrix (311) applying the first interpolation transformation matrix to the first reflection image (401) to obtain a first virtual reflection image (408) that is substantially aligned with the first fluorescence channel image (601); applying the second interpolation transformation matrix to the second reflection image (403) to obtain a second virtual reflection image (409) that is substantially aligned with the second fluorescence channel image (602); The method of any one of claims 2 to 5, further comprising as a step performed by: determining a transformation matrix for aligning the first and second virtual reflection images (408, 409).

8. The step of determining the transformation matrix (311) applying the first interpolation transformation matrix to align the first fluorescence channel image (601) with the first reflectance image (401); applying the second interpolation transformation matrix to align the second fluorescence channel image (602) with the second reflectance image (403); The method of any one of claims 2 to 5, comprising as a step performed by: determining a transformation matrix for aligning the first and second reflected images (401, 403).

9. The step (304) of determining the first reflected image (401) comprises: selecting a reflection image acquired closest to the first reference time; The step (310) of determining the second reflected image (403) comprises: selecting a reflection image acquired at a time closest to the second reference time; 2. The method of claim 1, wherein the step of determining the transformation matrix (311) is performed by determining a transformation matrix for aligning the selected first and second reflection images (401, 403).

10. 10. The method of claim 1 or 9, wherein the reflectance images (401, 403, 405) are acquired at times that correspond to reference times during the acquisition of the corresponding fluorescence channel images (601, 602, 603), respectively.

11. The method according to any one of the preceding claims, wherein the reflectance images (401-406) are acquired more frequently than or equally frequently to the fluorescence channel images (601, 602, 603).

12. the imaging system (100, 500) The reflection camera (250, 239b) is further provided, acquiring the series of reflected images (401-406) A method according to any one of the preceding claims, comprising acquiring the series of reflected images (401-406) using the reflected camera.

13. The method of claim 12 , wherein the reflectance camera has a field of view that substantially overlaps with the field of view of the fluorescence camera.

14. acquiring the series of reflected images acquiring a plurality of first reflection frames (407) over a period of time corresponding to an exposure time used to acquire said first fluorescence channel image (601); acquiring a first reflection image (401) by combining the plurality of first reflection frames (407) to form a single first reflection image (401); acquiring a plurality of second reflection frames over a period of time corresponding to an exposure time used to acquire the second fluorescence channel image (602); and acquiring a second reflected image (403) by combining the plurality of second reflected frames to form a single second reflected image (403).

15. determining (302, 307) whether either the first fluorescence channel image (601) or the second fluorescence channel image (602) is overexposed; 15. The method of claim 1, further comprising the step of outputting (303, 308) a warning regarding excessive ambient light if either the first fluorescence channel image (601) or the second fluorescence channel image (602) is determined to be overexposed.

16. determining (312) whether the motion estimated by the transformation matrix for registering the first and second fluorescence channel images (601, 602) exceeds a threshold; The method of any one of claims 1 to 15, further comprising the step of: outputting (313) an excessive motion warning if the estimated motion is determined to exceed the threshold.

17. 1. An imaging system (100, 500) for imaging a scene including at least one target body part (290, 109) including a fluorescent material, the imaging system comprising: a light source (210, 203b) configured to emit excitation light for illuminating the target body part (290, 109), the excitation light being suitable for causing fluorescence emission from the target body part; a fluorescence camera (245, 236b) having a fluorescence emission detection passband configured to detect the fluorescence emitted from the target body part; An imaging system (100, 500) configured to perform the method (300) of any one of claims 1 to 16.

18. A computer program product, comprising: a computer-readable storage medium; The storage medium has stored thereon computer program product which, when executed by a computing device of an imaging system according to claim 17, causes the imaging system to perform the method according to any one of claims 1 to 16.

19. 1. A surgical method comprising: imaging at least one target body part (290, 109) of a patient using an imaging system according to claim 17 and performing a method according to any one of claims 1 to 16, and outputting said background subtracted fluorescence image for display; and operating on the target body part according to the display of the background subtracted fluorescence image.

20. A diagnostic method comprising: imaging at least one target body part (290, 109) of a patient using an imaging system according to claim 17 and performing a method according to any one of claims 1 to 16, and displaying said fluorescence image after background subtraction; and analyzing the target body part according to the display of the background subtracted fluorescent image.

21. 1. A method of treatment comprising: imaging at least one target body part (290, 109) of a patient using an imaging system according to claim 17 and performing a method according to any one of claims 1 to 16, and displaying said fluorescence image after background subtraction; treating the target body part according to the display of the background subtracted fluorescence image.

Images