Systems and methods for specular reflection detection and reduction

Abstract

The invention relates to systems and methods for specular reflection detection and reduction. A method and system of specular reflection detection and reduction includes a processing unit including one or more processors and an imaging unit coupled to the processing unit. The imaging unit includes one or more first illuminators for providing illumination of the target area, one or more first detectors for detecting reflections of the illumination, one or more second illuminators for triggering the one or more fluorescent materials to fluoresce in the target area, and one or more second detectors for detecting the fluorescence of the fluorescent material. The processing unit is configured to receive the first image from the first detector, determine one or more highly specular reflection regions in the first image, obscure the highly specular reflection regions in the first image, and generate a composite image based on the obscured first image and the detected fluorescence. The first image includes the detected reflections.

Description

[0001] This application is a divisional application of Chinese patent application 202010452939.7, filed on February 27, 2015, entitled “System and method for specular reflection detection and reduction”. Chinese patent application 202010452939.7 is a divisional application of Chinese patent application 201580021291.5 (PCT / US2015 / 017950), filed on February 27, 2015, entitled “System and method for specular reflection detection and reduction”. Technical Field

[0002] The present disclosure relates generally to imaging devices and imaging devices supporting fluorescence imaging, and more particularly, to specular reflection detection and reduction. Background Art

[0003] In fluorescence-based imaging, a fluorescent material presented in the field of view of the imaging device is excited with light of one wavelength and emits light of a different wavelength. This can provide a useful mechanism to highlight portions of the field of view that are of particular interest to the observer. For example, in the case of fluorescent image-guided surgery, a fluorescent dye such as indocyanine green (ICG) can be introduced into portions of a patient's anatomy to highlight those portions of the patient's anatomy. When ICG is excited with near-infrared light (approximately 750-800 nm), it emits fluorescence at a slightly higher wavelength (around 845 nm). By introducing ICG into selected portions of a patient's anatomy, with the aid of an appropriate viewer or camera, a surgeon or other medical personnel can receive visual cues and / or highlights that help them identify and / or avoid the selected anatomy. For example, ICG or other dyes can be used to highlight sentinel lymph nodes, areas of relative perfusion, tumor margins, and the like. Other useful fluorescent dyes suitable for highlighting a patient's anatomy include methylene blue, fluorescein, and fluorescein from LI-COR, Inc. of Lincoln, Nebraska. In some cases, fluorescence may be in the visible band and no additional viewer or camera may be required. While fluorescence-based imaging is being used, it is common practice to use visible light to generally illuminate other areas of the patient's anatomy about the target area or fluorescence so that the surgeon and / or other medical personnel can see and visualize other anatomical structures.

[0004] Unfortunately, fluorescence-based imaging is susceptible to many of the same problems as other imaging applications. Many portions of a patient's anatomy are highly reflective of light due to, for example, the presence of moisture on the surface of anatomical structures (e.g., tissue and / or organ structures). Due to this high reflectivity, visible light used to generally illuminate the patient's anatomy may generate one or more regions of high specular reflection. These regions of high specular reflection may produce bright spots in an image of the patient's anatomy and / or saturate a sensor used to detect the image. As a result, regions of high specular reflection may be mistaken for fluorescing regions and / or obscure or mask fluorescing regions, thereby limiting the usefulness of fluorescence in detecting selected anatomy.

[0005] Therefore, it would be desirable to provide improved methods and systems to detect, reduce and / or compensate for regions of high specular reflection so that more useful fluorescence images can be obtained. Summary of the invention

[0006] In accordance with some embodiments, an imaging system includes a processing unit including one or more processors; and an imaging unit coupled to the processing unit. The imaging unit includes one or more first illuminators for providing illumination of a target area, one or more first detectors for detecting reflections of the illumination, one or more second illuminators for triggering one or more fluorescent materials in the target area to fluoresce, and one or more second detectors for detecting the fluorescence of the fluorescent materials. The processing unit is configured to: receive a first image from the first detector, determine one or more high specular reflection areas in the first image, mask the high specular reflection areas in the first image, and generate a composite image based on the masked first image and the detected fluorescence. The first image includes the detected reflections.

[0007] According to some embodiments, a method of imaging includes: emitting general illumination onto a target area using an imaging unit, detecting a first reflection of the general illumination using the imaging unit, forming a first image based on the detected first reflection, determining one or more high specular reflection areas in the first image, masking the high specular reflection areas in the first image, emitting fluorescence triggering illumination onto the target area using the imaging unit, detecting fluorescence of one or more fluorescent materials in the target area using the imaging unit, and generating a composite image based on the masked first image and the detected fluorescence.

[0008] In accordance with some embodiments, an imaging system includes a processing unit including one or more processors; and an imaging unit coupled to the processing unit. The imaging unit includes one or more first illuminators for providing illumination of a target area and one or more first detectors for detecting reflections of the illumination. The processing unit is configured to: receive a first image from the first detector, receive a second image from the first detector, determine a first high specular reflection area in the first image and the second image, estimate a first surface normal in the first high specular reflection area based on a difference between the first image and the second image, and provide one or more move-away cues based on the estimated first surface normal. The first image includes the detected reflection and is based on a first imaging geometry. The second image includes the detected reflection and is based on a second imaging geometry different from the first imaging geometry.

[0009] In accordance with some embodiments, an imaging system includes a processing unit, an illumination system coupled to the processing unit, and a detector system coupled to the processing unit. The processing unit includes one or more processors. The illumination system emits illumination to illuminate a target area and trigger one or more fluorescent materials in the target area to fluoresce. The detector system detects reflections of the emitted illumination and detects the fluorescence of the one or more fluorescent materials in the target area. The processing unit is configured to: receive a first image including the detected reflections from the detector system, determine one or more high specular reflection areas in the first image, mask the high specular reflection areas in the first image, and generate a composite image based on the masked first image and the detected fluorescence.

[0010] According to some embodiments, a method includes accessing an occluded image of a target area, accessing a fluorescent image of the target area, and generating a composite image based on the occluded image and the fluorescent image. The occluded image includes a first reflected image having an occluded high specular reflective region. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a simplified diagram of a fluorescence imaging system according to some embodiments.

[0012] Figure 2 is a simplified diagram of a method of high specular reflection detection and reduction according to some embodiments.

[0013] Figure 3 is a simplified diagram of another fluorescence imaging system according to some embodiments.

[0014] Figure 4 is a simplified diagram of another method of high specular reflection detection and reduction according to some embodiments.

[0015] In the drawings, elements with the same reference numerals have the same or similar functions. DETAILED DESCRIPTION

[0016] In the following description, specific details are set forth to describe some embodiments that conform to the present disclosure. However, it is obvious to those skilled in the art that some embodiments may be implemented without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than limiting. Those skilled in the art may appreciate that other elements that are not specifically disclosed herein are also within the scope and spirit of the present disclosure. In addition, in order to avoid unnecessary repetition, one or more features shown and described with respect to one embodiment may be incorporated into other embodiments, unless otherwise specifically described or if one or more features will make the embodiment non-functional.

[0017] Figure 1 is a simplified diagram of a fluorescence imaging system 100 according to some embodiments. Figure 1 As shown in FIG. 1 , a target area 110, such as an anatomical portion of a patient, is shown illuminated using an imaging unit 120. The imaging unit 120 includes a number of imaging emitters and detectors to illuminate the target area 110 and collect one or more images of the target area 110 based on both reflected illumination and fluorescence in the target area. Figure 1 1, but the imaging unit 120 may be part of an endoscopic tool, part of an end effector of a computer-assisted surgical instrument, and / or part of a computer-assisted imaging device, wherein the computer-assisted surgical instrument corresponds to the da Surgical System. To support fluorescence imaging, the imaging unit 120 may include at least two types of imaging emitters and detectors.

[0018] The general illuminator 132 is used to shine light on the target area 110, as shown by the example ray 142, which reaches the target area 110 at the incident point 115. When the ray 142 reaches the target area 110, the ray 142 may be reflected by the target area 110 according to the diffusion, reflection, specular reflection and / or other imaging properties of the target area 110 at the incident point 115. The reflected light is modeled as the example ray 144, which is reflected back toward the imaging unit 120, where the ray 144 is detected by the general illumination detector 134. The incident point 115 may be of particular interest during imaging because the imaging unit 120, and more specifically the general illuminator 132 and the general illumination detector 134, are located approximately near or above the surface normal at the incident point 115. When the incident point 115 has a high specular reflection, such as due to moisture at the incident point 115, the general illumination detector 134 may detect a high level of light and / or may become saturated. As a result, the point of incidence 115 may appear as a bright spot on the images of the target area 110 acquired by the imaging unit 120. By monitoring these images and looking for areas of high intensity and / or saturation, the imaging unit 120 and / or a system such as the processing unit 150 may detect areas of possible high specular reflection within the target area 110.

[0019] The imaging unit 120 further includes a fluorescence imaging element. The fluorescence trigger illuminator 136 can emit light of one or more wavelengths, as shown by the example light ray 146, which is shown as reaching the target area 110 at the incident point 115. When the light ray 146 reaches the incident point 115, the fluorescent material (if present) absorbs the light and responds by fluorescing at another wavelength (as shown by representative light ray 148), which returns to the imaging unit 120 and is detected by the fluorescence detector 138 at the imaging unit 120. For example, when the fluorescent material is ICG, the incident light of the light ray 146 and the fluorescence of the light ray 148 can be in the near infrared spectrum, but at different wavelengths.

[0020] According to some embodiments, the general illuminator 132 and / or the fluorescence trigger illuminator 136 can be any kind of light and / or electromagnetic emitter that can emit light of an appropriate frequency / wavelength and / or frequency band. In some examples, each of the illuminators can include one or more light emitting diodes, laser diodes, halogen light sources, and / or the like. According to some embodiments, the general illumination detector 134 and / or the fluorescence detector 138 can be any kind of detector that can detect light or electromagnetic signals of an appropriate frequency / wavelength and / or frequency band. In some examples, each of the detectors can include one or more phototransistors, charge coupled devices, active and / or passive filters (e.g., liquid crystal tunable filters (LCTFs) and / or acousto-optic tunable filters (AOTFs)), and / or the like.

[0021] Fluorescence imaging system 100 may further include processing unit 150. Processing unit 150 may be part of a larger computing device and / or electronic system (not shown) that may receive inputs from user controls, process those inputs, implement one or more control algorithms based on those inputs, provide fluorescence images for display, etc. Processing unit 150 includes processor 160 coupled to memory 170. Operation of processing unit 150 is controlled by processor 160. And while processing unit 150 is shown with only one processor 160, it should be understood that processor 160 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. in processing unit 150. Processing unit 150 may be implemented as a separate subsystem and / or board added to a computing device, or as a virtual machine. In some embodiments, processing unit 150 may be included as part of an operator workstation and / or may be independent of the operator workstation but operate in coordination with the operator workstation.

[0022] Memory 170 may be used to store software executed by processing unit 150 and / or one or more data structures used during operation of processing unit 150. Memory 170 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media from which a processor or computer is adapted to read.

[0023] As shown, the memory 170 includes an imaging application 180 that can be used to support the generation and / or use of images of the target area 110, including fluorescence images in the target area 110. The imaging application 180 may include one or more application programming interfaces (APIs) for implementing imaging methods, which may include issuing one or more commands and / or requesting images from the imaging unit 120 and receiving images from the imaging unit 120. And although the imaging application 180 is described as a software application, the imaging application 180 may be implemented using hardware, software, and / or a combination of hardware and software.

[0024] To support imaging application 180, processing unit 150 additionally includes an interface 190 having one or more I / O ports. Interface 190 may be of any suitable type, such as dedicated I / O lines, a network connection (e.g., Ethernet), an internal integrated circuit (I / O), or a processor.2 C) bus, serial peripheral interface (SPI) bus, parallel port, analog I / O line, etc. Interface 190 may include one or more lines and / or buses for coupling processing unit 150 to imaging unit 120. In some examples, interface circuitry associated with interface 190 may include one or more bus controllers, I / O controllers, analog-to-digital converters, digital-to-analog converters, data acquisition systems, image acquisition systems, routers, switches, network switching equipment, and / or the like.

[0025] As stated above and further emphasized herein, Figure 1 This is merely an example, which should not unduly limit the scope of what is claimed. Those skilled in the art will recognize many variations, substitutions, and modifications.

[0026] Those skilled in the art will note that Figure 1 The geometric shapes of 100 are not necessarily drawn to scale, but are drawn to emphasize important features of fluorescence imaging system 100 and imaging unit 120. For example, the relative size and / or positioning of general illuminator 132, general illumination detector 134, fluorescence trigger illuminator 136, and fluorescence detector 138 may be exaggerated to illustrate these features at the expense of a more accurate depiction of the actual paths of light rays 142-148.

[0027] According to some embodiments, different arrangements between imaging unit 120 and processing unit 150 may be possible. In some examples, processing unit 150 may be part of imaging unit 120. In some examples, imaging unit 120 may be part of processing unit 150. In some examples, processing unit 150 may be coupled to more than one imaging unit 120. In some examples, imaging unit 120 may share imaging data with more than one processing unit.

[0028] According to some embodiments, different arrangements and configurations of the general illuminator 132, the general illumination detector 134, the fluorescence trigger illuminator 136, and the fluorescence detector 138 are possible. In some examples, more and / or fewer illuminators and / or detectors may be used. In some examples, the relative positioning between the illuminators and / or detectors may be changed. In some examples, the general illumination detector 134 and the fluorescence detector 138 may be combined into a single detector, which may be used for both specular and / or fluorescence detection. In some examples, the combined detector may use time division multiplexing and / or frequency division multiplexing to separate the general image from the fluorescence image. In some examples, the general illuminator 132 and the fluorescence trigger illuminator 136 may be combined into a single illuminator or emitter, which may be used to generate both general illumination and / or fluorescence trigger illumination. In some examples, the combined illuminator may use time division multiplexing and / or frequency division multiplexing with the aid of one or more active filters or passive filters to generate the desired illumination. In some examples, general illumination 132 and general illumination detector 134 may also be part of a general illumination and imaging system for target area 110 .

[0029] According to some embodiments, multiple combinations of general illumination devices and general illumination detectors may be used. In some examples, multiple general illumination devices and / or a single general illumination device may generate general light of multiple frequencies / wavelengths and / or frequency bands. Corresponding multiple general illumination detectors and / or a single general illumination detector may detect each of the multiple frequencies / wavelengths and / or frequency bands and generate an image of each of the multiple frequencies / wavelengths and / or frequency bands. In some examples, when high intensity and / or saturation is detected at the same location in each of the images, a high specular reflection area may be detected.

[0030] The presence of high specular reflection areas in an image may generally be undesirable. Areas of high intensity and / or saturation caused by high specular reflections may reduce the quality and / or usefulness of the image containing them. In some examples, high specular reflection areas may distract an observer of the image containing them from other important areas in the image. In some examples, high specular reflection areas may be mistaken for fluorescing areas in a target area and / or obscure and / or cover features, such as fluorescence, which may otherwise be visible but are not visible in the presence of high specular reflections.

[0031] Many techniques have been developed to detect, reduce and / or compensate for high specular reflection areas in images. One such method is to detect areas with high intensity that have a high probability of being associated with high specular reflection and shielding those parts of the image. In some examples, this can include comparing the pixel intensity within the image to a threshold, such as a set of saturation percentages, and replacing any pixels with an intensity above the threshold with a replacement color (e.g., black or a color that indicates masking). The method can also include identifying and / or highlighting the boundaries of the shielded area. The method can remove transfers caused by high specular reflection areas, but it may also inappropriately shield areas with high intensity that are not caused by high specular reflections, as well as remove other important features from the image when the shielded area does not typically include potential image details.

[0032] Another approach uses two sets of image sensors. When high intensity areas (such as those caused by high specular reflections) are detected in one of the images, the corresponding area from the other image is replaced. This approach reduces the likelihood of masking desired image details, but may require that the two sets of image sensors be sufficiently spaced so that high specular reflections may not occur in the same area in both images.

[0033] Other approaches may use creative shielding and / or baffles around the illuminator / emitter and image detector to reduce the likelihood that areas of high specular reflection may be captured by the image detector. This approach may generally require tight control over the relative positioning of the illuminator / emitter and detector, and may not generalize well to every possible geometry of the imaging unit and target area.

[0034] A common approach to utilizing fluorescence imaging is to reduce the ordinary illumination provided by the ordinary illuminator and use a combined visual detector for both ordinary illumination and fluorescence. By reducing ordinary illumination, ordinary illumination and / or high specular reflection areas that can be generated are less likely to obscure image information obtained due to fluorescence-triggered illumination of the fluorescing material. However, this approach tends to generate images with low levels of ambient light, which can make it difficult to observe features of interest in the target area.

[0035] Therefore, most methods for detecting, reducing and / or compensating for high specular reflection areas do not provide a good solution when the desired image shows both visible detail and fluorescent detail. This is because important visible detail is sacrificed, fluorescent detail may be obscured and / or the solution may include a device having a geometry that is not practical for use with endoscopic equipment and / or the like. An improved solution may be obtained by using an imaging system (such as fluorescence imaging system 100) that allows for the collection of a normal illumination image separately from the fluorescent image. In this way, detecting, reducing and / or compensating for high specular reflection areas may be significantly less likely to obscure and / or cover important fluorescent information.

[0036] Figure 2 2 is a simplified diagram of a method 200 for high specular reflection detection and reduction according to some embodiments. One or more processes 210-280 of the method 200 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which when executed by one or more processors may cause one or more processors (e.g., processor 160) to perform one or more processes 210-280. In some embodiments, high specular reflection detection and reduction may be performed by an imaging application (e.g., imaging application 180).

[0037] At process 210, general illumination is emitted. Using one or more general illuminators or emitters (e.g., general illuminator 132), general illumination is illuminated on the target area. In some examples, the general illumination may include visible light in one or more bands of the visible spectrum. In some examples, the general illumination may be turned on using appropriate drivers and / or modulation circuits. In some examples, the general illumination may each include one or more light emitting diodes, laser diodes, halogen light sources, and / or the like. In some examples, when the general illumination includes signals from multiple bands, the multiple bands may be emitted simultaneously and / or via time division multiplexing.

[0038] In process 220, reflections of general illumination are detected. Reflections of general illumination emitted during process 210 are detected using one or more general illumination detectors (e.g., general illumination detector 134). In some examples, general illumination may be reflected from one or more structures, objects, portions of patient anatomy, and / or the like. In some examples, one or more general illumination detectors may detect general illumination in each frequency band emitted by a general illuminator and / or a subset thereof. In some examples, each of the general illumination detectors may include one or more phototransistors, charge coupled devices, active and / or passive filters (e.g., LCTF and / or AOTF), and / or the like. In some examples, one or more general illumination detectors may be time multiplexed to be synchronized with one or more general illuminators. In some examples, the detected general illumination may be detected as a single image and / or multiple images of a target area, wherein each of the multiple images corresponds to a frequency band emitted and / or detected for the corresponding image. In some examples, the single image and / or multiple images may be preprocessed. In some examples, preprocessing may include histogram equalization and / or contrast adjustment.

[0039] In process 230, one or more high specular reflection areas are detected. Each of the images detected during process 220 can be analyzed to determine one or more potential high specular reflection areas. In some examples, potential one or more high specular reflection areas can be determined by identifying one or more areas in the image with high intensity and / or saturation. In some examples, high intensity can be determined for any pixel with an intensity above a threshold. In some examples, the threshold can be set by an operator of the imaging system and / or set in response to other properties of the imaging system. In some examples, the threshold can be set to an absolute intensity threshold or to a percentage of the maximum possible intensity value. In some examples, when a single image is detected during process 220, the potential high specular reflection area can be one or more high specular reflection areas. In some examples, when multiple images are detected during process 220, high specular reflection can be detected in an area where each of the multiple images includes an area among the potential one or more high specular reflection areas. In some examples, because high specular reflections are generally independent of wavelength, using multiple images can reduce the likelihood that a region may be misclassified as a high specular reflection region by testing for high specular reflections at multiple frequencies or in multiple frequency bands.

[0040] At process 240, fluorescence trigger illumination is emitted. Using one or more fluorescence trigger illuminators or emitters (e.g., fluorescence trigger illuminator 136), the fluorescence trigger illumination is illuminated on the target area. In some examples, the fluorescence trigger illumination may include frequencies in the near infrared band. In some examples, the fluorescence trigger illuminator may be turned on using an appropriate driver and / or modulation circuit. In some examples, the fluorescence trigger illuminator may each include one or more light emitting diodes, laser diodes, halogen light sources, and / or the like. In some examples, when the fluorescence trigger illuminator may be the same as a conventional illuminator, but wherein the modulation and / or drive circuit is adapted to generate the fluorescence trigger illumination instead of conventional illumination. In some examples, the drive circuit may include one or more active or passive filters, such as LCTF and / or AOTF.

[0041] In process 250, one or more fluorescent areas are detected. One or more fluorescence detectors (e.g., fluorescence detector 138) are used to detect the fluorescence of the fluorescent material in the target area. In some examples, high specular reflections from fluorescent illumination may not be detected during process 250 because the fluorescence detector is generally insensitive to the frequency emitted by the fluorescence trigger illuminator. In some examples, one or more fluorescence detectors may be the same detector as one or more general illumination detectors. In some examples, each of the fluorescence detectors may include one or more phototransistors, charge coupled devices, active and / or passive filters (e.g., LCTF and / or AOTF) and / or the like. In some examples, the detected fluorescence may be detected as a fluorescence image. In some examples, the fluorescence image may also correspond to a general illumination image.

[0042] In process 260, one or more high specular reflection areas are masked. In some examples, one or more high specular reflection areas detected during process 230 can be masked by one or more normal illumination images acquired of the target area. In some examples, the masking can include replacing pixels in the one or more areas with pixels of a background or a default color (such as black). In some examples, when the normal illumination image also includes fluorescence information, the fluorescence information in the high specular reflection area can also be masked. In some embodiments, when the normal illumination image is separated from the fluorescence image detected during process 250, an image combination algorithm (such as alpha (α) blending, etc.) can be used to combine the fluorescence image information with the normal illumination image. These one or more composite images can then be used for the remaining processes of method 200.

[0043] In process 270, visual cues may be added to the high specular reflection areas. Because the masking of process 260 may conceal the presence of one or more high specular reflection areas, one or more visual cues may be added to the masked one or more normally illuminated images and / or one or more synthetic images to notify the operator of the possibility of poor quality and / or incomplete image information in one or more high specular reflection areas. In some examples, the visual cues may include adding an outline around one or more high specular reflection areas, adding a false color, adding a pattern, adding a flash, adding an arrow and / or other markings, and / or the like.

[0044] In process 280, one or more geometry adjustment prompts are provided. Because masking one or more high specular reflection areas during process 260 can remove and / or degrade information in one or more normal illumination images and / or synthetic images, geometry adjustment prompts can be provided to the operator of the imaging unit and / or the motion planning system. By adjusting the geometry of the imaging unit, it is possible to reduce the size of one or more high specular reflection areas in one or more normal illumination images and / or synthetic images and / or remove them because the orientation of the imaging relative to the surface normal in the target area can be changed. In some examples, the geometry adjustment prompts can include audio prompts in one or more normal illumination images and / or synthetic images, visual cues (such as arrows), feedback applied to operator input controls, feedback applied to motion planning algorithms, and / or the like.

[0045] Method 200 may further include repeating processes 210 - 280 to remove high specular reflection areas in multiple images and / or generate a video image of the target area.

[0046] As stated above and further emphasized herein, Figure 2 It is only an example, which should not overly limit the scope of the claimed protection. Those skilled in the art will recognize many variations, substitutions and modifications. In some embodiments, method 200 can be modified to utilize the ability to move the imaging unit relative to the target area using a motion planning algorithm. In some examples, an operator or other system using an imaging application can specify a volume in which the imaging unit can move without interfering with other devices, objects and / or patient anatomy in or near the target area. The imaging application can then move the imaging unit around the volume to collect a series of general illumination and / or fluorescence images. The high specular reflection area in some images can be replaced with images from the same area of ​​other images that have reduced specular reflection or do not have specular reflection. In some examples, one or more image stitching algorithms can be used to create a composite image with reduced and / or eliminated specular reflection. In some examples, one or more dynamic models of the imaging unit and the device for moving the imaging unit through the volume can be used to cross-correlate areas from different images. In some examples, when the target region includes moving objects and / or moving patient anatomy (e.g., a beating heart), image collection can be timed so that the series of images correspond approximately to the same position, phase, and / or orientation of the objects and / or patient anatomy in the target region.

[0047] Figure 3 is a simplified diagram of another fluorescence imaging system 300 according to some embodiments. Figure 3As shown in FIG. 3 , a target area 310, such as an anatomical portion of a patient, is shown illuminated using an imaging unit 320. The imaging unit 320 includes a number of imaging emitters and detectors to illuminate the target area 310 and collect one or more images of the target area 310 based on both the illumination and fluorescence reflected in the target area. Figure 3 Not shown, but imaging unit 320 may be part of an endoscopic tool, part of an end effector of a computer-assisted surgical instrument, and / or part of a computer-assisted imaging device. To support fluorescence imaging, imaging unit 320 may include several groups and types of imaging emitters and detectors.

[0048] The imaging unit 320 includes at least two general illuminators 332 and 334 separated by a known distance. In some examples, each of the general illuminators 332 and 334 can be similar to the general illuminator 132. Each of the general illuminators 332 and 334 is configured to shine general illumination on the target area 310, as shown by the corresponding exemplary light arrows from the general illuminators 332 and 334 to the surface in the target area. Based on the geometry of the surface as approximated by the surface normal 315 near the corresponding point where the general illumination is incident on the surface of the target area 310, the general illumination is reflected from the surface in the target area. The reflected general illumination returns to the imaging unit 320, where it can be captured in a separate stereo image by the general illumination detector 336. In some examples, the general illumination detector 336 can be similar to the general illumination detector 134. In some examples, the general illuminators 332 and 334 can alternatively provide illumination on the target area 310, so that a separate stereo image can be captured using the illumination from one of the general illuminators 332 and 334 at a time. Using separate stereo images and the known geometry of general illuminators 332 and 334 and general illumination detector 336, imaging unit 320 and / or a processing unit (e.g., processing unit 150) coupled to imaging unit 320 can determine the approximate position and / or orientation of surface normal 315 and other surface normals that may be located at other locations in target area 310. In some examples, separate stereo images can also be used to detect one or more high specular reflective regions in target area 310.

[0049] Similar to imaging unit 120, imaging unit 320 may also include fluorescence trigger illuminators 342 and 344 and fluorescence detector 346. In some examples, each of fluorescence trigger illuminators 342 and 344 may be similar to fluorescence trigger illuminator 136, and fluorescence detector 346 may be similar to fluorescence detector 138. One or both of fluorescence trigger illuminators 342 and 344 may be used to trigger fluorescence of fluorescent materials in target area 310. Fluorescence detector 346 may detect fluorescence triggered by fluorescence trigger illuminators 342 and 344.

[0050] By being close to a surface normal (e.g., surface normal 315), imaging unit 320 is better able to provide geometry adjustment cues as described with respect to process 280. The location and orientation of the surface normal can be used to provide better move-away cues to imaging unit 320, because the orientation with the highest specular reflections typically occurs when both the normal illuminator and the normal illumination detector are located above and oriented toward the surface normal. Thus, avoiding orientations where imaging unit 320 is directly or approximately aligned with surface normals (e.g., surface normal 315) can reduce and / or eliminate high specular reflections caused by surfaces at those surface normals.

[0051] As stated above and further emphasized herein, Figure 3 These are merely examples, which should not unduly limit the scope of what is claimed. Those skilled in the art will recognize many variations, substitutions, and modifications.

[0052] Those skilled in the art will note that Figure 3 The geometric shapes of the drawings are not necessarily drawn to scale, but have been drawn to emphasize important features of the fluorescence imaging system 300 and the imaging unit 320. For example, the relative size and / or positioning of the illuminator and the detector may be exaggerated to show the fluorescence imaging system 300 and the imaging unit 320. Figure 3 These features are shown at the expense of a more precise description of the actual paths of the corresponding light rays in FIG.

[0053] According to some embodiments, different arrangements between imaging unit 320 and processing unit 150 may be possible. In some examples, processing unit 150 may be part of imaging unit 320. In some examples, imaging unit 320 may be part of processing unit 150. In some examples, processing unit 150 may be coupled to more than one imaging unit 320. In some examples, imaging unit 320 may share imaging data with more than one processing unit.

[0054] According to some embodiments, different arrangements and configurations of illuminators and detectors can also provide desired stereoscopic images, detection of high specular reflection areas, and detection of fluorescent images. In some examples, more and / or fewer illuminators and / or detectors are possible. In some examples, two ordinary illuminators 332, 334 and ordinary lighting detector 336 can be replaced with two ordinary lighting detectors and one ordinary illuminator with known spacing. In some examples, one or other fluorescence trigger illuminator 342 and / or 344 can be omitted. In some examples, the remaining fluorescence trigger illuminator can be moved to a position closer to the fluorescence detector 346. In some examples, time division multiplexing and / or frequency division multiplexing can be used to combine ordinary illuminators and fluorescence trigger illuminators (e.g., ordinary illuminator 332 and fluorescence trigger illuminator 342). In some examples, time division multiplexing and / or frequency division multiplexing can be used to combine ordinary lighting detector 336 and fluorescence detector 346.

[0055] Figure 4 4 is a simplified diagram of another method 400 for high specular reflection detection and reduction according to some embodiments. One or more processes 410-450 of method 400 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which when executed by one or more processors may cause one or more processors (e.g., processor 160) to perform one or more processes 410-450. In some embodiments, high specular reflection detection and reduction may be performed by an imaging application (e.g., imaging application 180). In some embodiments, method 400 may be performed in conjunction with method 200 to provide both masking of one or more high specular reflection areas and providing a move-away prompt to reduce the amount of high specular reflection captured by the imaging system.

[0056] In process 410, general illumination is emitted. General illumination and / or general illumination is illuminated on the target area using one or more general illuminators or emitters (e.g., general illuminators 332 and / or 334). In some examples, general illumination can include visible light in one or more bands of the visible spectrum. In some examples, the general illuminators can be turned on using appropriate drivers and / or modulation circuits. In some examples, the general illuminators can each include one or more light emitting diodes, laser diodes, halogen light sources, and / or the like. In some examples, when the general illumination includes signals from multiple bands, the multiple bands can be emitted simultaneously and / or via time division multiplexing.

[0057] In process 420, stereo reflections are detected. Two or more stereo images of reflected ordinary illumination are obtained using one or more ordinary illumination detectors (e.g., ordinary illumination detector 336) together with one or more ordinary illuminators. In some examples, ordinary illumination can be reflected from one or more structures, objects, patient anatomical parts, and / or similar structures. In some examples, one or more ordinary illumination detectors can detect ordinary illumination in each of the frequency bands emitted by one or more ordinary illuminators and / or their subgroups. In some examples, each of the ordinary illumination detectors can include one or more phototransistors, charge coupled devices, active and / or passive filters (e.g., LCTF and / or AOTF), and / or the like. In some examples, one or more ordinary illumination detectors can be time multiplexed to synchronize with one or more ordinary illuminators.

[0058] In process 430, surface normals are estimated. The stereo images detected during process 420 can be used to estimate surface normals for one or more surfaces visible in the stereo images. In some examples, known differences in imaging geometry between the two stereo images can be used to estimate surface normals based on the differences in the stereo images. In some examples, stereo camera calibration data can be used to estimate surface normals. In some examples, surface normals can be estimated for one or more surfaces at multiple locations on those surfaces. In some examples, a process similar to process 230 can be used to detect high or potentially high specular reflection areas, wherein surface normals are estimated within those detected areas.

[0059] In process 440, the estimated surface normal is used to provide a move-away prompt. One or more estimated surface normals point to a position above one or more surfaces that the imaging unit in the target area should avoid approaching to reduce and / or eliminate the amount of high specular reflections in the detected and / or captured images. In order to prevent the imaging unit from being positioned above one or more surface normals and / or to encourage the imaging unit to move without aligning with one or more surface normals, a move-away prompt can be provided to the operator and / or motion planning system of the imaging unit. In some examples, a move-away prompt can be provided based on one or more dynamic models of the device arm on which the imaging unit is installed.

[0060] In some embodiments, one or more removal prompts may include adding a visual prompt (such as an arrow) to the image displayed to the operator, which suggests a movement direction and / or orientation change of the imaging unit. In some examples, the movement direction may be away from the position pointed to by one or more surface normals. In some examples, when the imaging unit is located near one of the positions pointed to by one or more surface normals, the movement prompt may include a set of restricted directions away from the surface normal. In some examples, when the imaging unit is aligned with the surface normal, the movement prompt may suggest any direction away from the surface normal. In some examples, when the surface normal can be avoided by a position and / or orientation change, the movement prompt may suggest that the imaging unit rotate appropriately to move away from the surface normal. In some examples, the rotation can be determined based on one or more dynamic models of the device arm on which the imaging unit is mounted.

[0061] In some embodiments, one or more move away prompts may include audio prompts. In some examples, the audio prompts may include directions such as left, right, up, down, away, turn, etc. In some examples, when multi-directional sounds are available, the move away prompt may provide a sound that can prompt the operator to move toward or away from the sound. In some examples, the sound intensity may increase or decrease when the audio prompt is successfully followed.

[0062] In some embodiments, one or more removal cues may include adding one or more removal points and / or lines to the motion planning geometry. In some examples, one or more removal points and / or lines may be implemented using one or more points or regions with a repulsing field and / or a repulsing charge. In some examples, one or more removal points and / or lines may be implemented using one or more virtual collision objects and / or virtual feedback mechanisms. In some examples, one or more removal points may be pointed to by a corresponding surface normal and located at a height above the corresponding surface normal based on the working distance of the imaging unit. In some examples, one or more removal points may be pointed to by a corresponding surface normal and located near the current position of the imaging unit. In some examples, one or more removal lines may be colinear with the surface normal and extend from the surface through at least the height of the imaging unit.

[0063] In process 450, feedback is provided. In some embodiments, one or more removal points and / or lines can be used to change the motion plan of the imaging unit. In some examples, one or more removal points and / or lines can create a repulsive force (repulsing force) and / or torque, which can cause feedback force and / or torque on the joint of the imaging unit. In some examples, the feedback force or torque caused can modify the force or torque of the corresponding joint of the imaging unit determined from other motion plans and / or operator command motions. In some examples, the Jacobian transpose or similar methods can be used to map the repulsive force and / or torque to the feedback force or torque caused. In some examples, the feedback force and / or torque can also be fed back to the operator through one or more tactile control devices.

[0064] Method 400 may further include repeating processes 410 - 450 to periodically estimate surface normals and provide removal cues and feedback to an operator and / or motion planning system.

[0065] As stated above and further emphasized herein, Figure 4 It is only an example, which should not overly limit the scope of the claimed protection. Those skilled in the art will recognize many variants, substitutions and modifications. In some embodiments, method 400 can be modified to utilize the ability of using a motion planning algorithm to move the imaging unit relative to the target area. In some examples, an operator or other system using an imaging application can specify a volume in which the imaging unit can move without interfering with other devices, objects and / or patient anatomy in or near the target area. The imaging application can then move around the volume to collect a series of stereo images. In some examples, the collected stereo images can be analyzed to determine one or more postures of the imaging unit, which minimizes the positioning and / or alignment of each of the surface normals of the imaging unit and the target area. In some examples, the collected stereo images can be analyzed to predict the preferred posture of the imaging unit to avoid the high specular reflection of the selected area in the target area and / or determine a set of preferred postures used by method 200 so that the composite image is spliced ​​together with the reduced and / or eliminated specular reflection. In some examples, when the target region includes moving objects and / or moving patient anatomy (e.g., a beating heart), image collection can be timed so that the series of stereo images correspond approximately to the same position, phase, and / or orientation of the objects and / or patient anatomy in the target region.

[0066] Some examples of processing units (e.g., processing unit 150) may include non-transitory, tangible machine-readable media that include executable code that, when executed by one or more processors (e.g., processor 160), causes the one or more processors to perform the processes of methods 200 and / or 400. Some common forms of machine-readable media that may include the processes of methods 200 and / or 400 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media from which a processor or computer is suitable for reading.

[0067] Although exemplary embodiments have been shown and described, a wide range of modifications, changes and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be applied without the corresponding use of other features. Those skilled in the art will recognize many variations, substitutions and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and it will be understood that the claims are to be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. An imaging system, comprising: a processing unit comprising one or more processors; as well as an imaging unit, which is coupled to the processing unit and comprises: one or more first illuminators for providing illumination of a target area; and one or more first detectors for detecting reflections of said illumination; The processing unit is configured as follows: receiving a first image from the first detector, the first image comprising the reflection of the illumination and based on a first imaging geometry; receiving a second image from the first detector, the second image comprising the reflection of the illumination and based on a second imaging geometry different from the first imaging geometry; determining a first high specular reflection region in the first image and the second image; estimating a first surface normal in the first high specular reflection region based on a difference between the first image and the second image; and One or more removal cues are provided based on the first surface normal. 2 . The imaging system of claim 1 , wherein providing the one or more removal cues comprises providing a removal point to a motion planning system, the removal point being pointed to by the first surface normal. 3 . The imaging system of claim 2 , wherein the removal point is located at a height above a normal to the first surface based on a working distance of the imaging unit. The imaging system according to claim 2 , wherein the removal point is located near the imaging unit.

5. The imaging system of claim 2, wherein providing the one or more removal cues comprises providing a removal line to a motion planning system, the removal line being co-linear with the first surface normal.

6. The imaging system according to claim 1, wherein the processing unit is further configured to: moving the imaging unit from a first posture to a second posture, the movement changing the first imaging geometry to a third imaging geometry and changing the second imaging geometry to a fourth imaging geometry; receiving a third image from the first detector, the third image comprising the reflection of the illumination and based on the third imaging geometry; receiving a fourth image from the first detector, the fourth image comprising the reflection of the illumination and based on the fourth imaging geometry; determining a second high specular reflection area in the third image and the fourth image; estimating a second surface normal in the second high specular reflection area based on a difference between the third image and the fourth image; as well as Based on the first surface normal and the second surface normal, the first high specular reflection area and the second high specular reflection area, and the first pose and the second pose, the first image and the second image are selected for use or the third image and the fourth image are selected for use.

7. The imaging system of claim 1, wherein: The imaging unit further comprises: one or more second illuminators for triggering one or more fluorescent materials to fluoresce in the target area; and one or more second detectors for detecting fluorescence of the fluorescent material; and the processing unit is further configured to: masking the first high specular reflection area in the first image to produce a masked first image; generating a fluorescent image of the target area; and A composite image is generated based on the masked first image and the fluorescent image.

8. An imaging system comprising a processing unit, wherein the processing unit is configured to: receiving a first image comprising a target area illuminated by a first illumination; determining a first high specular reflection region in the first image based on a reflection of the illumination on the target region; as well as The fluorescent image of the target area is refined using the first high specular reflection area.

9. The imaging system of claim 8, wherein the processing unit is configured to mask the first image by substituting a default color in the first high specular reflection area to produce a masked first image, thereby refining the fluorescent image, and generating a composite image based on the masked first image and the fluorescent image of the target area.

10. The imaging system of claim 8, wherein the first illumination comprises a visible wavelength range.

11. The imaging system of claim 8, wherein the processing unit is further configured to generate geometric guidance to reduce the first high specular reflection area.

Images