Medical imaging system and method

Abstract

To solve a problem with a conventional technique.SOLUTION: A 2×2 pixel array in an imaging sensor includes a first pixel, a second pixel, a third pixel, and a fourth pixel. A red filter covers the first pixel, a first blue filter covers the second pixel, a second blue filter covers the third pixel, and a green filter covers the fourth pixel. The red filter is configured to enable the first pixel to collect a red component of visible light, and prevent the first pixel from collecting a blue component and a green component of the visible light. The first and second blue filters are configured to enable the second and third pixels to collect a blue component respectively, and prevent the second and third pixels from collecting a red component and a green component. The green filter is configured to enable the fourth pixel to collect a green component, and to prevent the fourth pixel from collecting a red component and a blue component.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Patent Application No. 62 / 774,041, entitled “MEDICAL IMAGING SYSTEMS AND METHODS,” filed November 30, 2018, which is incorporated by reference in its entirety. [Background technology]

[0002] Conventional image sensors included in imaging devices typically include an array of pixels (also called photosites or photosensors), where each pixel detects light reflecting off surfaces in a scene. The detected light may then be converted into data representing an image of the scene.

[0003] To facilitate the generation of color images, the image sensor may include an arrangement of color filters covering the pixels. Each color filter is configured to enable its corresponding pixel to detect only a particular color component of light incident on the pixel. For example, a conventional color filter arrangement (e.g., a Bayer filter arrangement) is 50 percent green, 25 percent red, and 25 percent blue. In other words, for every 2×2 pixel array included in the image sensor, two of the pixels are covered by green filters configured to enable the two pixels to detect only the green component of visible light, one of the pixels is covered by a red filter configured to enable the pixel to detect only the red component of visible light, and one of the pixels is covered by a blue filter configured to enable the pixel to detect only the blue component of visible light. Such color filter arrangements are commonly referred to as RGGB, BGGR, RGBG, or GRGB. Unfortunately, such color filter arrangements are not well suited for medical applications in which images of internal anatomical structures are acquired using an imaging device (e.g., an endoscope) configured to be positioned within a patient. Summary of the Invention

[0004] The accompanying drawings illustrate various embodiments and are a part of this specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. [Brief description of the drawings]

[0005] [Figure 1] 1 illustrates an exemplary medical imaging system in accordance with principles described herein.

[0006] [Diagram 2] 1 illustrates an exemplary image sensor in accordance with principles described herein.

[0007] [Figure 3A] 1 illustrates an exemplary color filter configuration in accordance with principles described herein. [Figure 3B] 1 illustrates an exemplary color filter configuration in accordance with principles described herein.

[0008] [Figure 4] 1 illustrates an exemplary medical imaging system in accordance with principles described herein.

[0009] [Diagram 5] 2 shows spectral response curves for color filters in accordance with the principles described herein.

[0010] [Figure 6A] 1 illustrates a filter configuration in accordance with principles described herein. [Figure 6B] 1 illustrates a filter configuration in accordance with principles described herein.

[0011] [Figure 7] 2 shows spectral response curves for color filters in accordance with the principles described herein.

[0012] [Figure 8A] 1 illustrates an exemplary filter configuration in accordance with principles described herein. [Figure 8B] 1 illustrates an exemplary filter configuration in accordance with principles described herein. [Figure 9A] 1 illustrates an exemplary filter configuration in accordance with principles described herein. [Figure 9B] 1 illustrates an exemplary filter configuration in accordance with principles described herein. [Figure 10A] 1 illustrates an exemplary filter configuration in accordance with principles described herein. [Figure 10B] 1 illustrates an exemplary filter configuration in accordance with principles described herein.

[0013] [Figure 11] 1 illustrates an image processing system in accordance with principles described herein.

[0014] [Figure 12] 1 illustrates various operations that may be performed by an imaging processing system to selectively generate full-color images and full-color fluorescent images in accordance with the principles described herein.

[0015] [Figure 13] 1 illustrates an exemplary method in accordance with principles described herein.

[0016] [Figure 14] 1 illustrates an exemplary computer-assisted surgical system in accordance with principles described herein.

[0017] [Figure 15] 1 illustrates an exemplary computing device consistent with principles described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] A medical imaging system and method are described herein. As described in more detail below, an exemplary medical imaging system includes an image sensor including a 2×2 array of pixels. The 2×2 array includes a first pixel, a second pixel, a third pixel, and a fourth pixel. A red filter covers the first pixel, a first blue filter covers the second pixel, a second blue filter covers the third pixel, and a green filter covers the fourth pixel. The red filter is configured to enable the first pixel to collect a red component of visible light and prevent the first pixel from collecting blue and green components of visible light. The first and second blue filters are configured to enable the second and third pixels to collect a blue component of visible light, respectively, and prevent the second and third pixels from collecting red and green components of visible light, respectively. The green filter is configured to enable the fourth pixel to collect a green component of visible light and prevent the fourth pixel from collecting red and blue components of visible light.

[0019] Various advantages and benefits are associated with the medical imaging systems and methods described herein. For example, in medical imaging applications, it is often desirable to use blue-biased illumination (i.e., light having more of a blue component than a green or red component). This is because the wavelengths of the green and red components are longer than the blue components and therefore penetrate deeper into the patient anatomy. This in turn causes the green and red components to scatter more than the blue components, resulting in relatively more blurred red and green components in the image captured by the imaging device. Thus, in medical imaging applications, blue-biased illumination results in a clearer image than illumination biased to other colors (e.g., green). By configuring 50% of the pixels in a 2×2 pixel array to capture the blue component of light, the systems and methods described herein may more effectively capture blue-biased light and therefore may produce clearer and more accurate images of the patient anatomy.

[0020] Another exemplary medical imaging system described herein includes an image sensor including a 2×2 array of pixels including a first pixel, a second pixel, a third pixel, and a fourth pixel. A first color filter is configured to cover the first pixel and enable the first pixel to collect a first color component of visible light and prevent the first pixel from collecting a second and third component of visible light. A second color filter is configured to cover the second and third pixels. The second color filter is configured to enable the second and third pixels to collect a second color component of visible light, respectively, and prevent the second and third pixels from collecting the first and third color components of visible light, respectively. A third color filter is configured to cover a fourth pixel and enable the fourth pixel to collect a third color component of visible light and prevent the fourth pixel from collecting the first and second color components of visible light. A pixel-level broadband infrared cut filter (blocking filter) is configured to cover the second pixel and prevent the second pixel from collecting infrared light having a wavelength included in the first wavelength range. In some examples, a narrowband infrared cut filter is configured to cover the first, third, and fourth pixels and prevent the first, third, and fourth pixels from collecting infrared light having a wavelength included in the second wavelength range. The second wavelength range is included in the first wavelength range and is narrower than the first wavelength range. As described below, this configuration advantageously provides three color component channels and a fluorescent illumination channel that may enable the systems and methods described herein to selectively display full-color fluorescent images. By displaying full-color fluorescent images as opposed to conventional grayscale fluorescent images, the systems and methods described herein may enable medical personnel (e.g., surgeons) to more accurately visualize and evaluate blood vessels, bile ducts, tissue perfusion, and other anatomical structures.

[0021] FIG. 1 illustrates an exemplary medical imaging system 100 configured to capture an image of a scene. In some examples, the scene may include a surgical field associated with a patient. In certain examples, the surgical field may be located entirely within the patient, and the surgical field may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissues internal to the patient, the surgical field may include the tissue, anatomical structures underlying the tissue, as well as the space around the tissue where, for example, surgical instruments used to perform the surgical procedure are located. In other examples, the surgical field may be located at least partially external to the patient.

[0022] As shown, the medical imaging system 100 includes an imaging device 102 and a controller 104. The medical imaging system 100 may include additional or alternative components that may be useful in a particular implementation. For example, the medical imaging system 100 may include various optical and / or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, etc.), cables configured to house electrical wires and / or optical fibers and interconnect the imaging device 102 and the controller 104, etc.

[0023] The imaging device 102 may be implemented by an endoscope or other camera device configured to capture an image of a scene. As shown, the imaging device 102 includes a camera head 106, a shaft 108 coupled to and extending away from the camera head 106, image sensors 110 (i.e., right image sensor 110-R and left image sensor 110-L) at a distal end of the shaft 108, and an illumination channel 112. In the example of FIG. 1, the imaging device 102 is stereoscopic. Alternatively, in other examples, the imaging device 102 may be monoscopic (e.g., by including one image sensor 110 instead of two image sensors 110).

[0024] The imaging device 102 may be handled and controlled manually (e.g., by a surgeon performing a surgical procedure on a patient). Alternatively, the camera head 106 may be coupled to a manipulator arm of a computer-assisted surgical system, an example of which is provided below. In this configuration, the imaging device 102 may be controlled using robotic and / or teleoperation techniques.

[0025] The distal end of shaft 108 may be positioned at or near a scene to be imaged by imaging device 108. For example, the distal end of shaft 108 may be inserted into a patient. In this configuration, imaging device 102 may be used to capture images of anatomical structures and / or other objects within the patient.

[0026] The image sensors 110 may be implemented by any suitable image sensor, such as a charge-coupled device ("CCD") image sensor, a complementary metal-oxide semiconductor ("CMOS") image sensor, or the like. In some examples, as shown in FIG. 1, the image sensor 110 is positioned at a distal end of the shaft 108. Alternatively, the image sensor 110 may be positioned closer to a proximal end of the shaft 108, inside the camera head 106, or outside the external imaging device 102 (e.g., inside the controller 104). In these alternative configurations, optics (e.g., lenses, optical fibers, etc.) included in the shaft 108 and / or camera head 106 may convey light from the scene to the image sensor 110.

[0027] Image sensor 110 is configured to detect (e.g., capture, collect, sense, or otherwise obtain) light. For example, image sensor 110-R is configured to detect light from a right field of view, and image sensor 110-L is configured to detect light from a left field of view. In the example of FIG. 1 , the light detected by image sensor 110 includes visible light reflected from objects disposed within the scene. However, as described herein, image sensor 110 may additionally or alternatively detect infrared fluorescent illumination emitted by a fluorescent imaging agent disposed within the scene. As illustrated below, image sensor 110 may convert the detected light into data representative of one or more images. Exemplary implementations of image sensor 110 are described herein.

[0028] Illumination channel 112 may be implemented by one or more optical components (e.g., optical fibers, light guides, lenses, etc.) As described below, illumination may be provided via illumination channel 112 to illuminate a scene.

[0029] The controller 104 may be implemented by any suitable combination of hardware and software configured to control and / or interface with the imaging device 102. For example, the controller 104 may be implemented at least in part by a computing device included in a computer-assisted surgical system.

[0030] The controller 104 includes a camera control unit 114 (CCU) and a visible light illumination source 116. The controller 104 may include additional or alternative components as may be useful in a particular implementation. For example, the controller 104 may include circuitry configured to provide power to components included in the image capture device 102. In some examples, the CCU 114 and / or the visible light illumination source 116 are alternatively included in the image capture device 102 (e.g., the camera head 106).

[0031] The CCU 114 is configured to control various parameters of the image sensor 110 (e.g., activation time, auto-exposure, etc.). As described below, the CCU 114 may be further configured to receive and process image data from the image sensor 110. Although the CCU 114 is shown in FIG. 1 as being a single unit, the CCU 114 may alternatively be implemented with a first CCU configured to control the right image sensor 110-R and a second CCU configured to control the left image sensor 110-L.

[0032] Visible light illumination source 116 may be configured to generate and emit visible light 118. Visible light 118 may travel via illumination channel 112 to the distal end of shaft 108, where it exits and illuminates the scene.

[0033] Visible light 118 may include one or more color components. For example, visible light 118 may include white light that includes a full spectrum of color components (e.g., a red component, a green component, and a blue component). The red component has a wavelength between about 635 and 700 nanometers ("nm"). The green component has a wavelength between about 520 and 560 nm. The blue component has a wavelength between about 450 and 490 nm.

[0034] In some examples, the visible light 118 is biased to include more of one color component than another color component. For example, the visible light 118 may be blue biased by including more blue components than red and green components.

[0035] To capture one or more images of a scene, the controller 104 (or any other suitable computing device) may activate (activate) the visible light illumination source 116 and the image sensor 110. While activated, the visible light illumination source 116 emits visible light 118, which travels to the scene via the illumination channel 112. The image sensor 110 detects the visible light 118 reflected from one or more surfaces in the scene. If the visible light 118 includes fluorescent excitation illumination, the image sensor 110 may additionally or alternatively detect fluorescent illumination induced by the fluorescent excitation illumination.

[0036] Image sensor 110 (and / or other circuitry included in imaging device 102) may convert the detected light into image data 120 that represents one or more images of a scene. For example, image sensor 110-R outputs image data 120-R that represents an image captured from a right field of view, and image sensor 110-L outputs image data 120-L that represents an image captured from a left field of view. Image data 120 may have any suitable format.

[0037] The image data 120 is transmitted from the image sensor 110 to the CCU 114. The image data 120 may be transmitted over any suitable communication link between the image sensor 110 and the CCU 114. For example, the image data 120 may be transmitted over wires included in a cable interconnecting the imaging device 102 and the controller 104.

[0038] The CCU 114 may process (e.g., packetize and / or format) the image data 120 and output processed image data 122 (e.g., processed image data 122-R corresponding to image data 120-R and processed image data 122-L corresponding to image data 120-L). The processed image data 122 may be transmitted to an image processing system, which may prepare the processed image data 122 for display on one or more display devices (e.g., in the form of video content and / or one or more still images). For example, the image processing system may generate one or more full color and / or grayscale images for display on one or more display devices based on the image data 122.

[0039] FIG. 2 illustrates an exemplary image sensor 200. The image sensor 200 may implement the image sensor 110 and / or any other image sensor included in the imaging device. As illustrated, the image sensor 200 includes a plurality of pixel arrays 202 (e.g., pixel arrays 202-1 through 202-n). Each pixel array 202 includes a 2×2 arrangement of pixels 204. For example, pixel array 202-1 includes a first pixel 204-1, a second pixel 204-2, a third pixel 204-3, and a fourth pixel 204-4 arranged as shown in FIG. 2. The image sensor 200 may include any suitable number of pixel arrays 202 as may be useful for a particular implementation. Although a 2×2 array is illustrated in FIG. 2, it will be appreciated that each array may include any number of pixels 204 (e.g., a 3×3 pixel array, a 4×4 pixel array, etc.).

[0040] To facilitate color imaging of a scene, the image sensor 200 may include an arrangement of color filters overlying the pixels 204. Each color filter is configured to enable its corresponding pixel 204 to detect only a particular color component of light incident on the pixel 204.

[0041] The color filters may cover the pixels 204 by being coated or otherwise adhered onto a surface (e.g., a surface onto which light may be incident) of the pixels 204. Alternatively, the color filters may cover the pixels 204 in any other suitable manner.

[0042] 3A-3B show example color filter arrangements that may be used in accordance with the systems and methods described herein to cover pixels 204. As shown, a different color filter 302 may cover each pixel 204. For ease of illustration, FIG. 3A shows the filters 302 slightly offset from the pixels 204, while FIG. 3B shows the filters 302 directly covering the pixels 204.

[0043] 3A-3B, a red filter 302-R covers pixel 204-1, a first blue filter 302-B1 covers pixel 204-2, a second blue filter 302-B2 covers pixel 204-3, and a green filter 302-G covers pixel 204-4. Because of the order in which the color filters 302 are arranged, the filter arrangement shown in Figures 3A-3B is referred to herein as RBBG.

[0044] Other color filter arrangements including two blue filters may be used in accordance with the systems and methods described herein. For example, in an alternative embodiment, a first blue filter 302-B1 may cover pixel 204-1, a red filter 302-R may cover pixel 204-2, a green filter 302-G may cover pixel 204-3, and a second blue filter 302-B2 may cover pixel 204-4. This alternative filter configuration may be referred to as BRBG.

[0045] As mentioned above, by using a color filter configuration that is 50% blue, such as the configuration shown in Figures 3A-3B, the systems and methods described herein may more effectively capture blue bias light used in medical imaging applications as compared to a conventional color filter configuration that is only 25% blue. Thus, the 50% blue color filter configuration may produce a clearer and more accurate image of the surgical area associated with the patient.

[0046] In some instances, the color filter configurations described herein may facilitate other types of imaging that may be useful in medical settings. For example, the color filter configurations described herein may be used to generate full-color fluorescent images. By displaying full-color fluorescent images as opposed to traditional grayscale fluorescent images, the systems and methods described herein may enable medical personnel (e.g., surgeons) to more accurately visualize and evaluate blood vessels, bile ducts, tissue perfusion, and other anatomical structures.

[0047] 4 illustrates an exemplary medical imaging system 400 configured to capture a fluorescent image in addition to or in combination with a color image. The medical imaging system 400 is similar to the medical imaging system 100, except that the medical imaging system 400 further includes a fluorescent excitation illumination source 402. The fluorescent excitation illumination source 402 is configured to generate and emit fluorescent excitation illumination 404, which may be applied to a scene (e.g., a surgical field associated with a patient) via the illumination channel 112.

[0048] The fluorescent excitation illumination 404 is configured to induce fluorescent illumination by the fluorescent imaging agent. The fluorescent imaging agent may include any suitable dye, protein, or other substance that may be introduced (e.g., injected) into the patient's bloodstream or other anatomical structure. When exposed to the fluorescent excitation illumination, the fluorescent imaging agent may emit fluorescent illumination (i.e., the fluorescent imaging agent may fluoresce). The fluorescent illumination may be detected by any of the image sensors described herein (e.g., image sensor 110 or image sensor 200) and used to generate fluorescent images indicative of various cellular activities or structures (e.g., blood vasculature in real time).

[0049] In some examples, both the fluorescent excitation illumination and the fluorescent illumination are infrared light. In other words, both the fluorescent excitation illumination and the fluorescent illumination have wavelengths in the infrared light region. The infrared light region includes wavelengths from about 700 nm (the edge of the red visible light region) to about 1 mm. More specifically, both the fluorescent excitation illumination and the fluorescent illumination may have wavelengths in the infrared light region and included in the near infrared light region, including wavelengths from about 700 nm to about 950 nm. For example, an exemplary fluorescent imaging agent fluoresces at 830 nm when excited by fluorescent excitation illumination having a wavelength of 803 nm. In the examples herein, it is assumed that both the fluorescent excitation illumination and the fluorescent excitation illumination have wavelengths in the infrared light region.

[0050] In some scenarios (e.g., when it is desired to generate a full-color fluorescent image), the visible light illumination source 116 and the fluorescent excitation illumination source 402 simultaneously emit visible light and fluorescent excitation illumination, respectively. In these scenarios, the pixels of the image sensor 110 collect both the visible light and the fluorescent illumination without distinguishing between the two types of illumination. Thus, the image data 120 output by the image sensor 110 includes data representing both the color components and the fluorescent illumination components.

[0051] For illustrative purposes, Fig. 5 shows spectral response curves 502 for color filters 302. For example, Fig. 5 shows a spectral response curve 502-1 for red filter 302-R, a spectral response curve 502-2 for blue filters 302-B1 and 302-B2, and a spectral response curve 502-3 for green filter 302-G. The spectral response curves 502 are represented by different variations of solid and dashed lines to visually distinguish the spectral response curves 502 from one another. As shown, the spectral response curves 502 are plotted along horizontal axes representing wavelengths included in the visible light range 504 and the infrared light range 506.

[0052] Spectral response curve 502 illustrates how each color filter 302 enables its corresponding pixel to collect a particular color component of visible light 118 while preventing the pixel from collecting other color components of visible light 118. For example, spectral response curve 502-1 has a relatively high response at wavelengths in the red region of visible light range 504 and a relatively low response at wavelengths in other regions of visible light range 504. Thus, red filter 302-R enables pixel 204-1 to collect the red component of visible light 118 while preventing pixel 204-1 from collecting blue and green components of visible light 118. Spectral response curve 502-2 has a relatively high response at wavelengths in the blue region of visible light range 504 and a relatively low response at wavelengths in other regions of visible light range 504. Thus, blue filters 302-B1 and 302-B2 enable pixels 204-2 and 204-3 to collect the blue component of visible light 118 while preventing pixels 204-2 and 204-3 from collecting the red and green components of visible light 118. Spectral response curve 502-3 has a relatively high response at wavelengths in the green region of visible light range 504 and a relatively low response at wavelengths in other regions of visible light range 504. Thus, green filter 302-G enables pixel 204-4 to collect the green component of visible light 118 while preventing pixel 204-4 from collecting the red and blue components of visible light 118.

[0053] 5, the spectral response curves 502 each have a relatively high response in the infrared light range 506. In other words, each color filter 302 corresponding to a spectral response curve 502 does not prevent the pixel 204 from collecting at least some type of infrared light. For purposes of illustration, as shown in FIG. 5, the color filters 302 do not prevent the pixel 204 from collecting near-infrared light.

[0054] Thus, in accordance with the systems and methods described herein, one or more infrared cut filters may cover one or more of the pixels 204 such that the image processing system may distinguish between the color components contained in the image data 122 and the fluorescent lighting components, thereby instructing the display device to display a full-color fluorescent image.

[0055] 6A-6B illustrate a filter configuration in which a pixel-level broadband infrared cut filter 602 ("cut filter 602") covers pixel 204-2. Cut filter 602 may be implemented by a coating configured to adhere to a surface of pixel 204-2 and / or in any other manner.

[0056] For ease of illustration, Figure 6A shows filters 302 and 602 slightly offset from pixel 204, while Figure 6B shows filters 302 and 602 directly overlying pixel 204. Cut filter 602 is illustrated in Figures 6A-6B as being above blue color filter 302-B1. However, in alternative embodiments, blue color filter 302-B1 may be above cut filter 602.

[0057] As shown, a pixel-level broadband infrared cut filter similar to cut filter 602 does not cover pixels 204-1, 204-3, and 204-4. This may allow the image processing system to advantageously distinguish between color components and fluorescent lighting components contained in image data 122, as described below.

[0058] Cut filter 602 is configured to prevent pixel 204-2 from collecting infrared light having wavelengths that fall within a relatively wide range of wavelengths (e.g., the entire near-infrared range). This is illustrated in Figure 7, which is similar to Figure 5, except that in Figure 7, the spectral response 502-2 of color filter 302-2 is relatively flat in the near-infrared range (e.g., between 700 nm and 950 nm). This is highlighted in Figure 7 by callout 702.

[0059] By preventing pixel 204-2 from collecting infrared light, cut filter 602 may effectively enable pixel 204-2 to collect only the blue component of visible light 118. In contrast, pixel 204-3 is not covered by a cut filter similar to cut filter 602, so pixel 204-3 collects both the blue component and the fluorescent illumination induced by the fluorescent excitation illumination 404 when both visible light illumination source 116 and the fluorescent excitation illumination source 402 are activated and emit light simultaneously. As described below, the image processing system may subtract a signal representative of light collected by pixel 204-3 from a signal representative of light collected by pixel 204-2 to identify the fluorescent illumination component included in the signal representative of light collected by pixel 204-2.

[0060] In some alternative embodiments, one or more of pixels 204-1, 204-3, and 204-4 may be covered by a broadband infrared cut filter, similar to cut filter 602. For example, FIGS. 8A-8B illustrate a filter configuration in which a pixel-level broadband infrared cut filter 602 covers both pixels 204-2 and 204-3. For ease of illustration, FIG. 8A illustrates cut filter 602 slightly offset from pixels 204-2 and 204-3, while FIG. 8B illustrates filter 602 directly covering pixels 204-2 and 204-3. The filter configurations of FIGS. 8A-8B may increase the sharpness of color and grayscale fluorescence images generated based on light captured by pixel 204.

[0061] 9A-9B show an alternative configuration in which a cut filter 602 is used with the RGGB filter configuration. As shown, a red filter 302-R covers pixel 204-1, a first green filter 302-G1 covers pixel 204-2, a second green filter 302-G2 covers pixel 204-3, and a blue filter 302-B covers pixel 204-4. As also shown, the cut filter 602 covers pixel 204-2 without a similar cut filter covering the remaining pixels 204-1, 204-3, and 204-4. For ease of illustration, FIG. 9A shows the filter 302 and cut filter 602 slightly offset from pixel 204, while FIG. 9B shows the filter 302 and cut filter 602 directly covering pixel 204. The configuration shown in Figures 9A-9B may be used, for example, in imaging applications (eg, non-medical applications) where the visible light used for illumination is green biased.

[0062] In some examples, the fluorescent imaging agent fluoresces at a different wavelength than the fluorescent excitation illumination 402. For example, as described above, an exemplary fluorescent imaging agent fluoresces at 830 nm when excited by fluorescent excitation illumination having a wavelength of 803 nm. Because it is desirable to prevent the pixels 204 from detecting the fluorescent excitation illumination 404, while at the same time allowing at least some of the pixels 204 to detect the fluorescent illumination emitted by the fluorescent imaging agent, a narrow band infrared cutoff filter may further be included in the medical imaging systems described herein.

[0063] 10A shows an example filter configuration in which a narrowband infrared cut filter 1002 covers all of the pixels 204. The narrowband infrared cut filter 1002 may be implemented, for example, as a single glass filter that covers all of the pixels 204 and is above the color filters 302 and the cut filters 602. The narrowband infrared cut filter 1002 is configured to prevent the pixels 204 from collecting infrared light having wavelengths that are included in a relatively narrow range of wavelengths (e.g., a range of 20 nm or less) that includes the wavelength of the fluorescence excitation illumination 404. Thus, the narrowband infrared cut filter 1002 effectively prevents the pixels 204 from detecting the fluorescence excitation illumination 404.

[0064] 10B illustrates an alternative configuration in which pixel-level narrow-band infrared cut filters 1004-1, 1004-2, and 1004-3 cover pixels 204-1, 204-3, and 204-4, respectively. The pixel-level narrow-band cut filters 1004 may be implemented in any suitable manner. Similar to the narrow-band infrared cut filter 1002, the pixel-level narrow-band cut filter 1004 may effectively prevent pixels 204-1, 204-3, and 204-4 from detecting the fluorescent excitation illumination 404. It will be appreciated that the cut filter 602 similarly prevents pixel 204-2 from detecting the fluorescent excitation illumination 404, since the wavelength of the fluorescent excitation illumination 404 is within the range of wavelengths blocked by the cut filter 602.

[0065] 11 illustrates an image processing system 1100 that may be configured to generate one or more images for display on a display device based on signals representative of light collected by the pixels 204. The image processing system 1100 may be included in or connected to any of the medical imaging systems described herein. For example, in some examples, the image processing system 1100 is communicatively coupled to the controller 104. In this configuration, the image processing system 1100 may receive processed image data 122 as an input.

[0066] As shown, the system 1100 may include, but is not limited to, a storage facility 1102 and a processing facility 1104 selectively and communicatively coupled to one another. The facilities 1102 and 1104 may each include or be implemented by hardware and / or software components (e.g., a processor, memory, communication interfaces, instructions stored in a memory for execution by a processor, etc.). For example, the facilities 1102 and 1104 may be implemented by any component within a computer-assisted surgery system. In some examples, the facilities 1102 and 1104 may be distributed among multiple devices and / or multiple locations as may be useful for a particular implementation.

[0067] The storage facility 1102 may maintain (e.g., store) executable data used by the processing facility 1104 to perform any of the operations described herein. For example, the storage facility 1102 may store instructions 1106 that may be executed by the processing facility 1104 to perform any of the operations described herein. The instructions 1106 may be implemented by any suitable application, software, code, and / or other executable data instance. The storage facility 1102 may maintain any data received, generated, managed, used, and / or transmitted by the processing facility 1104.

[0068] The processing facility 1104 may be configured to perform various operations related to generating images for display on a display device (e.g., execute instructions 1106 stored in the storage facility 1102 to perform various such operations).

[0069] For example, the processing facility 1104 may activate the visible light illumination source 116 and the fluorescent excitation illumination source 402 to simultaneously emit visible light 118 and fluorescent excitation illumination 404. While the visible light 118 and fluorescent excitation illumination 404 are being simultaneously emitted (or alternatively after the visible light 118 and fluorescent excitation illumination 404 have been emitted), the processing facility 1104 generates a first signal S representative of the light collected by pixel 204-1. 1 , a second signal S representing the light collected by pixel 204-2 2 , a third signal S representing the light collected by pixel 204-3 3 , and a fourth signal S representing the light collected by pixel 204-4. 4 may be received.

[0070] In this example, pixel 204 is covered by the filter configuration shown in Figure 10A or 10B (i.e., an RBBG color filter arrangement with cut filter 602 covering pixel 204-2 and narrow band infrared cut filter 1002 or narrow band infrared cut filter 1004 covering the remaining pixels 204-1, 204-3, and 204-4). For purposes of this example, it is assumed that the fluorescence excitation illumination 404 has a wavelength of 803 nm and the fluorescence illumination induced by the fluorescence excitation illumination 404 has a wavelength of 830 nm.

[0071] Thus, the signal received by the processing facility 1104 may be represented by the following equation:

[0072] S1=R+FI

[0073] S2=B

[0074] S3=B+FI

[0075] S4=G+FI

[0076] In these equations, R represents the red component of the light captured by pixel 204-1, B represents the blue component of the light captured by pixels 204-2 and 204-3, G represents the green component of the light captured by pixel 204-4, and FI represents the fluorescent illumination captured by pixels 204-1, 204-3, and 204-4. Note that none of the signals S1-S4 includes the fluorescent excitation illumination 404 due to the presence of the cut filter 602 covering pixel 204-2 and the narrow band infrared cut filter 1002 or narrow band infrared cut filter 1004 covering pixels 204-1, 204-3, and 204-4.

[0077] The processing facility 1104 may process the signals S1-S4 to selectively generate full color (non-fluorescent) images and full color fluorescent images. For example, Figure 12 illustrates various operations that may be performed by the processing facility 1104 to selectively generate full color images and full color fluorescent images.

[0078] In summing block 1202, the processing facility 1104 first subtracts S2 from S3 to identify the fluorescent lighting component FI contained in S3. This subtraction may be performed using any suitable signal processing operation.

[0079] To identify the red and green components R and G, the processing facility 1104 subtracts FI from S1 in summation block 1204 and subtracts FI from S4 in summation block 1206. As shown, the output of summation block 1204 is a processed first signal representing only the red component R, and the output of summation block 1206 is a processed fourth signal representing only the green component G.

[0080] As shown, R, G, B, and FI are all input to an image generator 1208, which may be implemented by the processing facility 1104 in any suitable manner. The image generator 1208 may generate various types of images for display by a display device based on a received image mode selector command 1210. The image mode selector command 1210 may be provided by a user (e.g., a surgeon), by any component included in the computer-assisted surgery system, and / or by any other source that may be useful for a particular implementation. The image mode selector command 1210 may be received by the image generator 1208 while the visible light 118 and the fluorescent excitation illumination 404 are emitted simultaneously and / or at any other suitable time.

[0081] To illustrate, the image mode selector command 1210 may include a command to present a full color image that does not include a fluorescent image. In response to receiving this command, the image generator 1208 may generate a full color image based on the R, G, and B signals using any suitable image processing technique.

[0082] Alternatively, if the image mode selector command 1210 includes a command to present a full color fluorescent image, the image generator 1208 may generate a full color fluorescent image based on the R, G, B, and IR signals, which may be done using any suitable image processing technique.

[0083] Figure 13 illustrates an example method 1300 that may be performed by image processing system 1100. While Figure 13 illustrates example operations according to one embodiment, other embodiments may omit, add, reorder, and / or modify any of the operations shown in Figure 13.

[0084] In operation 1302, the image processing system 1100 receives a first signal representative of light collected by a first pixel in a 2×2 array of pixels, the first signal including data representative of a combination of a red component of visible light and fluorescent illumination. Operation 1302 may be performed in any of the methods described herein.

[0085] In operation 1304, the image processing system 1100 receives a second signal representative of light collected by a second pixel in the 2×2 array of pixels, the second signal representing a blue component of visible light. Operation 1304 may be performed in any of the methods described herein.

[0086] In operation 1306, the image processing system 1100 receives a third signal representative of light collected by a third pixel in the 2×2 array of pixels, the third signal representing a combination of the blue component and the fluorescent illumination. Operation 1306 may be performed in any of the methods described herein.

[0087] In operation 1308, the image processing system 1100 receives a fourth signal representing light collected by a fourth pixel in the 2×2 array of pixels, the fourth signal representing a combination of the green component of the visible light and the fluorescent illumination. Operation 1308 may be performed in any of the methods described herein.

[0088] In operation 1310, the image processing system 1100 subtracts the second signal from the third signal to identify a fluorescent lighting component of the third signal that is representative of the fluorescent lighting. Operation 1310 may be performed in any of the methods described herein.

[0089] In operation 1312, the image processing system 1100 subtracts the fluorescent illumination component from the first signal to generate a processed first signal representing only the red component. Operation 1312 may be performed in any of the methods described herein.

[0090] In operation 1314, the image processing system 1100 subtracts the fluorescent illumination component from the fourth signal to generate a processed fourth signal representing only the green color component. Operation 1314 may be performed in any of the methods described herein.

[0091] The systems and methods described herein may be used in conjunction with a computer-assisted surgical system used to perform a surgical procedure on a patient. For example, the medical imaging systems 100 and 400 may be included in a computer-assisted surgical system.

[0092] FIG. 14 illustrates an exemplary computer-assisted surgical system 1400 ("surgical system 1400"). As shown, the surgical system 1400 may include an operation system 1402, a user control system 1404, and an auxiliary system 1406 communicatively coupled to each other. The surgical system 1400 may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient 1408. As shown, the surgical team may include a surgeon 1410-1, an assistant 1410-2, a nurse 1410-3, and an anesthesiologist 1410-4, all of which may be collectively referred to as "surgical team members 1410." Additional or alternative surgical team members may be present during a surgical session as may be useful in a particular implementation.

[0093] 14 illustrates a minimally invasive surgical procedure in progress, it will be appreciated that the surgical system 1400 may likewise be used to perform open surgical procedures or other types of surgical procedures that may likewise benefit from the precision and convenience of the surgical system 1400. In addition, it will be appreciated that a surgical session in which the surgical system 1400 may be utilized may include not only the operative phase of a surgical procedure as shown in FIG 14, but may also include pre-operative, post-operative, and / or other appropriate phases of a surgical procedure. A surgical procedure may include any procedure in which manual and / or instrument techniques are used on a patient to investigate or treat a physical condition of the patient.

[0094] 14, the manipulation system 1402 may include multiple manipulator arms 1412 (e.g., manipulator arms 1412-1 through 1412-4) to which multiple surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction capabilities), medical tool, imaging device (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for computer-assisted surgical procedures on a patient 1408 (e.g., by being at least partially inserted into the patient 1408 and manipulated to perform a computer-assisted surgical procedure on the patient 1408). Although the manipulator system 1402 is depicted and described herein as including four manipulator arms 1412, it will be appreciated that the manipulator system 1402 may include only a single manipulator arm 1412 or any other number of manipulator arms as may be useful in a particular implementation.

[0095] The manipulator arm 1412 and / or a surgical instrument attached to the manipulator arm 1412 may include one or more displacement transducers, orientation sensors, and / or position sensors that are used to generate raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 1400 may be configured to use the kinematic information to track (e.g., determine the position of) and / or control the surgical instrument.

[0096] The user control system 1404 may be configured to facilitate control by the surgeon 1410-1 of the manipulator arm 1412 and the surgical instruments attached to the manipulator arm 1412. For example, the surgeon 1410-1 may interact with the user control system 1404 to remotely move or manipulate the manipulator arm 1412 and the surgical instruments. To this end, the user control system 1404 may provide the surgeon 1410-1 with imagery (e.g., high definition 3D images) of the surgical area associated with the patient 1408 as captured by an imaging system (e.g., any of the medical imaging systems described herein). In certain examples, the user control system 1404 may include a stereo viewer having two displays, and stereoscopic images of the surgical area associated with the patient 1408 and generated by the stereoscopic imaging system may be viewed by the surgeon 1410-1. The surgeon 1410-1 may use the image to perform one or more procedures using one or more surgical instruments attached to the manipulator arm 1412.

[0097] To facilitate control of the surgical instruments, the user control system 1404 may include a set of master controllers. These master controllers may be operated by the surgeon 1410-1 to control the movements of the surgical instruments (e.g., by utilizing robotic and / or teleoperation techniques). The master controllers may be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 1410-1. In this manner, the surgeon 1410-1 may intuitively perform a procedure with one or more surgical instruments.

[0098] The auxiliary system 1406 may include one or more computing devices configured to perform primary processing operations of the surgical system 1400. In such a configuration, one or more computing devices included in the auxiliary system 1406 may control and / or coordinate operations performed by various other components of the surgical system 1400 (e.g., the manipulation system 1402 and the user control system 1404). For example, a computing device included in the user control system 1404 may send instructions to the manipulation system 1402 via one or more computing devices included in the auxiliary system 1406. As another example, the auxiliary system 1406 may receive and process image data from the manipulation system 1402 representing an image captured by an imaging device attached to one of the manipulator arms 1412.

[0099] In some examples, the auxiliary system 1406 may be configured to present visual content to surgical team members 1410 who may not have access to images provided to the surgeon 1410-1 by the user control system 1404. To this end, the auxiliary system 1406 may include a display monitor 1414 configured to display one or more user interfaces, such as images of the surgical field (e.g., 2D images), information related to the patient 1408 and / or the surgical procedure, and / or any other visual content that may be useful in a particular implementation. For example, the display monitor 1414 may display images of the surgical field along with additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the images. In some embodiments, the display monitor 1414 may be implemented by a touch screen display, and the surgical team members 1410 may interact with the touch screen display (e.g., by touch gestures) to provide user input to the surgical system 1400.

[0100] The operation system 1402, the user control system 1404, and the auxiliary system 1406 may be communicatively coupled to one another in any suitable manner. For example, as shown in FIG. 14, the operation system 1402, the user control system 1404, and the auxiliary system 1406 may be communicatively coupled by control lines 1416, which may represent any wired or wireless communication links that may be useful in a particular implementation. To this end, the operation system 1402, the user control system 1404, and the auxiliary system 1406 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0101] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and / or the computing device to perform one or more operations, including one or more operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0102] A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memories ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0103] 15 illustrates an exemplary computing device 1500 that may be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 1500.

[0104] As shown in Figure 15, computing device 1500 may include a communication interface 1502, a processor 1504, a storage device 1506, and an input / output ("I / O") module 1508 that are communicatively connected to each other via a communication infrastructure 1510. Although an exemplary computing device 1500 is shown in Figure 15, the components illustrated in Figure 15 are not intended to be limiting. In other embodiments, additional or alternative components may be used. The components of computing device 1500 shown in Figure 15 will now be described in additional detail.

[0105] The communications interface 1502 may be configured to communicate with one or more computing devices. Examples of the communications interface 1502 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0106] The processor 1504 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1504 may perform operations by executing computer-executable instructions 1512 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1506.

[0107] The storage device 1506 may include one or more data storage media, devices, or configurations and may utilize any type, form, and combination of data storage media and / or devices. For example, the storage device 1506 may include, but is not limited to, any combination of non-volatile and / or volatile media described herein. Electronic data, including data described herein, may be stored temporarily and / or permanently in the storage device 1506. For example, data representing computer-executable instructions 1512 configured to direct the processor 1504 to perform any of the operations described herein may be stored in the storage device 1506. In some examples, the data may be located in one or more databases residing in the storage device 1506.

[0108] The I / O module 1508 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1508 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1508 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0109] I / O module 1508 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, I / O module 1508 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may be useful in a particular implementation.

[0110] In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto and additional embodiments may be implemented without departing from the scope of the invention as set forth in the following claims. For example, specific features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Thus, the specification and drawings should not be regarded in a restrictive sense, but rather in an illustrative sense.

Claims

1. A method of operating a processing system including one or more processors, comprising: receiving, by the processing system, a first signal representative of light collected by a first pixel included in a 2×2 array of pixels, the first signal including data representative of a combination of a first color component of visible light and fluorescent illumination; receiving, by the processing system, a second signal representative of light collected by a second pixel in the 2×2 array of pixels, the second signal representing a second color component of the visible light; receiving, by the processing system, a third signal representative of light collected by a third pixel in the 2×2 array of pixels, the third signal representing a combination of the second color component and the fluorescent illumination; receiving, by the processing system, a fourth signal representative of light collected by a fourth pixel in the 2×2 array of pixels, the fourth signal representing a combination between a third color component of the visible light and the fluorescent illumination; the processing system subtracting the second signal from the third signal to identify a fluorescent lighting component of the third signal representative of the fluorescent lighting; the processing system subtracting the fluorescent lighting component from the first signal to generate a processed first signal representative of only the first color component; and the processing system subtracting the fluorescent lighting component from the fourth signal to generate a processed fourth signal representative of only the third color component. How it works.

2. receiving a command from the processing system to present a full color image that does not include a fluorescent image; and in response to the command, the processing system generates the full color image based on the processed first signal, the second signal, and the processed fourth signal. The method of claim 1 .

3. The method of claim 2, further comprising: receiving a command to display a full-color fluorescent image, the processing system comprising: and in response to receiving the command, the processing system generates the full-color fluorescent image based on the processed first signal, the second signal, the processed fourth signal, and a fluorescent illumination component. The method of claim 1 .

4. the first pixel is covered by a first color filter configured to enable the first pixel to collect the first color component of the visible light and to prevent the first pixel from collecting the second and third color components of the visible light; the second and third pixels are covered by a second color filter configured to enable the second and third pixels to collect the second color component of the visible light, respectively, and to prevent the second and third pixels from collecting the first and third color components of the visible light, respectively; the fourth pixel is covered by a third color filter configured to enable the fourth pixel to collect the third color component of the visible light and to prevent the fourth pixel from collecting the first and second color components of the visible light; a pixel-level broadband infrared cutoff filter configured to cover the second pixel and prevent the second pixel from collecting infrared light having a wavelength included in a first wavelength range; The method of claim 1 .

5. 5. The method of claim 4, wherein a narrow band infrared cutoff filter is configured to cover the first, third, and fourth pixels and prevent the first, third, and fourth pixels from collecting infrared light having a wavelength included in a second wavelength range, the second wavelength range being included in the first wavelength range and narrower than the first wavelength range.

6. 6. The method of claim 4 or 5, wherein the first, third and fourth pixels are not covered by a pixel level broadband infrared cutoff filter configured to prevent the first, third and fourth pixels from collecting infrared light having a wavelength included in the first wavelength range.

7. The narrow band infrared cutoff filter is a first pixel level narrow band infrared cutoff filter covering the first pixel; a second pixel level narrow band infrared cutoff filter covering the third pixel; and a third pixel level narrow band infrared cutoff filter covering the fourth pixel.

6. The method of claim 5.

8. The method of claim 5, further comprising the processing system activating a visible light illumination source and a fluorescent excitation illumination source to illuminate a scene with the visible light and fluorescent excitation illumination that induces fluorescent illumination, and the 2x2 array of pixels collecting the color components of the visible light after the visible light reflects off one or more surfaces in the scene.

9. the fluorescence excitation illumination has a wavelength that is included in the second wavelength range such that the pixel-level broadband infrared cutoff filter and the narrowband infrared cutoff filter prevent the first, second, third, and fourth pixels from collecting the fluorescence excitation illumination; the fluorescent illumination has a wavelength that is included in the first wavelength range but not included in the second wavelength range, such that the pixel-level broadband infrared cutoff filter prevents the second pixel from collecting the fluorescent illumination, while the narrowband infrared cutoff filter allows the first, third, and fourth pixels to collect the fluorescent illumination.

9. The method of claim 8.

10. The processing system activates the visible light illumination source and the fluorescent excitation illumination source to simultaneously emit the visible light and the fluorescent excitation illumination; receiving the first through fourth signals occurs while the visible light and the fluorescent excitation illumination are emitted simultaneously; 9. The method of claim 8.

Images