Medical imaging system and method

Abstract

Medical imaging systems and methods. A 2 * 2 pixel array within an image sensor includes a first pixel, a second pixel, a third pixel, and a fourth pixel. The red filter covers the first pixel, the first blue filter covers the second pixel, the second blue filter covers the third pixel, and the green filter covers the fourth pixel. The red filter is configured to allow the first pixel to collect a red component of the visible light and to prevent the first pixel from collecting blue and green components of the visible light. The first and second blue filters are configured to allow the second and third pixels to each collect a blue component and to prevent the second and third pixels from collecting red and green components. The green filter is configured to allow the fourth pixel to collect a green component and to prevent the fourth pixel from collecting red and blue components.

Description

[0001] This application is a divisional application of Chinese patent application 2019800784096 (PCT / US2019 / 063182), whose international application date is November 26, 2019 and whose national phase entry date is May 28, 2021, and whose name is “Medical Imaging Systems and Methods”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 774,041, filed on November 30, 2018, entitled “Medical Imaging Systems and Methods,” the contents of which are hereby incorporated by reference in their entirety. Background Art

[0004] Conventional image sensors included in imaging devices typically include an array of pixels (also called photosites or light sensors), each of which detects light reflected from surfaces in a scene. The detected light can then be converted into data representing an image of the scene.

[0005] To facilitate the generation of color images, the image sensor may include an arrangement of color filters covering the pixels. Each color filter is configured to allow its corresponding pixel to detect only a specific color component of the light incident on the pixel. For example, a conventional color filter arrangement (e.g., a Bayer filter arrangement) is 50% green, 25% red, and 25% blue. In other words, for each 2×2 pixel array included in the image sensor, two of the pixels are configured to be covered by a green filter that allows the two pixels to detect only the green component of visible light, one of the pixels is configured to be covered by a red filter that allows the pixel to detect only the red component of visible light, and one of the pixels is configured to be covered by a blue filter that allows the pixel to detect only the blue component of visible light. This color filter arrangement is commonly referred to as RGGB, BGGR, RGBG, or GRGB. Unfortunately, this color filter arrangement is not well suited for medical applications in which images of internal anatomical structures are acquired using an imaging device (e.g., an endoscope) configured to be positioned within a patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings illustrate various embodiments and are part of the specification. The embodiments shown are examples only and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0007] Figure 1 An exemplary medical imaging system according to the principles described herein is shown.

[0008] Figure 2 An illustrative image sensor according to the principles described herein is shown.

[0009] Figure 3A-3B An exemplary color filter arrangement is shown according to the principles described herein.

[0010] Figure 4 An exemplary medical imaging system according to the principles described herein is shown.

[0011] Figure 5 Shown is the spectral response curve of a color filter based on the principles described herein.

[0012] Figure 6A-6B Shown is a filter configuration according to the principles described herein.

[0013] Figure 7 Shown is the spectral response curve of a color filter based on the principles described herein.

[0014] Figures 8A-10B An exemplary filter configuration is shown according to the principles described herein.

[0015] Fig.11 An image processing system according to the principles described herein is shown.

[0016] Fig.12 Various operations that may be performed by an imaging processing system to selectively generate full-color images and full-color fluorescence images in accordance with the principles described herein are shown.

[0017] Fig.13 An exemplary method according to the principles described herein is shown.

[0018] Fig.14 An exemplary computer-assisted surgery system according to the principles described herein is shown.

[0019] Fig.15 An exemplary computing device according to the principles described herein is shown. DETAILED DESCRIPTION

[0020] Medical imaging systems and methods are described herein. As will be described in more detail below, an exemplary medical imaging system includes an image sensor comprising a 2×2 pixel array. The 2×2 array includes a first pixel, a second pixel, a third pixel, and a fourth pixel. A red filter covers the first pixel, a first blue filter covers the second pixel, a second blue filter covers the third pixel, and a green filter covers the fourth pixel. The red filter is configured to allow the first pixel to collect a red component of visible light and prevent the first pixel from collecting blue and green components of visible light. The first and second blue filters are configured to allow the second and third pixels to each collect a blue component of visible light and prevent the second and third pixels from each collecting red and green components of visible light. The green filter is configured to allow the fourth pixel to collect a green component of visible light and prevent the fourth pixel from collecting red and blue components of visible light.

[0021] Various advantages and benefits are associated with the medical imaging systems and methods described herein. For example, in medical imaging applications, it is often desirable to use blue-biased illumination (i.e., light having more blue components than green or red components). This is because the wavelengths of the green and red components are longer than the blue component and therefore penetrate deeper into the patient's anatomy. This, in turn, causes the green and red components to scatter more than the blue component, resulting in the red and green components being relatively more blurred in the image acquired by the imaging device. Therefore, in medical imaging applications, blue-biased illumination produces clearer images than illumination that is biased toward other colors (e.g., green). By configuring 50% of the pixels in a 2×2 pixel array to acquire the blue component of light, the systems and methods described herein can more efficiently acquire blue-biased light and thereby produce clearer and more accurate images of patient anatomy.

[0022] Another exemplary medical imaging system described herein includes an image sensor including a 2×2 pixel array, the 2×2 pixel array including a first pixel, a second pixel, a third pixel, and a fourth pixel. A first color filter covers the first pixel and is configured to allow the first pixel to collect a first color component of visible light and prevent the first pixel from collecting a second and third component of visible light. A second color filter covers the second and third pixels. The second color filter is configured to allow the second and third pixels to each collect a second color component of visible light and prevent the second and third pixels from each collecting a first and third color component of visible light. The third color filter covers the fourth pixel and is configured to allow the fourth pixel to collect a third color component of visible light and prevent the fourth pixel from collecting first and second color components of visible light. A pixel-level broadband infrared cutoff filter covers the second pixel and is configured to prevent the second pixel from collecting infrared light having a wavelength included in the first wavelength range. In some instances, a narrowband infrared cutoff filter covers the first, third, and fourth pixels and is configured to prevent the first, third, and fourth pixels from collecting infrared light having a wavelength included in the second wavelength range. The second wavelength range is included in the first wavelength range and is narrower than the first wavelength range. As described below, this configuration advantageously provides three color component channels and a fluorescence illumination channel that can allow the systems and methods described herein to selectively display a full-color fluorescence image. By displaying a full-color fluorescence image as opposed to a conventional grayscale fluorescence image, the systems and methods described herein can allow medical personnel (e.g., surgeons) to more accurately visualize and assess blood vessels, bile ducts, tissue perfusion, and other anatomical features.

[0023] Figure 1 An exemplary medical imaging system 100 configured to acquire an image of a scene is shown. In some instances, the scene can include a surgical field associated with a patient. In some instances, the surgical field can be completely disposed within the patient and can include an area within the patient's body at or near a location where a surgical procedure is planned, is being performed, or has been performed. For example, for minimally invasive surgery performed on tissue inside a patient, the surgical field can include the tissue, the anatomical structures underlying the tissue, and the space surrounding the tissue, such as the location where surgical instruments used to perform the surgical procedure are positioned. In other instances, the surgical field can be at least partially disposed outside the patient.

[0024] As shown, the medical imaging system 100 includes an imaging device 102 and a controller 104. The medical imaging system 100 may include additional or optional components that may be useful for a particular implementation. For example, the medical imaging system 100 may include various optical and / or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, etc.), cables that house wires and / or optical fibers and are configured to interconnect the imaging device 102 and the controller 104, etc.

[0025] The imaging device 102 may be implemented by an endoscope or other camera device configured to acquire images of a scene. As shown, the imaging device 102 includes a camera head 106, a shaft 108 coupled to the camera head 106 and extending away therefrom, image sensors 110 at the distal end of the shaft 108 (i.e., right image sensor 110-R and left image sensor 110-L), and an illumination channel 112. Figure 1 In the example of , the imaging device 102 is stereoscopic. Alternatively, in other examples, the imaging device 102 can be monoscopic (eg, by including one image sensor 110 instead of two image sensors 110).

[0026] The imaging device 102 can be manually operated and controlled (e.g., by a surgeon performing surgery on a patient). Alternatively, the camera head 106 can be coupled to a manipulator arm of a computer-assisted surgery system, an example of which will be provided below. In this configuration, the imaging device 102 can be controlled using robotics and / or teleoperation technology.

[0027] The distal end of the shaft 108 can be positioned at or near a scene to be imaged by the imaging device 108. For example, the distal end of the shaft 108 can be inserted into a patient. In this configuration, the imaging device 102 can be used to acquire images of anatomical structures and / or other objects within the patient.

[0028] Image sensors 110 may each be implemented by any suitable image sensor, such as a charge coupled device (“CCD”) image sensor, a complementary metal oxide semiconductor (“CMOS”) image sensor, etc. In some examples, as Figure 1 , the image sensor 110 is positioned at the distal end of the shaft 108. Alternatively, the image sensor 110 may be positioned closer to the proximal end of the shaft 108, inside the camera head 106, or outside the imaging device 102 (e.g., inside the controller 104). In these alternative configurations, optics (e.g., lenses, optical fibers, etc.) included in the shaft 108 and / or the camera head 106 may transmit light from the scene to the image sensor 110.

[0029] Image sensor 110 is configured to detect (e.g., acquire, collect, sense, or otherwise obtain) light. For example, image sensor 110-R is configured to detect light from a right angle, and image sensor 110-L is configured to detect light from a left angle. Figure 1In the example of , the light detected by the image sensor 110 includes visible light reflected by objects located within the scene. However, as described herein, the image sensor 110 may additionally or alternatively detect infrared fluorescent illumination emitted by a fluorescent imaging agent located within the scene. As shown below, the image sensor 110 may convert the detected light into data representing one or more images. Exemplary implementations of the image sensor 110 are described herein.

[0030] The illumination channel 112 may be implemented by one or more optical components (eg, optical fibers, light pipes, lenses, etc.) As described below, illumination may be provided by way of the illumination channel 112 to illuminate a scene.

[0031] The controller 104 may be implemented by any suitable combination of hardware and software configured to control and / or interface with the imaging device 102. For example, the controller 104 may be implemented at least in part by a computing device included in a computer-assisted surgery system.

[0032] Controller 104 includes a camera control unit (“CCU”) 114 and a visible light illumination source 116. Controller 104 may include additional or optional components, as may serve a particular implementation. For example, controller 104 may include circuitry configured to provide power to components included in imaging device 102. In some examples, CCU 114 and / or visible light illumination source 116 are optionally included within imaging device 102 (e.g., within camera head 106).

[0033] CCU 114 is configured to control various parameters of image sensor 110 (e.g., activation time, automatic exposure, etc.). As described below, CCU 118 may be further configured to receive image data from image sensor 110 and perform processing. Figure 1 The CCU 114 is shown as a single unit, but the CCU 114 may alternatively be implemented by a first CCU configured to control the right image sensor 110 -R and a second CCU configured to control the left image sensor 110 -L.

[0034] Visible light illumination source 116 may be configured to generate and emit visible light 118. Visible light 118 may travel by way of illumination channel 112 to a distal end of shaft 108 where visible light 118 exits to illuminate a scene.

[0035] Visible light 118 may include one or more color components. For example, visible light 118 may include white light that includes a full spectrum of color components (e.g., red, green, and blue color components). The wavelength of the red component is between approximately 635 and 700 nanometers ("nm"). The wavelength of the green component is between approximately 520 and 560 nm. The wavelength of the blue component is between approximately 450 and 490 nm.

[0036] In some examples, visible light 118 is biased to include more of one color component than another color component. For example, visible light 118 can be biased toward blue by including more blue components than red and green components.

[0037] To acquire one or more images of a scene, controller 104 (or any other suitable computing device) may activate visible light illumination source 116 and image sensor 110. When activated, visible light illumination source 116 emits visible light 118, which travels to the scene via illumination channel 112. Image sensor 110 detects visible light 118 reflected by one or more surfaces in the scene. In the case where visible light 118 includes fluorescence excitation illumination, image sensor 110 may additionally or alternatively detect fluorescence illumination induced by the fluorescence excitation illumination.

[0038] Image sensor 110 (and / or other circuitry included in imaging device 102) may convert the detected light into image data 120 representing one or more images of a scene. For example, image sensor 110-R outputs image data 120-R representing an image taken from a right angle, and image sensor 110-L outputs image data 120-L representing an image taken from a left angle. Image data 120 may have any suitable format.

[0039] Image data 120 is sent from image sensor 110 to CCU 114. Image data 120 may be sent by way of any suitable communication link between image sensor 110 and CCU 114. For example, image data 120 may be sent by way of wires included in a cable interconnecting imaging device 102 and controller 104.

[0040] The CCU 114 may process (e.g., packetize and / or format) the image data 120 and output processed image data 122 (e.g., processed image data 122-R corresponding to image data 120-R and processed image data 122-L corresponding to image data 120-L). The processed image data 122 may be sent to an image processing system, which may prepare the processed image data 122 for display on one or more display devices (e.g., in the form of video content and / or one or more still images). For example, the image processing system may, based on the image data 122, generate one or more full-color and / or grayscale images for display on one or more display devices.

[0041] Figure 2 An exemplary image sensor 200 is shown. Image sensor 200 may implement image sensor 110 and / or any other image sensor included in an imaging device. As shown, image sensor 200 includes a plurality of pixel arrays 202 (e.g., pixel array 202-1 through pixel array 202-n). Each pixel array 202 includes a 2×2 arrangement of pixels 204. For example, pixel array 202-1 includes pixels 204. Figure 2 2, the first pixel 204-1, the second pixel 204-2, the third pixel 204-3, and the fourth pixel 204-4 are arranged as shown in FIG. Image sensor 200 may include any suitable number of pixel arrays 202 as may be useful for a particular implementation. Figure 2 A 2×2 array is shown in FIG. 2 , but it will be appreciated that each array may include any number of pixels 204 (eg, a 3×3 pixel array, a 4×4 pixel array, etc.).

[0042] To facilitate color imaging of a scene, image sensor 200 may include an arrangement of color filters overlying pixels 204. Each color filter is configured to allow its corresponding pixel 204 to detect only a particular color component of light incident on pixel 204.

[0043] The color filter may cover the pixel 204 by being coated on or otherwise adhered to a surface (eg, a surface upon which light may be incident) of the pixel 204. The color filter may alternatively cover the pixel 204 in any other suitable manner.

[0044] Figure 3A-3B 204 and can be used in accordance with the systems and methods described herein. As shown, a different color filter 302 can cover each pixel 204. For ease of illustration, Figure 3A Filter 302 is shown slightly offset from pixel 204, while Figure 3B Filter 302 is shown directly overlying pixel 204 .

[0045] like Figure 3A-3B As shown in FIG. 3 , the red filter 302-R covers the pixel 204-1, the first blue filter 302-B1 covers the pixel 204-2, the second blue filter 302-B2 covers the pixel 204-3, and the green filter 302-G covers the pixel 204-4. Due to the arrangement of the color filters 302 in sequence, Figure 3A-3B The filter arrangement shown in is referred to herein as RBBG.

[0046] Other color filter arrangements including two blue filters may also be used according to the systems and methods described herein. For example, in an alternative embodiment, a first blue filter 302-B1 may cover pixel 204-1, a red filter 302-R may cover pixel 204-2, a green filter 302-G may cover pixel 204-3, and a second blue filter 302-B2 may cover pixel 204-4. This alternative filter arrangement may be referred to as BRBG.

[0047] As mentioned, by using a 50% blue color filter arrangement, as compared to a conventional color filter arrangement of only 25% blue, e.g. Figure 3A-3B As shown in FIG. 1 , the systems and methods described herein can more effectively acquire blue-biased light for medical imaging applications. Thus, a 50% blue color filter arrangement can produce a clearer and more accurate image of a surgical area associated with a patient.

[0048] In some instances, the color filter arrangements described herein can facilitate other types of imaging that may be useful in a medical setting. For example, the color filter arrangements described herein can be used to generate full-color fluorescence images. By displaying full-color fluorescence images as opposed to conventional grayscale fluorescence images, the systems and methods described herein can allow medical personnel (e.g., surgeons) to more accurately visualize and assess blood vessels, bile ducts, tissue perfusion, and other anatomical features.

[0049] Figure 4 An exemplary medical imaging system 400 is shown that is configured to capture fluorescence images in addition to or in combination with color images. Medical imaging system 400 is similar to medical imaging system 100, except that medical imaging system 400 further includes a fluorescence excitation illumination source 402. Fluorescence excitation illumination source 402 is configured to generate and emit fluorescence excitation illumination 404, which can be applied to a scene (e.g., a surgical area associated with a patient) by way of illumination channel 112.

[0050] Fluorescence excitation illumination 404 is configured to induce fluorescence illumination through a fluorescent imaging agent. The fluorescent imaging agent may include any suitable dye, protein, or other substance that can be introduced (e.g., injected) into the patient's bloodstream or other anatomical features. When exposed to the fluorescence excitation illumination, the fluorescent imaging agent may emit fluorescent illumination (i.e., the fluorescent imaging agent may fluoresce). The fluorescent illumination may be detected by any image sensor described herein (e.g., image sensor 110 or image sensor 200) and used to generate a fluorescent image indicating various cell activities or structures (e.g., real-time vascular system).

[0051] In some examples, both the fluorescence excitation illumination and the fluorescent illumination are infrared light. In other words, both the fluorescence excitation illumination and the fluorescent illumination have wavelengths in the infrared light region. The infrared light region includes wavelengths from about 700nm (the edge of the red visible light region) to about 1 mm. More specifically, both the fluorescence excitation illumination and the fluorescent illumination may have wavelengths included in the near infrared light region, which is within the infrared light region and includes wavelengths from about 700nm to about 950nm. For example, when excited by fluorescence excitation illumination having a wavelength of 803nm, an exemplary fluorescent imaging agent fluoresces at 830nm. In the examples herein, it will be assumed that both the fluorescence excitation illumination and the fluorescent excitation illumination have wavelengths in the infrared light region.

[0052] In some scenarios (e.g., when it is desired to generate a full-color fluorescence image), the visible light illumination source 116 and the fluorescence excitation illumination source 402 emit visible light and fluorescence excitation illumination, respectively, simultaneously. In these scenarios, the pixels of the image sensor 110 collect both visible light and fluorescence illumination without distinguishing between the two types of illumination. Therefore, the image data 120 output by the image sensor 110 includes data representing both the color component and the fluorescence illumination component.

[0053] To illustrate, Figure 5 A spectral response curve 502 is shown for the color filter 302. For example, Figure 5 Spectral response curve 502-1 of red filter 302-R, spectral response curve 502-2 of blue filters 302-B1 and 302-B2, and spectral response curve 502-3 of green filter 302-G are shown. Spectral response curves 502 are represented by different variations of solid and dashed lines to visually distinguish spectral response curves 502 from each other. As shown, spectral response curves 502 are plotted along a horizontal axis, which represents wavelengths included in a visible light range 504 and an infrared light range 506.

[0054] Spectral response curves 502 illustrate how each color filter 302 allows its corresponding pixel to collect a particular color component of visible light 118 while blocking the pixel from collecting other color components of visible light 118. For example, spectral response curve 502-1 has a relatively high response at wavelengths within the red region of visible light range 504, and a relatively low response at wavelengths within other regions of visible light range 504. Thus, red filter 302-R allows pixel 204-1 to collect a red component of visible light 118 while blocking pixel 204-1 from collecting blue and green components of visible light 118. Spectral response curve 502-2 has a relatively high response at wavelengths within the blue region of visible light range 504, and a relatively low response at wavelengths within other regions of visible light range 504. Thus, blue filters 302-B1 and 302-B2 allow pixels 204-2 and 204-3 to collect a blue component of visible light 118 while blocking pixels 204-2 and 204-3 from collecting red and green components of visible light 118. Spectral response curve 502-3 has a relatively high response at wavelengths within the green region of visible light range 504 and a relatively low response at wavelengths within other regions of visible light range 504. Thus, green filter 302-G allows pixel 204-4 to collect the green component of visible light 118 while blocking pixel 204-4 from collecting the red and blue components of visible light 118.

[0055] Also like Figure 5 As shown in , spectral response curves 502 each have a relatively high response in the infrared light range 506. In other words, each color filter 302 corresponding to spectral response curve 502 does not prevent pixel 204 from collecting at least some types of infrared light. Figure 5 As shown in , color filter 302 may not prevent pixel 204 from collecting near infrared light.

[0056] Thus, in accordance with the systems and methods described herein, one or more infrared cut filters may cover one or more of pixels 204 such that an image processing system can distinguish between color components and fluorescent illumination components included in image data 122, thereby instructing a display device to display a full-color fluorescent image.

[0057] To illustrate, Figure 6A-6B A filter configuration is shown in which a pixel-level broadband infrared cut filter 602 ("cut filter 602") covers pixel 204-2. Cut filter 602 may be implemented by a coating configured to adhere to a surface of pixel 204-2 and / or in any other manner.

[0058] For ease of explanation, Fig. 6AFilters 302 and 602 are shown slightly offset from pixel 204. Figure 6B Filters 302 and 602 are shown directly overlying pixel 204. Cutoff filter 602 is Figure 6A-6B However, in an alternative embodiment, the blue filter 302 - B1 may be above the cutoff filter 602 .

[0059] As shown, pixels 204-1, 204-3, and 204-4 are not covered by a pixel-level broadband infrared cut filter similar to cut filter 602. As described below, this can advantageously allow an image processing system to distinguish between color components and fluorescent illumination components included in image data 122.

[0060] Cut-off filter 602 is configured to block pixel 204-2 from collecting infrared light having a wavelength included in a relatively wide wavelength range (eg, the entire near infrared range). Figure 7 It is shown in Figure 5 , the difference is that in Figure 7 , the spectral response 502-2 of the color filter 302-2 is relatively flat in the near infrared range (e.g., between 700 nm and 950 nm). Figure 7 It is highlighted by label 702.

[0061] By blocking pixel 204-2 from collecting infrared light, cutoff filter 602 can effectively allow pixel 204-2 to collect only the blue component of visible light 118. In contrast, because pixel 204-3 is not covered by a cutoff filter similar to cutoff filter 602, when both visible light illumination source 116 and fluorescence excitation illumination source 402 are activated and emitting light simultaneously, pixel 204-3 collects the blue component and the fluorescence illumination induced by fluorescence excitation illumination 404. As described below, the image processing system can subtract the signal representing the light collected by pixel 204-3 from the signal representing the light collected by pixel 204-2 to identify the fluorescence illumination component included in the signal representing the light collected by pixel 204-2.

[0062] In some optional embodiments, one or more of the pixels 204-1, 204-3, and 204-4 may also be covered by a broadband infrared cut-off filter similar to the cut-off filter 602. For example, Figures 8A-8B A filter configuration is shown in which a pixel-level broadband infrared cut filter 602 covers both pixels 204-2 and 204-3. For ease of illustration, Fig. 8A The cutoff filter 602 is shown slightly offset from pixels 204-2 and 204-3, while Figure 8B Filter 602 is shown directly overlying pixels 204-2 and 204-3. Figures 8A-8B The filter configuration may increase the clarity of color images and grayscale fluorescence images generated based on light captured by pixel 204 .

[0063] Figures 9A-9B An alternative configuration is shown in which a cutoff filter 602 is used in conjunction with an RGGB filter configuration. As shown, a red filter 302-R covers pixel 204-1, a first green filter 302-G1 covers pixel 204-2, a second green filter 302-G2 covers pixel 204-3, and a blue filter 302-B covers pixel 204-4. Also as shown, the cutoff filter 602 also covers pixel 204-2 without a similar cutoff filter covering the remaining pixels 204-1, 204-3, and 204-4. For ease of illustration, Fig.9A Filter 302 and cutoff filter 602 are shown slightly offset from pixel 204, and Fig. 9B Filter 302 and cutoff filter 602 are shown directly overlying pixel 204 . Figures 9A-9B The configuration shown in , for example, can be used in imaging applications (eg, non-medical uses) where the visible light used as illumination has a green tint.

[0064] In some examples, the fluorescent imaging agent fluoresces at a different wavelength than the fluorescent excitation illumination 402. For example, as mentioned above, the exemplary fluorescent imaging agent fluoresces at 830 nm when excited by the fluorescent excitation illumination having a wavelength of 803 nm. Because it is desirable to prevent the pixels 204 from detecting the fluorescent excitation illumination 404 while allowing at least some of the pixels 204 to detect the fluorescent illumination emitted by the fluorescent imaging agent, a narrow-band infrared cutoff filter may be further included in the medical imaging system described herein.

[0065] To illustrate, Fig. 10A An exemplary filter configuration is shown in which a narrow-band infrared cut filter 1002 covers all pixels 204. The narrow-band infrared cut filter 1002 can be implemented, for example, as a single glass filter covering all pixels 204 and above the color filter 302 and the cut filter 602. The narrow-band infrared cut filter 1002 is configured to block the pixels 204 from collecting infrared light having a wavelength included in a relatively narrow wavelength range (e.g., a range of 20 nm or less) that includes the wavelength of the fluorescent excitation illumination 404. Therefore, the narrow-band infrared cut filter 1002 effectively blocks the pixels 204 from detecting the fluorescent excitation illumination 404.

[0066] Fig. 10BAn optional configuration is shown in which pixel-level narrow-band infrared cutoff filters 1004-1, 1004-2, and 1004-3 cover pixels 204-1, 204-3, and 204-4, respectively. The pixel-level narrow-band cutoff filter 1004 can be implemented in any suitable manner. Like the narrow-band infrared cutoff filter 1002, the pixel-level narrow-band cutoff filter 1004 can effectively prevent the pixels 204-1, 204-3, and 204-4 from detecting the fluorescent excitation illumination 404. It will be appreciated that the cutoff filter 602 also prevents the pixel 204-2 from detecting the fluorescent excitation illumination 404 because the wavelength of the fluorescent excitation illumination 404 is within the wavelength range blocked by the cutoff filter 602.

[0067] Fig.11 An image processing system 1100 is shown that can be configured to generate one or more images displayed on a display device based on signals representing light collected by pixels 204. The image processing system 1100 can be included in any medical imaging system described herein or connected to any medical imaging system described herein. For example, in some examples, the image processing system 1100 is communicatively coupled to the controller 104. In this configuration, the image processing system 1100 can receive processed image data 122 as input.

[0068] As shown, the system 1100 may include, but is not limited to, a storage facility 1102 and a processing facility 1104 that are selectively and communicatively coupled to each other. The facilities 1102 and 1104 may each include or be implemented by hardware and / or software components (e.g., a processor, a memory, a communication interface, instructions stored in the memory for execution by the processor, etc.). For example, the facilities 1102 and 1104 may be implemented by any component in a computer-assisted surgery system. In some instances, the facilities 1102 and 1104 may be distributed among multiple devices and / or multiple locations, as may be useful for a particular implementation.

[0069] The storage facility 1102 may maintain (e.g., store) executable data used by the processing facility 1104 to perform any of the operations described herein. For example, the storage facility 1102 may store instructions 1106 that may be executed by the processing facility 1104 to perform any of the operations described herein. The instructions 1106 may be implemented by any suitable application, software, code, and / or other executable data instance. The storage facility 1102 may also maintain any data received, generated, managed, used, and / or sent by the processing facility 1104.

[0070] The processing facility 1104 may be configured to perform (eg, execute instructions 1106 stored in the storage facility 1102 to perform) various operations associated with generating an image for display on a display device.

[0071] For example, processing facility 1104 may activate visible light illumination source 116 and fluorescent excitation illumination source 402 to simultaneously emit visible light 118 and fluorescent excitation illumination 404. While visible light 118 and fluorescent excitation illumination 404 are being emitted simultaneously (or, alternatively, after visible light 118 and fluorescent excitation illumination 404 have been emitted), processing facility 1104 may receive a first signal S representing light collected by pixel 204-1. 1 , a second signal S representing the light collected by the pixel 204-2 2 , a third signal S representing the light collected by the pixel 204-3 3 and a fourth signal S representing the light collected by pixel 204-4 4 .

[0072] In this example, pixel 204 is Fig. 10A or Fig. 10B The filter arrangement shown covers (i.e., a RBBG color filter arrangement in which the cut filter 602 covers the pixel 204-2 and the narrow-band infrared cut filter 1002 or the narrow-band infrared cut filter 1004 covers the remaining pixels 204-1, 204-3, and 204-4). For the purposes of this example, it will also be assumed that the wavelength of the fluorescence excitation illumination 404 is 803 nm and the wavelength of the fluorescence illumination induced by the fluorescence excitation illumination 404 is 830 nm.

[0073] Therefore, the signal received by the processing facility 1104 can be represented by the following equation:

[0074] S 1 =R+FI

[0075] S 2 =B

[0076] S 3 =B+FI

[0077] S 4 =G+FI

[0078] In these equations, R represents the red component of the light acquired by pixel 204-1, B represents the blue component of the light acquired by pixels 204-2 and 204-3, G represents the green component of the light acquired by pixel 204-4, and FI represents the fluorescent illumination acquired by pixels 204-1, 204-3, and 204-4. Note that due to the presence of cutoff filter 602 covering pixel 204-2 and narrow-band infrared cutoff filter 1002 or narrow-band infrared cutoff filter 1004 covering pixels 204-1, 204-3, and 204-4, signal S 1 To S 4 Neither include fluorescence excitation illumination 404.

[0079] The processing facility 1104 may process the signal S 1 To S 4 To selectively generate full-color images (no fluorescence) and full-color fluorescence images. For example, Fig.12 Various operations are shown that may be performed by processing facility 1104 to selectively generate full-color images and full-color fluorescence images.

[0080] At summation block 1202, processing facility 1104 first subtracts S2 from S3 to identify the fluorescent illumination component FI included in S3. Any suitable signal processing operation may be used to perform this subtraction.

[0081] To identify the red component R and the green component G, the processing facility 1104 obtains the red component R and the green component G from S at the summation block 1204. 1 Subtract FI and at summation block 1206 from S 4 Subtract FI. As shown, the output of summing block 1204 is a processed first signal representing only the red component R, and the output of summing block 1206 is a processed fourth signal representing only the green component G.

[0082] As shown, R, G, B, and FI are all input to the image generator 1208, which can be implemented by the processing facility 1104 in any suitable manner. The image generator 1208 can generate various types of images for display by the display device based on the received image mode selector command 1210. The image mode selector command 1210 can be provided by a user (e.g., a surgeon), by any component included in the computer-assisted surgery system, and / or by any other source, as can be used for a particular implementation. The image mode selector command 1210 can be received by the image generator 1208 when the visible light 118 and the fluorescent excitation illumination 404 are emitted simultaneously and / or at any other suitable time.

[0083] To illustrate, image mode selector commands 1210 may include a command to render a full-color image that does not include a fluorescent image. In response to receiving the command, image generator 1208 may generate a full-color image based on the R, G, and B signals using any suitable image processing technique.

[0084] Alternatively, if image mode selector commands 1210 include a command to render a full-color fluorescence image, image generator 1208 may generate a full-color fluorescence image based on the R, G, B, and IR signals. This may be done using any suitable image processing technique.

[0085] Fig.13 An exemplary method 1300 is shown which may be performed by the image processing system 1100. Fig.13Exemplary operations according to one embodiment are shown, but other embodiments may omit, add, reorder, and / or modify Fig.13 Any of the actions shown in .

[0086] In operation 1302, image processing system 1100 receives a first signal representing light collected by a first pixel included in a 2x2 pixel array, the first signal including data representing a combination of a red component of visible light and fluorescent illumination. Operation 1302 may be performed in any of the manners described herein.

[0087] In operation 1304, the image processing system 1100 receives a second signal representing light collected by a second pixel included in the 2x2 pixel array, the second signal representing a blue component of visible light. Operation 1304 may be performed in any of the manners described herein.

[0088] In operation 1306, the image processing system 1100 receives a third signal representing light collected by a third pixel included in the 2x2 pixel array, the third signal representing a combination of a blue component and fluorescent illumination. Operation 1306 may be performed in any of the manners described herein.

[0089] In operation 1308, the image processing system 1100 receives a fourth signal representing light collected by a fourth pixel included in the 2x2 pixel array, the fourth signal representing a combination of a green component of visible light and fluorescent illumination. Operation 1308 may be performed in any of the manners described herein.

[0090] In operation 1310, the image processing system 1100 subtracts the second signal from the third signal to identify a fluorescent illumination component of the third signal that represents fluorescent illumination. Operation 1310 may be performed in any of the manners described herein.

[0091] In operation 1312, the image processing system 1100 subtracts the fluorescent illumination component from the first signal to generate a processed first signal representing only the red component. Operation 1312 may be performed in any of the manners described herein.

[0092] In operation 1314, the image processing system 1100 subtracts the fluorescent illumination component from the fourth signal to generate a processed fourth signal representing only the green component. Operation 1314 may be performed in any of the manners described herein.

[0093] The systems and methods described herein can be used in conjunction with a computer-assisted surgery system for performing surgery on a patient. For example, the medical imaging systems 100 and 400 can be included in a computer-assisted surgery system.

[0094] Fig.14An exemplary computer-assisted surgery system 1400 ("surgical system 1400") is shown. As shown, surgical system 1400 may include a manipulation system 1402, a user control system 1404, and an auxiliary system 1406 that are communicatively coupled to one another. A surgical team may utilize surgical system 1400 to perform computer-assisted surgery on a patient 1408. As shown, the surgical team may include a surgeon 1410-1, an assistant 1410-2, a nurse 1410-3, and an anesthesiologist 1410-4, all of whom are collectively referred to as "surgical team members 1410." During the surgical procedure, additional or optional surgical team members may be present as may be useful for a particular implementation.

[0095] Although Fig.14 A minimally invasive surgical procedure is shown being performed, but it should be understood that the surgical system 1400 can be similarly used to perform open surgical procedures or other types of surgical procedures that can similarly benefit from the accuracy and convenience of the surgical system 1400. Additionally, it should be understood that the surgical procedures that can employ the surgical system 1400 throughout the procedure can include not only Fig.14 The surgical stage of a surgical procedure shown in , but may also include preoperative, postoperative and / or other suitable stages of a surgical procedure. A surgical procedure may include any procedure in which manual and / or instrumental techniques are used on a patient to study or treat a physical condition of the patient.

[0096] like Fig.14 As shown in , the manipulation system 1402 may include a plurality of manipulation arms 1412 (e.g., manipulation arms 1412-1 to 1412-4) to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction functionality), medical tool, imaging device (e.g., an endoscope), sensing instrument (e.g., a force sensing surgical instrument), diagnostic instrument, etc. that may be used to perform computer-assisted surgery on a patient 1408 (e.g., by being at least partially inserted into the patient 1408 and manipulated to perform computer-assisted surgery on the patient 1408). Although the manipulation system 1402 is depicted and described herein as including four manipulation arms 1412, it should be appreciated that the manipulation system 1402 may include only a single manipulation arm 1412 or any other number of manipulation arms, as may be useful for a particular implementation.

[0097] Manipulator arm 1412 and / or a surgical instrument attached to manipulator arm 1412 may include one or more of a displacement transducer, an orientation sensor, and / or a position sensor for generating raw (i.e., uncorrected) kinematic information. One or more components in surgical system 1400 may be configured to use kinematic information to track (e.g., determine its position) and / or control a surgical instrument.

[0098] The user control system 1404 can be configured to facilitate the surgeon 1410-1's control of the manipulator arm 1412 and the surgical instruments attached to the manipulator arm 1412. For example, the surgeon 1410-1 can interact with the user control system 1404 to remotely move or manipulate the manipulator arm 1412 and the surgical instruments. To this end, the user control system 1404 can provide the surgeon 1410-1 with an image (e.g., a high-definition 3D image) of a surgical area associated with the patient 1408 acquired by an imaging system (e.g., any of the medical imaging systems described herein). In some instances, the user control system 1404 can include a stereoscopic viewer with two displays, wherein the surgeon 1410-1 can view a stereoscopic image of the surgical area associated with the patient 1408 and generated by the stereoscopic imaging system. The surgeon 1410-1 can utilize the image to perform one or more surgeries using one or more surgical instruments attached to the manipulator arm 1412.

[0099] To facilitate control of surgical instruments, user control system 1404 may include a set of master controls. These master controls may be manipulated by surgeon 1410-1 to control the movement of surgical instruments (e.g., by utilizing robotics and / or teleoperation techniques). The master controls may be configured to detect various hand, wrist, and finger movements performed by surgeon 1410-1. In this manner, surgeon 1410-1 may intuitively perform surgery using one or more surgical instruments.

[0100] The auxiliary system 1406 may include one or more computing devices configured to perform the primary processing operations of the surgical system 1400. In such a configuration, the one or more computing devices included in the auxiliary system 1406 may control and / or coordinate operations performed by various other components of the surgical system 1400 (e.g., the manipulator system 1402 and the user control system 1404). For example, a computing device included in the user control system 1404 may send instructions to the manipulator system 1402 by way of one or more computing devices included in the auxiliary system 1406. As another example, the auxiliary system 1406 may receive and process image data from the manipulator system 1402 representing an image captured by an imaging device attached to one of the manipulator arms 1412.

[0101] In some instances, the assistance system 1406 can be configured to present visual content to surgical team members 1410 who may not be able to access the images provided to the surgeon 1410-1 at the user control system 1404. To this end, the assistance system 1406 can include a display monitor 1414 that is configured to display one or more user interfaces such as an image of the surgical area (e.g., a 2D image), information associated with the patient 1408 and / or the surgical procedure, and / or any other visual content as may be useful for a particular implementation. For example, the display monitor 1414 can display an image of the surgical area and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 1414 is implemented via a touch screen display, and the surgical team members 1410 can interact with the touch screen display (e.g., by way of touch gestures) to provide user input to the surgical system 1400.

[0102] The operating system 1402, the user control system 1404, and the auxiliary system 1406 may be communicatively coupled to one another in any suitable manner. Fig.14 As shown in , the operating system 1402, the user control system 1404, and the auxiliary system 1406 can be communicatively coupled via a control line 1416, which can represent any wired or wireless communication link as may be used for a particular implementation. To this end, the operating system 1402, the user control system 1404, and the auxiliary system 1406 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0103] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided according to the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and / or computing device to perform one or more operations, including one or more operations described herein. Any of a variety of known computer-readable media may be used to store and / or transmit such instructions.

[0104] As referred to herein, non-transitory computer-readable media may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0105] Fig.15 An exemplary computing device 1500 is shown, which may be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein may be implemented by computing device 1500.

[0106] like Fig.15 As shown, computing device 1500 may include a communication interface 1502, a processor 1504, a storage device 1506, and an input / output ("I / O") module 1508, which are communicatively connected to each other via a communication infrastructure 1510. Fig.15 An exemplary computing device 1500 is shown in FIG. Fig.15 The components shown in are not intended to be limiting. Additional or optional components may be used in other embodiments. Fig.15 Components of computing device 1500 are shown in FIG.

[0107] Communication interface 1502 may be configured to communicate with one or more computing devices. Examples of communication interface 1502 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0108] The processor 1504 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1504 may operate by executing computer-executable instructions 1512 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1506.

[0109] Storage device 1506 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1506 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 1506. For example, data representing computer-executable instructions 1512 configured to direct processor 1504 to perform any of the operations described herein may be stored within storage device 1506. In some instances, the data may be arranged in one or more databases residing within storage device 1506.

[0110] The I / O module 1508 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1508 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1508 may include hardware and / or software for obtaining user input, including, but not limited to, a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0111] The I / O module 1508 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1508 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content, as may be useful for a particular implementation.

[0112] In the previous description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and that additional embodiments may be implemented without departing from the scope of the invention set forth in the claims. For example, certain features of one embodiment described herein may be combined with or replaced with features of another embodiment described herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. An image processing system, comprising: a memory for storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to: receiving a first signal representative of light collected by a first pixel included in the 2×2 pixel array, the first signal comprising data representative of a combination of a first color component of visible light and fluorescent illumination, receiving a second signal representing light collected by a second pixel included in the 2×2 pixel array, the second signal representing a second color component of the visible light, receiving a third signal representative of light collected by a third pixel included in the 2×2 pixel array, the third signal representative of a combination of the second color component and the fluorescent illumination, receiving a fourth signal representative of light collected by a fourth pixel included in the 2×2 pixel array, the fourth signal representative of a combination of the third color component of the visible light and the fluorescent illumination, subtracting the second signal from the third signal to identify a fluorescent illumination component of the third signal representative of the fluorescent illumination, subtracting the fluorescent illumination component from the first signal to generate a processed first signal representing only the first color component, and The fluorescent illumination component is subtracted from the fourth signal to generate a processed fourth signal representing only the third color component.

2. The image processing system of claim 1 , wherein the processor is further configured to execute the instructions to: receiving a command to render a full-color image excluding a fluorescent image; and In response to the command, the full-color image is generated based on the processed first signal, the second signal, and the processed fourth signal.

3. The image processing system according to claim 1, wherein the image processing system is further configured as follows: receiving a command to render a full-color fluorescent image; and In response to the command, the full-color fluorescence image is generated based on the processed first signal, the second signal, the processed fourth signal, and the fluorescence illumination component.

4. The image processing system according to claim 1, wherein: The first color component includes a red component of the visible light; The second color component includes a blue component of the visible light; and The third color component includes a green component of the visible light.

5. The image processing system according to claim 1, wherein: The first color component includes a red component of the visible light; The second color component includes a green component of the visible light; and The third color component includes a blue component of the visible light.

6. The image processing system according to claim 1, wherein: the first pixel being covered by a first color filter, the first color filter being configured to allow the first pixel to collect the first color component of the visible light and to prevent the first pixel from collecting the second color component and the third color component of the visible light; The second pixel and the third pixel are covered by a second color filter, and the second color filter is configured to allow the second pixel and the third pixel to each collect the second color component of the visible light and prevent the second pixel and the third pixel from each collecting the first color component and the third color component of the visible light; the fourth pixel being covered by a third color filter, the third color filter being configured to allow the fourth pixel to collect the third color component of the visible light and to prevent the fourth pixel from collecting the first color component and the second color component of the visible light; and A pixel-level broadband infrared cutoff filter covers the second pixel and is configured to prevent the second pixel from collecting infrared light having a wavelength included in the first wavelength range.

7. The image processing system of claim 6, wherein a narrow-band infrared cutoff filter covers the first pixel, the third pixel, and the fourth pixel and is configured to prevent the first pixel, the third pixel, and the fourth pixel from collecting infrared light having a wavelength included in a second wavelength range, wherein the second wavelength range is included in the first wavelength range and is narrower than the first wavelength range.

8. The image processing system of claim 7, wherein the first pixel, the third pixel, and the fourth pixel are not covered with a pixel-level broadband infrared cutoff filter configured to prevent the first pixel, the third pixel, and the fourth pixel from collecting infrared light having a wavelength included in the first wavelength range.

9. The image processing system according to claim 7, wherein the narrow-band infrared cut-off filter comprises: a first pixel-level narrow-band infrared cutoff filter covering the first pixel; a second pixel-level narrow-band infrared cut-off filter covering the third pixel; and A third pixel-level narrow-band infrared cut-off filter covering the fourth pixel.

10. A method comprising: receiving a first signal representative of light collected by a first pixel included in the 2×2 pixel array, the first signal comprising data representative of a combination of a first color component of visible light and fluorescent illumination, receiving a second signal representing light collected by a second pixel included in the 2×2 pixel array, the second signal representing a second color component of the visible light, receiving a third signal representative of light collected by a third pixel included in the 2×2 pixel array, the third signal representative of a combination of the second color component and the fluorescent illumination, receiving a fourth signal representative of light collected by a fourth pixel included in the 2×2 pixel array, the fourth signal representative of a combination of the third color component of the visible light and the fluorescent illumination, subtracting the second signal from the third signal to identify a fluorescent illumination component of the third signal representative of the fluorescent illumination, subtracting the fluorescent illumination component from the first signal to generate a processed first signal representing only the first color component, and The fluorescent illumination component is subtracted from the fourth signal to generate a processed fourth signal representing only the third color component.

11. The method according to claim 10, further comprising: receiving a command to render a full-color image excluding a fluorescent image; and In response to the command, the full-color image is generated based on the processed first signal, the second signal, and the processed fourth signal.

12. The method according to claim 10, further comprising: receiving a command to render a full-color fluorescence image; and In response to the command, the full-color fluorescence image is generated based on the processed first signal, the second signal, the processed fourth signal, and the fluorescence illumination component.

13. The method of claim 10, wherein: The first color component includes a red component of the visible light; The second color component includes a blue component of the visible light; and The third color component includes a green component of the visible light.

14. The method of claim 10, wherein: The first color component includes a red component of the visible light; The second color component includes a green component of the visible light; and The third color component includes a blue component of the visible light.

15. A system comprising: An image sensor including a first pixel, a second pixel, a third pixel, and a fourth pixel; as well as a pixel-level broadband infrared cutoff filter covering the second pixel and configured to prevent the second pixel from collecting infrared light having a wavelength included in the first wavelength range; Wherein the first pixel, the third pixel and the fourth pixel are not covered by a pixel-level broadband infrared cutoff filter configured to prevent the first pixel, the third pixel and the fourth pixel from collecting infrared light having a wavelength included in the first wavelength range.

Images