Single-chip camera for multispectral imaging and use method thereof

Abstract

The invention relates to a single-chip camera for multispectral imaging and a use method thereof. Methods and systems are provided for implementing multiple imaging modes through a single camera. The camera may include a single chip system capable of measuring light of different wavelengths on different regions of the same image sensor to generate different images. A light source in the system may adjust the output spectrum by using multiple luminophors and filters, and may generate white light images, fluorescence images, perfusion images, fluorescein images, etc., using measurements from different light channels of different regions of the image sensor.

Description

Background Art

[0001] During the course of a surgical operation or surgical procedure, one or more images of the surgical site may be captured. Imaging may enable a surgeon to view the surgical site and make decisions based on the imaging. For example, a surgeon may use an image or real-time feed of the surgical site to manipulate surgical tools or make visual diagnoses, perform biopsies, etc. During a surgical operation or surgical procedure, different imaging modalities may be used for different purposes, with each mode providing a different view or different set of information about the surgical site.

[0002] A camera, typically connected to an imaging scope (such as an endoscope or exoscope), can provide imaging modes such as white light imaging and fluorescence imaging, but is limited in the number of different imaging modes that can be provided simultaneously. In addition, some such imaging modes require complex time synchronization between the camera and the corresponding light source to accurately capture the image information required for a given mode, which will generally result in a reduction in the effective frame rate for a given mode. For example, common fluorescence imaging (FI) techniques require the light source to be gated between white light illumination and fluorescence excitation light illumination. Each frame displayed combines the first collected white light frame with the subsequently collected FI frame (after appropriate processing) to create a superposition of the two images. This process necessarily reduces the possible display frame rate by at least 50%.

[0003] The camera may also include multiple complex or specialized image sensor combinations; however, these complex sensor systems result in corresponding cost increases. For example, some FI technologies may require the camera to employ two independent image sensors in order to generate two independent images superimposed by an image processor. In such an example, one of the image sensors may require infrared detection capabilities. Therefore, the cost of the camera increases because an additional dedicated sensor is required to generate the FI image. Additional optics may be required to further direct light to the second sensor, which may also increase the costs associated with the design and manufacture of such multi-chip FI-capable cameras. Summary of the invention

[0004] The shortcomings of the systems discussed above can be addressed by the systems and methods disclosed herein. By providing a light source having multiple individual light emitters and one or more possible optical filters, the light source can output light of different wavelengths corresponding to the requirements of the current imaging mode. In addition, by providing a single-chip camera discussed herein, different imaging modes can be achieved. The camera includes a single image sensor chip that receives separated transmitted light into different separated areas on a single image sensor, so that the light received from the surgical scene can be processed according to the selected imaging mode without using different cameras. The imaging sensor may include a color filter array (CFA), such as a Bayer array, which is capable of capturing data in various spectral bands (e.g., wavelength ranges) of light. The various color channels of the image sensor can be processed to generate images in various modes.

[0005] The camera may beneficially enable a physician to alternate between different imaging modes during the course of a surgery or surgical procedure to capture different images of the surgical scene. For example, a physician may be able to start in white light mode and capture white light images of the surgical scene, and then switch the imaging system to oxygenation mode to generate perfusion images to examine how blood is flowing in the surgical site without having to switch cameras. In addition, image processing may use this data from the image sensor to produce a false color overlay of a fluorescence image on a visible light image, which is beneficial for performing various surgeries and diagnostics. The use of a single camera may also be beneficial for reducing the cost and time required to perform a surgery or surgical procedure.

[0006] According to an embodiment of the present disclosure, a camera enables various imaging modes to be generated from a single image sensor, which can greatly reduce the cost of the camera, which is superior to more conventional multi-chip systems. For example, a camera can be manufactured to include only a single image sensor, while omitting additional image sensors. Eliminating the need for multiple image sensors can reduce the overall cost associated with the manufacturing process in manufacturing the camera, as well as the price of the camera to the consumer. It should be noted that the camera can be manufactured with multiple image sensors, but only a single image sensor is used to capture the various imaging modes discussed herein. In such embodiments, additional image sensors can be used to perform other tasks, such as providing additional image data that can be used to generate high dynamic range (HDR) or extended depth of field images for display. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A block diagram of a system according to at least one exemplary embodiment is shown;

[0008] Figure 2 shows a light source according to at least one exemplary embodiment;

[0009] Figure 3AA single-chip imaging system is shown according to at least one exemplary embodiment;

[0010] Figure 3B An alternative single-chip imaging system is shown according to at least one example embodiment; and

[0011] Figure 4 A method according to at least one exemplary embodiment is shown.

[0012] Figure 5A Three spectral channel sensitivities and associated six spectral bands are shown according to at least one exemplary embodiment;

[0013] Figure 5B shows three spectral channel sensitivities of a first region of a single chip according to at least one exemplary embodiment;

[0014] Figure 5C shows three spectral channel sensitivities for a first region of at least one exemplary embodiment, wherein the optical system includes a filter for blocking excitation light between 700 and 800 nm; and

[0015] Figure 5D Three spectral channel sensitivities of a second region of a single chip according to at least one example embodiment are shown. DETAILED DESCRIPTION

[0016] The exemplary systems and methods of the present disclosure will be described in conjunction with imaging. However, in order to avoid unnecessarily obscuring the present disclosure, many known structures and devices are omitted from the description. This omission should not be construed as a limitation on the scope of the claimed disclosure. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure can be implemented in a variety of ways, beyond the specific details set forth herein.

[0017] First go to Figure 1 , various aspects of system 100 are shown according to at least one exemplary embodiment. System 100 includes imaging device 104, network 140, user interface 144, display 148, network interface 152. In some embodiments, system 100 may include additional or alternative components, and Figure 1 The components shown in are by no means limiting. For example, the system 100 may include additional or alternative imaging devices, additional surgical tools (e.g., surgical drills, endoscopes, etc.), combinations thereof, etc. As another example, the controller 108 disposed in the imaging device 104 may alternatively be disposed external to the imaging device 104 and may communicate with the imaging device 104 (or components thereof) wirelessly and / or via a wired connection. Such an external controller may be included in a camera control unit (CCU).

[0018] The imaging device 104 may capture one or more images by illuminating the surgical scene using, for example, a light source 112. The light source 112 may provide continuous illumination that reflects, scatters, and / or causes fluorescence to be emitted from elements of the surgical scene, and such light is captured by the camera 116. Typically there will be an intermediate device between the camera and the surgical scene, such as the optical elements of an endoscope, borescope (for industrial applications), or exoscope, which captures light from the scene as well as conditions and relays the light to the camera 116. It should be noted that light collected from the imaging scene will be referred to throughout as "reflected" or "reflected" from the surgical scene, however, it should be understood that this "reflected" light from the image scene includes all light collected by the imaging optics and passed to the camera 116. Thus, this "reflection" includes light reflected from elements of the scene, light scattered from surfaces, fluorescence emitted from fluorophores, light directly from other illumination sources, etc.

[0019] The camera 116 may be a device including a lens assembly 132 and an imaging assembly 136 that measures received light reflected from a surgical scene. The imaging assembly 136 includes a prism assembly 137 and one or more image sensors. The lens assembly 132 may include a plurality of optical components that condition the light received from the surgical scene and / or direct the light into the prism assembly 137. The one or more image sensors of the imaging assembly 136 measure the captured light, wherein the controller 108 uses these measurements to generate one or more images of the surgical scene. While some embodiments discussed herein refer to a single image sensor, it should be understood that other embodiments may include a plurality of additional image sensors dedicated to other imaging modes not discussed herein, and the number of image sensors present or available is in no way limited.

[0020] The processor 120 may provide processing functionality and may correspond to one or more computer processing devices. For example, the processor 120 may be configured as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any other type of integrated circuit (IC) chip, an IC chip set, a microcontroller, a microcontroller set, a GPU, etc. As another example, the processor 120 may be configured as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), and / or multiple microprocessors configured to execute instructions or algorithms 128 and / or data stored in the memory 124. The processor 120 enables various functions of the system 100 when executing the instructions or algorithms 128 and / or data stored in the memory 124.

[0021] The memory 124 may be or may include a computer-readable medium containing instructions executable by the controller 108 and / or the processor 120. The memory 124 may include any type of computer memory device and may be volatile or non-volatile in nature. In some embodiments, the memory 124 may include a plurality of different memory devices. Non-limiting examples of the memory 124 include random access memory (RAM), read-only memory (ROM), flash memory, electronically erasable programmable ROM (EEPROM), dynamic RAM (DRAM), etc. The memory 124 may include instructions that enable the controller 108 to control various elements of the system 100 and, for example, store data in and retrieve information from the database 156. The memory 124 may be local to the imaging device 104 (e.g., integrated with the imaging device) or separate from the imaging device 104.

[0022] The instructions 128 include computer-readable software executable by the controller 108 and / or the processor 120 that causes the controller 108 and / or the processor 120 to perform one or more functions. When processed by the processor 120, the instructions 128 may cause the processor 120 to execute one or more algorithms for image processing, for controlling one or more components of the system 100, combinations thereof, etc. As an example, the instructions 128 may cause the processor to perform one or more image processing techniques (e.g., edge detection techniques, interpolation, demosaicing algorithms, Bayer filter pattern algorithms, image overlays, etc.) to convert measurements from one or more image sensors into one or more images for storage and / or display.

[0023] The user interface 144 includes hardware and / or software that implements user input to the system 100 and / or any one or more of its components. The user interface 144 may include a keyboard, a mouse, a touch-sensitive pad, a touch-sensitive button, a mechanical button, a switch and / or other control elements for providing user input to the system 100, so that the user can control certain functions of the system 100 (e.g., operating the lighting and / or imaging functions of the imaging device 104, presenting the processed video to the display 148, etc.). The user interface 144 may include buttons, switches or other control devices arranged on the imaging device 104 itself, which are independent of or supplementary to the user interface controls not arranged in the imaging device 104. As an illustrative example only, the imaging device 104 and / or the display 148 may have input buttons and switches, and in addition, the keyboard or mouse may be directly connected to the processor 120 (in an embodiment in which the processor 120 is arranged outside the imaging device 104). All of these together constitute the user interface 144.

[0024] Display 148 may be or may include a liquid crystal display (LCD), a light emitting diode (LED) display, a high definition (HD) display, a 4K display, etc. Display 148 may be a stand-alone display or a display integrated as part of another device (such as a smartphone, a laptop, a tablet, a headset or a head-mounted device, etc.). In one embodiment, display 148 may be a monitor or other viewing equipment disposed in an operating room so that a video feed captured from a surgical operation or surgical procedure may be presented to display 148 for viewing by a physician. In some embodiments, display 148 may include multiple displays, for example, depending on the system design.

[0025] The network interface 152 may enable one or more components of the system 100 to communicate with each other or with components external to the system 100 by wire and / or wireless. These communication interfaces that allow the components of the system 100 to communicate using the network interface 152 include wired and / or wireless communication interfaces for exchanging data and control signals with each other. Examples of wired communication interfaces / connections include Ethernet connections, HDMI connections, connections that comply with the PCI / PCIe standards and the SATA standards, etc. Examples of wireless interfaces / connections include Wi-Fi connections, LTE connections, Bluetooth connections, and the like. connection, NFC connection, etc.

[0026] Database 156 includes the same or similar structure as the above-described memory 124. In at least one exemplary embodiment, database 156 is included in a remote server and stores video data captured during a surgical procedure or surgical procedure (e.g., a camera on an endoscope captures a real-time feed during an endoscopy).

[0027] although Figure 1 The various elements in the system 100 are shown separated from each other, but it should be understood that some or all of the elements can be integrated with each other if desired. For example, a single desktop or laptop computer or CCU may include a processor 120, a memory 124, a user interface 144, and an optional display 148. It should also be understood that each element in the system 100 includes one or more communication interfaces that enable communication with other elements in the system 100 through, for example, a network interface 152. Another example of a preferred embodiment of the system 100 includes an endoscope combined with a camera 116 (or detachably connected to the camera), the endoscope having a built-in user interface 144, the built-in user interface is connected to an external camera control unit (CCU), the CCU includes a controller 108, a memory 124, a processor 120, a network interface 152 and a user interface 144, and the CCU is also connected so that it can output image data to a display 148.

[0028] Figure 2Aspects of a light source 112 according to at least one exemplary embodiment of the present disclosure are shown. The light source 112 may be or may include a lighting device that includes a first light emitter 204, a second light emitter 208, a third light emitter 212, a fourth light emitter 216, and a fifth light emitter 220. Each of the light emitters may be capable of generating and / or emitting light at a set wavelength or wavelength range. In one embodiment, the first light emitter 204 may generate green light (e.g., light with a central wavelength of approximately 550 nanometers (nm)), or in some embodiments where a white light source is used, may generate white light (e.g., light with a wavelength between 400nm and 700nm); the second light emitter 208 may generate blue light (e.g., light with a central wavelength of approximately 450nm); the third light emitter 212 may generate infrared (IR) light with a wavelength of approximately 940nm; the fourth light emitter 216 may generate infrared light with a wavelength of approximately 770nm; and the fifth light emitter 220 may generate red light with a central wavelength of approximately 630nm. The light generated by the light emitters may pass through a series of filters before being output from the light source 112, so that the final light output by the light source 112 includes a set of wavelengths generated by the separate light emitters. For example, the first filter 224 may reflect red light and transmit all other light, so that the red light generated by the fifth light emitter 220 is directed toward the third filter 232. Similarly, the second filter 228 may reflect light with wavelengths below 500nm and transmit light with wavelengths above 500nm. Thus, the second filter 228 may direct the blue light generated by the second light emitter 208 to the third filter 232. The filtering parameters of the first filter 224 and the second filter 228 may enable the green light generated by the first light emitter 204 to pass through the first filter 224 and the second filter 228 as the green light travels toward the third filter 232. This configuration of filters and light emitters allows for precise selection of spectral bands (also referred to herein as a set or range of wavelengths) in the resulting combined output light 244. The spectral values ​​(eg, wavelengths of light) of the illuminants and filters of the light sources may be adjusted to correspond to a desired configuration of the imaging assembly 136 of the camera 116, details of various embodiments of which are described below.

[0029] As used herein and unless otherwise specified, a list of wavelengths of light includes approximate wavelength values. For example, a green light having a wavelength of 550 nm may also include percentage variations thereof (e.g., values ​​1% lower and higher than 550 nm, values ​​2% lower and higher than 550 nm, values ​​10% lower and higher than 550 nm, values ​​25% lower and higher than 550 nm, etc.). In addition, the term "about" includes percentage variations in the wavelength of light. For example, "about 550 nm" may cover all wavelengths between 544.5 and 555.5 nm. In some embodiments, the wavelength of light listed may be a wavelength at the center of a wider spectrum or range of wavelengths constituting the light. For example, a green light having a wavelength of 550 nm may indicate that the green light includes a wide spectrum (e.g., a range) of wavelengths centered at 550 nm (e.g., between 485 nm and 615 nm). It is understood that such a wide spectrum of wavelengths is not limited to green light, and other illumination sources discussed herein may similarly produce light having a wide spectrum of wavelengths. In some preferred embodiments, the light generated by different light sources may have overlapping spectra (e.g., overlapping wavelength ranges). In other words, the fifth light emitter 220 and the first light emitter 204 may both generate light including overlapping wavelength ranges (e.g., overlapping wavelengths between 560nm and 590nm, overlapping wavelengths between 550nm and 600nm, etc.), and the first light emitter 204 and the second light emitter 208 may both generate light including overlapping wavelength ranges (e.g., overlapping wavelengths between 480nm and 550nm, overlapping wavelengths between 450nm and 580nm, etc.). However, it should be understood that these exemplary ranges are by no means limiting, and wider or narrower ranges of spectral overlap of light generated by different light emitters are possible.

[0030] The third filter 232 is operable to separate visible light emitters from infrared light emitters by allowing light with wavelengths below 650 nm to pass therethrough, but reflecting light with wavelengths above 650 nm. Thus, visible light generated and emitted by the first light emitter 204, the second light emitter 208, and / or the fifth light emitter 220 passes through the third filter 232, while any infrared light generated by the first light emitter 204, the second light emitter 208, and / or the fifth light emitter 220 is reflected. The placement of the third filter 232 ensures that infrared light output from the first light emitter 204, the second light emitter 208, and / or the fifth light emitter 220 is not an element of the combined light output 244. The fourth filter 236 can reflect light with wavelengths between about 725 and 800 nm, while transmitting light with wavelengths shorter than 725 nm and greater than 800 nm. The fourth filter 236 directs infrared light generated by the fourth light emitter 216, which may have a wavelength of about 770 nm, to the exit of the light source 112. Similarly, the fifth filter 240 can reflect wavelengths of about 940 nm while allowing all other wavelengths to pass through it; the fifth filter 240 ensures that the infrared light generated by the third light emitter 212 (whose wavelength can be about 940 nm) is directed to the exit of the light source 112. The fourth filter 236 and the fifth filter 240 allow visible wavelengths to pass through them and exit the light source 112. Figure 2 As shown, the positioning of the first filter 224, the second filter 228, the third filter 232, the fourth filter 236, and the fifth filter 240 results in a combined light output 244 being output from the light source 112, wherein the combined light output 244 includes the wavelengths of light generated by each light emitter. In other words, based on the number of active light emitters, the combined light output 244 may include two, three, or more spectrally different light portions. As used herein, the term "spectrally different" refers to a different range, number, set, or content of wavelengths of light. In some cases, a first light portion may overlap with a second light portion in terms of spectral content, but the first light portion may be considered spectrally different when it includes spectral content (e.g., a set or range of wavelengths of light) that is not present in the second light portion (or vice versa).

[0031] In some embodiments, the light source 112 may be gated. In other words, the first light emitter 204, the second light emitter 208, the third light emitter 212, the fourth light emitter 216, and / or the fifth light emitter 220 may be modulated or duty cycled to periodically emit light such that the combined light output 244 periodically illuminates the surgical site. In other embodiments, the light source 112 may not be gated. In other words, the first light emitter 204, the second light emitter 208, the third light emitter 212, the fourth light emitter 216, and / or the fifth light emitter 220 may continuously generate light such that the combined light output 244 illuminates the surgical site as long as the light source 112 remains on.

[0032] FIG. 3A to FIG. 3B 1 shows aspects of a camera 116 according to at least one exemplary embodiment of the present disclosure. The camera 116 includes a lens assembly 132 and an imaging assembly 136 including a prism assembly 137. The lens assembly 132 may include a collection of mirrors, lenses, filters, polarizers, and / or windows that can condition received light and direct the light into the prism assembly 137. FIG. 3A to FIG. 3B As shown, the lens assembly 132 includes a plurality of lenses 312A-312N and a first filter 316, which can direct light 302 to the prism assembly 137. In some embodiments, the light 302 can be light reflected from the surgical scene and captured by the optical device of the attached endoscope or the optical device of the exoscope, and is relayed or transmitted to the camera 116. The first filter 316 can filter out one or more wavelengths of light captured by the camera 116. The first filter 316 can be a replaceable, selectable or tunable filter, so that its characteristics can be selected depending on the requirements of the desired mode. For example, in one embodiment, the first filter 316 can be a dichroic filter that passes light within a first wavelength range but rejects light within a second wavelength range. In another embodiment, the first filter 316 can be a different dichroic filter that passes light within a third wavelength range and rejects light from a fourth wavelength range. Various specific embodiments are discussed below. In one embodiment, the first filter 316 blocks wavelengths between 725nm and 800nm. In some embodiments, the first filter 316 can be removed from the lens assembly 132. The first filter 316 can be used in an imaging mode (such as an indocyanine green (ICG) or OTL-38 (OTL) fluorescence imaging mode) to block illumination wavelengths between 725 nm and 800 nm used to stimulate fluorescence. Although the wavelengths between 725 nm and 800 nm are filtered out, the transmitted light may still contain additional spectrally different portions of light, or in other words, other wavelengths of light in addition to the wavelengths between 725 nm and 800 nm. For example, the transmitted light may include fluorescence wavelengths as well as visible light wavelengths, which can be separated separately by the prism assembly 137, as discussed further below.

[0033] In some embodiments, the lens assembly 132 may condition the light 302 so that the conditioned light 304 is passed into the prism assembly 137 and ultimately focused on the image sensor 332. The prism assembly 137 includes a first prism 320 that receives the conditioned light 304. Between the first prism 320 and the second prism 340 is a first dichroic beam splitter or spectral filter 324, which may be disposed on a surface of the first prism 320 or the second prism 340. The dichroic beam splitter 324 filters and / or reflects light of different wavelengths, thereby forming a break between different spectral bands. For example, the first dichroic beam splitter 324 may transmit light with a wavelength below 500nm and above 650nm, while reflecting light with a wavelength between 500nm and 650nm. As another example, the first dichroic beam splitter 324 may reflect light with a wavelength below 500nm and above 650nm, while transmitting light with a wavelength between 500nm and 650nm. Thus, the spectral band with wavelengths between 500 nm and 650 nm is transmitted from the first prism 320 as transmitted light 336, while the remaining spectral bands are retained in the first prism 320 as reflected light 328. It should be noted that while a dichroic beam splitter is utilized in the preferred embodiment, this is not limiting, and other beam splitting configurations are also contemplated. For example, the dichroic beam splitter 324 can be replaced with a half-silvered mirror that splits the beam in some other ratio (e.g., 50-50), and the beam splitter can then be followed by one or more filters to filter out the undesired wavelength bands of each split beam. This is of course less than ideal than the preferred embodiment because light intensity is unnecessarily lost, among other considerations.

[0034] The reflected light 328 may be bent or reflected within the first prism 320 until the reflected light 328 passes into the first region 356 of the image sensor 332. The image sensor 332 may receive the reflected light 328, and a photosensitive element (e.g., a photodiode, a pixel) within the first region 356 of the image sensor 332 may generate a corresponding electrical signal. In some embodiments, the electrical signal may be passed to one or more other components of the system 100 (such as the controller 108) and further used to generate one or more images. Although Figure 3A The image sensor 332 is shown positioned generally parallel to the modulated light 304 (eg, the image sensor 332 is in a horizontal position), but the image sensor 332 may also be positioned generally orthogonal to the modulated light 304 (eg, the image sensor 332 is in a vertical position), as shown in FIG. Figure 3B Notwithstanding the foregoing, the orientation of the image sensor 332 relative to any one or more components of the camera head 116 is in no way limited, and the image sensor 332 may be positioned differently depending on, for example, the type of endoscope or exoscope being used, the type of surgical procedure being performed, manufacturer specifications, design requirements, combinations thereof, and the like.

[0035] The first region 356 of the image sensor 332 may have a color filter array 334 capable of receiving one or more wavelengths of light, such as visible light (e.g., red, blue, and green), infrared light, combinations thereof, and the like. In some embodiments, the CFA 334 may be positioned so that the photodiodes arranged in the array can capture light of different wavelengths. For example, the CFA 334 may be a Bayer filter arranged in a Bayer pattern (e.g., a filter pattern having half green, one quarter red, and one quarter blue color filters). In some embodiments, the image sensor 332 may include four spectrally distinct spectral channels: a red light channel, a blue light channel, a green light channel, and an infrared light channel. Each channel may be used to determine a pixel value when reconstructing one or more images of the surgical site. The channels of each image sensor region are defined at least in part by the CFA 334 positioned between the prism assembly 137 and the image sensor 332. In some embodiments, the image sensor 332 may have three spectrally distinct spectral channels: a blue light channel, a green light channel, and a channel capable of detecting a combination of red light and infrared light. In some embodiments, image sensor 332 may be used to measure light having wavelengths reflected by first dichroic beam splitter 324 (eg, light having wavelengths below 500 nm and above 650 nm).

[0036] CFA 334 may filter reflected light 328 into a plurality of spectral channels. Each spectral channel (also referred to herein as a color channel) may have a wavelength range to which the channel is sensitive, such that each spectral channel is different from the other spectral channels on CFA 334. For example, Figure 3A As shown, reflected light 328 may pass through CFA 334 and arrive at first region 356 of image sensor 332 in the form of three spectral channels: a red channel, a blue channel, and a green channel. It should be understood that while CFA 334 is shown as being separate from image sensor 332, this is for illustration purposes only and the space between CFA 334 and image sensor 332 may be minimal. Figure 3A and Figure 5A and Figure 5C , due to the red channel's sensitivity to such wavelengths, the red channel may see the range of wavelengths of light represented by trace / line 504. Similarly, the blue channel may see the range of wavelengths of light represented by trace / line 512, and the green channel may see the range of wavelengths of light represented by trace / line 508. The wavelengths seen by each color channel may be measured by first region 356 of image sensor 332 and used in image processing to generate an image, as is known in the art.

[0037] The transmitted light 336 may pass through the first dichroic beam splitter 324 and may enter the second prism 340 and the third prism 348. The second prism 340 and the third prism 348 may bend and / or reflect the transmitted light 336 through another optical conditioning element 352 and onto a second region 360 of the image sensor 332. The other optical conditioning element 352 may be, for example, a transparent optical element whose refractive index is selected so as to achieve co-focusing of the reflected light 328 and the transmitted light 336 onto the image sensor 332. In some preferred embodiments, the second region 360 may be similar to the first region 356, i.e., the entire image sensor 332 will include a single CFA, however this is not limiting. For example, in some embodiments, the second region 360 may include a CFA different from the CFA of the first region, capable of receiving one or more wavelengths of light, such as visible light, infrared light, combinations thereof, etc. The second region 360 may include filters arranged in a Bayer pattern, or another arrangement of filters, such that the second region 360 is capable of collecting four spectrally distinct spectral channels: a red channel, a blue channel, a green channel, and other such arrangements of an infrared channel. In other embodiments, the second region 360 may include three spectrally distinct channels: a blue channel, a green channel, and a channel capable of detecting a combination of red and infrared light. In one embodiment, the second region 360 of the image sensor 332 may be configured to detect light having a wavelength between 500 nm and 650 nm.

[0038] As described above, each spectral or color channel of CFA 334 may have a wavelength range to which the channel is sensitive, such as a blue channel, a green channel, and a red channel. It should be understood that while first region 356 and second region 360 may have three or more different spectral channels that are identical to one another (such as when image sensor 332 has a single CFA 334), the spectral content (e.g., wavelength range) of light transmitted onto first region 356 may be different than the spectral content (e.g., wavelength range) of light transmitted onto second region 360 due to, for example, filtering performed by components of imaging assembly 136 (including dichroic filter 324). For example, referring to Figure 3A and Figure 5A and Figure 5D, the red channel of the second region 360 of the image sensor 332 may see the wavelength range of light represented by trace / line 516 due to the sensitivity of the red channel to such wavelengths. Similarly, the blue channel of the second region 360 of the image sensor 332 may see the wavelength range of light represented by trace / line 524, and the green channel of the second region 360 of the image sensor 332 may see the wavelength range of light represented by trace / line 520. Due to the differences in spectral content (e.g., different wavelength ranges of light) and the presence of three different spectral channels on the CFA 334, the image sensor 332 may be able to receive and process up to six different spectral bands. In other words, the first region 356 may receive at least three spectral bands from the color channels of the CFA 334, and the second region 360 may also receive spectral bands from the color channels of the CFA 334, but their spectral bands are different from the spectral bands seen by the first region 356. Because the spectral band received at the first region 356 may be different from the spectral band received at the second region 360 (e.g., because the wavelength of light incident on the CFA 334 in the first region 356 is different from the wavelength of light incident on the CFA in the second region 360), the image sensor 332 may receive up to six different spectral bands for image processing. This facilitates multiple imaging modes with a single image sensor.

[0039] In some embodiments, the first region 356 and the second region 360 of the image sensor 332 may each include three different spectral channels: a red channel, a blue channel, and a green channel. In such embodiments, the red channel on the first region 356 and / or the second region 360 may also be sensitive to infrared light. Thus, the first region 356 and / or the second region 360 of the image sensor 332 may be able to detect blue, green, and infrared spectral channels.

[0040] For purposes of illustration only, the following are examples of methods for performing various imaging modes using the camera 116 and other components of the system 100 .

[0041] In the "white light imaging" mode, the light source 112 and the camera 116 can use white light to image the surgical scene. The white light can be output from the first light emitter 204, or it can come from a combination of green light, blue light, and red light output from the first light emitter 204, the second light emitter 208, and the fifth light emitter 220, respectively. In such a case, the third light emitter 212 and the fourth light emitter 216 can be turned off or disabled so that the combined light output 244 contains only white light. In some embodiments, the first light emitter 204 can emit white light, while the second light emitter 208 and the fifth light emitter 220 also emit blue light and red light, respectively. In other words, all three of the first light emitter 204, the second light emitter 208, and the fifth light emitter 220 can emit light in the white light mode. Once light is emitted from the light source 112, the light can illuminate the surgical scene. Light reflected from elements of the scene, light scattered from surfaces, fluorescence emitted from fluorophores, light directly from other illumination sources, etc. can enter the camera 116, and more specifically enter the lens assembly 132. The lens assembly 132 may then condition the light and / or direct the light (eg, conditioned light 304 ) into the prism assembly 137 of the imaging assembly 136 .

[0042] In an embodiment where the "white light imaging" mode is enabled, the first region 356 of the image sensor 332 may capture light having a wavelength below 500 nm and light having a wavelength above 650 nm directed thereto by the beam splitter 324, while the second region 360 of the image sensor 332 may capture light having a wavelength between 500 nm and 650 nm. The processor 120 may process the measurements from the first region 356 and / or the second region 360 of the image sensor 332 to generate a true color image. In some embodiments, the processor 120 may process the color channels on the first region 356 and / or the second region 360 of the image sensor 332 using one or more algorithms (e.g., image reconstruction algorithms, bilinear interpolation, gradient direction interpolation algorithms, etc.) to generate a true color image. In one embodiment, the green and blue channels (e.g., spectral channels that detect green and blue light, respectively) of the first region 356 of the image sensor 332 and the red, green, and blue channels of the second region 360 of the image sensor 332 may be used to reconstruct a true color image.

[0043] As another illustrative example, the imaging device 104 can be operated in a simultaneous white light imaging mode with ICG / OTL fluorescence. In this mode, white light is generated in a manner similar to the white light mode described above, but in addition, the fourth illuminant 216 can be enabled so that infrared light is included in the combined light output 244 of the light source 112. The wavelength or wavelength range emitted by the fourth illuminant is selected to correspond to the excitation wavelength of the target fluorophore. The combined light output 244 can then illuminate the surgical scene, and light reflected from elements of the scene, light scattered from surfaces, fluorescence emitted from fluorophores, light directly from other illumination sources, etc. can be captured by the camera 116. In this case, ICG / OTL fluorescence may occur at infrared light wavelengths (e.g., above 800nm ​​wavelength) that are different from the infrared light generated by the light source 112 (e.g., about 770nm). However, the direct infrared light generated by the fourth illuminant 216 may be blocked by the first filter 316 before reaching the prism assembly 137. A true color image may be created using the same color channels of the first region 356 and the second region 360 of the image sensor 332 as described above in the white light imaging mode, while ICG / OTL fluorescence based on the infrared spectral band is detected by the infrared channel of the first region 356 of the image sensor 332. Channel measurements of the first region 356 and the second region 360 of the image sensor 332 may be sent to the processor 120, which may use one or more algorithms to generate an image of the surgical scene. The image of the surgical scene may include a white light image and an overlay image representing the collected fluorescence image. Typically, the combined image is represented as a false color image in which the fluorescence image is overlaid with the white light image, as is known in the art.

[0044] FIG. 5A to FIG. 5D Shows Figure 3A and Figure 3B 0048 ] Channel sensitivity of one or more exemplary embodiments of an imaging assembly 136 (and more generally a camera 116) having the above configuration is shown in , wherein the image sensor 332 is an RGB image sensor having a standard Bayer-type CFA 334, and the image sensor 332 includes a first region 356 and a second region 360. Figure 5A The full spectral sensitivity 500 of all channels represented on two regions of the image sensor is shown. The sensitivity of the blue sensor is represented by the curves indicated as 512 and 524. The sensitivity of the green sensor is represented by the curves indicated as 508 and 520. Finally, the sensitivity of the red sensor is represented by the curves indicated as 516 and 504. As previously discussed, the first dichroic beam splitter 324 in this example reflects light having wavelengths shorter than 500 nm and longer than 650 nm and directs this reflected light 328 to the first region 356 of the image sensor 332.

[0045] Figure 5BThe sensitivity 528 of the red, blue, and green channels of the first region 356 of the image sensor 332 relative to the reflected light 328 is shown. Trace 512 shows the sensitivity of the blue channel, trace 508 shows the sensitivity of the green channel, and trace 504 shows the sensitivity of the red channel. Of course, the region between 500 nm and 650 nm does not show sensitivity because light in this spectral band has been directed to the second region by the dichroic beam splitter 324. It should be noted that the sensitivity depicted by trace 504 for the red channel extends into the near infrared (NIR) spectral band and is therefore sensitive to NIR radiation, which is particularly useful for some fluorescence imaging applications (such as those using ICG) because its emission peak is in this range.

[0046] Figure 5C The sensitivity 532 of the three RGB channels of the first region 356 of the image sensor 332 is shown when the first filter 316 is an element of the lens assembly 132 of the camera 116 and blocks light representing, for example, the excitation range of ICG (i.e., light between 700 nm and 800 nm). Thus, when this excitation band is blocked, the sensitivity 504 of the red channel now shows a notch 536 because the wavelength range of the reflected light 328 will not include any light in this spectral band because it was previously blocked by the optical system. Thus, when operating in ICG mode, the only light detected by the red channel of the first region 356 of the image sensor 332 will be the NIR signal corresponding to the ICG emission light because the excitation light has been blocked by the first filter 316 and the light source 112 will not emit any light in the 650-700 nm range when operating in this mode. When operating in CY5 / CY5.5 mode, the red channel of the first region 356 of the image sensor is used to detect the emission of these fluorophores occurring in the 650-700 nm spectral band. In addition, when the imaging system is combined with a conventional white light source (rather than Figure 2 ), this embodiment can take advantage of sensitivity in this range when operating in a "white light imaging mode." In this case, the first region 356 of the image sensor will generate blue, green, and red signals, and the combined signals from the first and second regions of the image sensor can generate a fuller spectral image.

[0047] Figure 5DThe sensitivity 540 of the three RGB channels of the second region 360 of the image sensor 332 to the light 336 transmitted by the dichroic beam splitter 324 in the wavelength range 500-650 nm is shown. Here we can see the sensitivity 524 of the blue channel, the sensitivity 520 of the green channel and the sensitivity 516 of the red channel. Thus, we see six different spectral bands received by two different regions of the image sensor, in this case two blue spectral bands, two green spectral bands, one red spectral band and one spectral band that will operate in the red spectral band or the NIR spectral band, depending on the operating mode. In this exemplary embodiment, the image data thus received by the six different spectral bands can be used to generate white light, as well as provide a fluorescent image overlay.

[0048] As another illustrative example, the imaging device 104 may operate with oxygenation / perfusion using a simultaneous white light imaging mode. In this mode, the first light emitter 204, the second light emitter 208, and the fifth light emitter 220 may generate white light together, and the fifth light emitter 220 may generate a discrete red light spectrum centered at a wavelength of 630 nm. In addition, the third light emitter 212 may generate an infrared spectrum centered at a wavelength of about 940 nm. Since the fifth light emitter 220 generates a red light spectrum centered at 630 nm and the third filter 232 blocks light with wavelengths greater than 650 nm, the light in the combined light output 244 may not include light with wavelengths greater than 650 nm (except for 940 nm infrared light). The light entering the prism assembly 137 may be directed to the first region 356 and the second region 360 of the image sensor 332 in the manner discussed above, and a true color white image may be generated based on the green and blue light channels of the first region 356 of the image sensor 332 and the red, green, and blue light channels of the second region 360 of the image sensor 332. Perfusion is determined by comparing red light channel data from second region 360 of image sensor 332 to 940 nm signals captured by infrared light channel in first region 356 of image sensor 332 using one or more algorithms known in the art. Based on the comparison, a perfusion image may be generated.

[0049] In another illustrative example, imaging device 104 can operate in a simultaneous white light imaging mode using cyanine 5 (Cy5) or Cy5.5 fluorescence. In this mode, light source 112 can generate white light, wherein the red light spectrum is generated by a discrete red light source having a center wavelength of 630 nm. Due to the position of third filter 232, combined light output 244 may not have light with a wavelength greater than 650 nm. Light reflected from the surgical scene can then be directed to prism assembly 137 through lens assembly 132. A white light image can be generated based on the green and blue light channels of first region 356 of image sensor 332 and the red, blue, and green light channels of second region 360 of image sensor 332. A fluorescent image can be generated based on the red light channel of first region 356 of image sensor 332.

[0050] In yet another illustrative example, imaging device 104 may operate in a simultaneous grayscale imaging mode using fluorescein fluorescence. In this mode, light source 112 may output combined light 244 that includes blue light and does not include wavelengths longer than 500 nm. In other words, green light and red light may not be present in combined light 244. In some optional embodiments, combined light output 244 may also include wavelengths longer than 650 nm (e.g., near infrared or infrared light). Light reflected from the surgical scene may be received by camera 116, and a grayscale image may be created based on all color channels (e.g., blue, green, red, and infrared light channels) of first region 356 of image sensor 332. In addition, the green and blue channels of second region 360 of image sensor 332 may be used to generate a fluorescein fluorescence image.

[0051] Figure 4 A method 400 is shown in accordance with at least one exemplary embodiment of the present disclosure. For example, the method 400 can be used to select and generate one or more images for a selected image mode.

[0052] The method 400 starts and then proceeds to step 404, where an imaging mode is selected. The imaging mode may be a white light imaging mode, an ICG / OTL fluorescence mode, an oxygenation / perfusion mode, a Cy5 / Cy5.5 fluorescence mode, a fluorescein fluorescence mode, etc. The mode may be selected based on physician input (e.g., the physician selects the mode by pressing a virtual button on a screen presented on the display 148).

[0053] Then, the method 400 continues to step 408, where a light source including a plurality of light emitters generates illumination light to illuminate the surgical scene, wherein the activated light emitter is determined by the selected imaging mode. The light source may be similar or identical to the light source 112. The controller 108 may determine, based on the physician input and using the instructions and / or algorithm 128, that one or more light emitters should be enabled to generate the required illumination. For example, in the white light mode, the controller 108 may determine that the first light emitter 204, the second light emitter 208, and the fifth light emitter 220 should be enabled to generate red light, blue light, and green light, while also determining that the third light emitter 212 and the fourth light emitter 216 should not be enabled. In another example, in the oxygenation / perfusion mode, the controller 108 may determine that the first light emitter 204, the second light emitter 208, and the fifth light emitter 220 should be enabled, and the third light emitter 212 (which generates 940nm infrared light) should also be enabled. Once the light emitter is enabled or disabled, the controller 108 may cause the light emitter to emit light so that the light source 112 illuminates the surgical scene with the light required for the imaging mode.

[0054] Method 400 then proceeds to step 412, where light is reflected and / or fluoresced from the surgical scene and captured at a camera, such as camera 116. Method 400 then proceeds to step 416, where the reflected and / or fluoresced light passes through one or more optical components, such as lens assembly 132, and enters prism assembly 137 of imaging assembly 136 (e.g., in the form of conditioned light 304). The light may be separated by one or more beam splitters, such as beam splitting prisms, which may include selective spectral filters (e.g., filters designed to filter out a range of wavelengths of light), where the transmitted and reflected light is passed to different areas of the image sensor.

[0055] The method 400 then proceeds to step 420, where one or more images are generated based on measurements from two different regions of the image sensor. The number and type of images depends on the current imaging mode of the camera. For example, during white light mode, a true color image may be generated based on color channels of two different regions of the image sensor. As another example, when the camera is in ICG mode, an ICG fluorescence image may be generated based on the detected infrared light. The number and type of images generated are not limited in any way, and any image discussed herein may be generated in step 420. The controller 108 may be used in conjunction with the instructions 128 to access and implement one or more image processing algorithms to convert data received from two different regions of the image sensor (e.g., from the first region and the second region) into corresponding images.

[0056] The method 400 then continues to step 424, where the one or more images are presented to a display. The display may be similar or identical to the display 148. In some embodiments, the one or more images may be captured in the form of a video by repeatedly looping back to step 412 after step 424 and thus repeating steps 412, 416, 420, and 424 for each consecutive video frame. In such a video display, the camera typically continuously receives light and continuously generates images from the sensor measurements. The method 400 then ends. In some embodiments, the method 400 may then be repeated during the course of a surgical procedure or surgical procedure as a different imaging mode is used, for example, by repeating step 404 and selecting a different imaging mode.

[0057] Any of the steps, functions and operations discussed herein may be performed continuously and automatically.

[0058] Although the exemplary embodiments described herein show various components of the system as being collocated, certain components of the system may be remotely located, located in a remote portion of a distributed network (such as a LAN and / or the Internet), or located within a dedicated system. Thus, it should be understood that the components of the system may be combined into one or more devices, such as a server, a communication device, or collocated at a specific node of a distributed network, such as an analog and / or digital telecommunications network, a packet switching network, or a circuit switching network. It will be understood from the foregoing description, and for reasons of computational efficiency, that the components of the system may be arranged at any location in a distributed component network without affecting the operation of the system.

[0059] In addition, it should be understood that the various links connecting the elements can be wired or wireless links, or any combination thereof, or any other known or later developed element capable of supplying and / or communicating data to and from the connected elements. These wired or wireless links can also be secure links and can be capable of communicating encrypted information. For example, the transmission medium used as a link can be any suitable carrier for electrical signals, including coaxial cables, copper wires, and optical fibers, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0060] While the flow charts have been discussed and illustrated with respect to a particular sequence of events, it should be understood that changes, additions, and omissions may be made to the sequence without materially affecting the operation of the disclosed embodiments, configurations, and aspects.

[0061] Various variations and modifications of the disclosure may be used.Some features of the disclosure may be provided without providing other features.

[0062] In yet another embodiment, the system and method of the present disclosure may be implemented in combination with a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit components, an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit (such as a discrete component circuit), a programmable logic device or gate array (such as a PLD, PLA, FPGA, PAL), a special purpose computer, any similar device, etc. In general, any device or device capable of implementing the methods described herein may be used to implement various aspects of the present disclosure. Exemplary hardware that may be used for the present disclosure includes computers, handheld devices, phones (e.g., cellular phones, Internet-enabled phones, digital phones, analog phones, hybrid phones, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. In addition, alternative software implementations may also be constructed, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing to implement the methods described herein.

[0063] In yet another embodiment, the disclosed methods can be easily implemented with software using object or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed systems can be implemented in part or in whole in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware systems or microprocessor or microcomputer systems utilized.

[0064] In yet another embodiment, the disclosed method may be implemented in part in software, which may be stored on a storage medium and executed on a programmed general purpose computer, a special purpose computer, a microprocessor, etc. having cooperation of a controller and a memory. In these cases, the system and method of the present disclosure may be implemented as a program embedded on a personal computer, such as an applet, or CGI scripts, as resources resident on a server or computer workstation, as routines embedded in a dedicated measurement system, system component, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0065] Although the present disclosure describes the components and functions implemented in the embodiments with reference to specific standards and protocols, the present disclosure is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in the present disclosure. In addition, the standards and protocols mentioned herein and other similar standards and protocols not mentioned herein are regularly replaced by faster or more effective equivalents with substantially the same functions. Such alternative standards and protocols with the same functions are considered to be included in the present disclosure.

[0066] In various embodiments, configurations, and aspects, the present disclosure includes components, methods, processes, systems, and / or devices substantially identical to those depicted and described herein, including various embodiments, subcombinations, and subsets thereof. Those skilled in the art will understand how to make and use the systems and methods disclosed herein after understanding the present disclosure. In various embodiments, configurations, and aspects, the present disclosure includes providing devices and processes in the absence of items not described and / or described herein or in its various embodiments, configurations, or aspects (including in the absence of items that may have been used in previous devices or processes), for example, to improve performance, achieve simplicity, and / or reduce the cost of a specific implementation.

[0067] The foregoing discussion of the present disclosure is for the purpose of illustration and description. The foregoing is not intended to limit the present disclosure to one or more forms disclosed herein. For example, in the foregoing detailed description, various features of the present disclosure are grouped together in one or more embodiments, configurations or aspects to simplify the present disclosure. The features of the embodiments, configurations or aspects of the present disclosure may be combined in alternative embodiments, configurations or aspects other than those discussed above. This disclosure method should not be interpreted as reflecting the intention that the claimed disclosure requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the attached claims, the creative aspects are present in less than all features of a single foregoing disclosed embodiment, configuration or aspect. Therefore, the attached claims are hereby incorporated into the present detailed description, wherein each claim is independently a separate preferred embodiment of the present disclosure.

[0068] In addition, although the description of the present disclosure includes descriptions of one or more embodiments, configurations or aspects and certain variations and modifications, other variations, combinations and modifications are also within the scope of the present disclosure, for example, after understanding the present disclosure, it is possible to be within the skill and knowledge of those skilled in the art. It is intended to obtain rights that include alternative embodiments, configurations or aspects within the scope allowed, including structures, functions, scopes or steps that are alternative, interchangeable and / or equivalent to those claimed, regardless of whether such alternative, interchangeable and / or equivalent structures, functions, scopes or steps are disclosed herein, and it is not intended to publicly commit to any patentable subject matter.

[0069] Example aspects of the present disclosure include:

[0070] According to at least one embodiment of the present disclosure, an endoscopic or exoscopic imaging device includes: an optical element configured to separate input light into at least a first spectrally different portion of output light and a second spectrally different portion of output light; and an image sensor having a color filter array (CFA), the image sensor being positioned to receive light from the first spectrally different portion of the output light at a first region of the image sensor and to receive light from the second spectrally different portion of the output light at a second region of the image sensor, wherein the CFA is configured to receive the first portion of the output light and the second portion of the output light incident thereon, and to filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally different from each other, namely, a first spectral channel, a second spectral channel, and a third spectral channel collected by the first region of the image sensor, and a fourth spectral channel, a fifth spectral channel, and a sixth spectral channel collected by the second region of the image sensor.

[0071] According to any of the aspects of the present invention, light of a first wavelength is present in at least two of the first, second, and third spectral channels collected by the first portion of the image sensor and / or at least two of the fourth, fifth, and sixth spectral channels collected by the second portion of the image sensor.

[0072] According to any of the aspects of the present invention, light of the first wavelength is not present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first portion of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second portion of the image sensor.

[0073] According to any of the aspects herein, wherein the at least six resulting spectrally distinct spectral channels of the CFA include a combined red and infrared channel, a green channel, and a blue channel.

[0074] According to any of the aspects herein, wherein the at least six resulting spectrally distinct spectral channels of the CFA include a red light channel, a green light channel, a blue light channel, and an infrared light channel.

[0075] According to any of the aspects herein, further comprising a first spectral filter configured to filter out a certain wavelength range of the input light.

[0076] According to any of the aspects of the present invention, the optical element is configured to separate the input light by a dichroic beam splitter, which reflects light with a wavelength shorter than 500 nanometers (nm), reflects light with a wavelength longer than 650nm, and transmits light with a wavelength between 500nm and 650nm.

[0077] According to any of the aspects herein, the optical element is configured to separate the input light by a dichroic beam splitter that reflects light with wavelengths between 500 nanometers (nm) and 650nm, transmits light with wavelengths shorter than 500nm, and transmits light with wavelengths longer than 650nm.

[0078] According to any of the aspects herein, wherein the first spectral filter blocks a wavelength band corresponding to a fluorescent excitation signal.

[0079] According to any of the aspects herein, further comprising: an illumination device that illuminates the scene to be imaged with light comprising the first set of wavelengths.

[0080] According to any of the aspects herein, wherein the illumination device provides white light having a wavelength between about 450 nanometers (nm) and about 650 nm, and wherein a white light image is created based on a blue light channel and a green light channel of the first area of ​​the image sensor and based on a blue light channel, a green light channel, and a red light channel of the second area of ​​the image sensor.

[0081] According to any of the aspects herein, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm) and an infrared spectral band centered at approximately 940 nm, and wherein an oxygenation or perfusion image is created based at least on an infrared light channel of the first region of the image sensor and also based on a red light channel of the second region of the image sensor.

[0082] According to any of the aspects herein, wherein the illumination device generates illumination having a wavelength between about 725 nanometers (nm) and about 800 nm, and wherein an ICG or OTL fluorescence image is imaged onto the first area of ​​the image sensor.

[0083] According to any of the aspects herein, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm), and wherein a Cy5 or Cy5.5 fluorescent image is imaged onto the first area of ​​the image sensor.

[0084] According to any of the aspects herein, wherein the illumination device generates illumination including a blue spectral band having wavelengths between about 450 nanometers (nm) and about 500 nm, wherein a grayscale image is created based on the illumination imaged onto the first area of ​​the image sensor, and wherein a fluorescein fluorescence image is imaged onto the second area of ​​the image sensor.

[0085] According to any of the aspects herein, wherein the illumination includes spectral components having wavelengths longer than 650 nm, wherein the spectral components are imaged onto the first area of ​​the image sensor, and wherein a partial color image is created based on one or more color channels of the first area of ​​the image sensor.

[0086] According to at least one embodiment of the present disclosure, an endoscopic or exoscopic imaging system includes: an optical element configured to separate input light into at least a first spectrally different portion of output light and a second spectrally different portion of output light directed to at least two discrete regions of an image sensor; a processor and a memory storing instructions thereon, the instructions causing the processor, when processed by the processor, to: detect a first band, a second band, and a third band of light from a first region of the image sensor through a color filter array (CFA), the CFA being configured to receive the first spectrally different portion of the output light and the second spectrally different portion of the light incident thereon and filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally different from each other; and detect a fourth band, a fifth band, and a sixth band of light from a second region of the image sensor through the CFA.

[0087] According to any of the aspects herein, wherein the at least six resulting spectrally distinct spectral channels include a combined red and infrared channel, a green channel, and a blue channel.

[0088] According to any of the aspects herein, the at least six resulting spectrally different spectral channels include a red light channel, a green light channel, a blue light channel, and an infrared light channel.

[0089] According to any of the aspects herein, wherein the illumination device provides white light having a wavelength between about 450 nanometers (nm) and about 650 nm, and wherein a white light image is created based on a blue light channel and a green light channel of the first area of ​​the image sensor and based on a blue light channel, a green light channel, and a red light channel of the second area of ​​the image sensor.

[0090] According to any of the aspects herein, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm) and an infrared spectral band centered at approximately 940 nm, and wherein an oxygenation or perfusion image is created based at least on a combined red and infrared light channel of the first region of the image sensor and also on a red light channel of the second region of the image sensor.

[0091] According to any of the aspects herein, wherein the illumination device generates illumination having a wavelength between about 725 nanometers (nm) and about 800 nm, and wherein an ICG or OTL fluorescence image is imaged onto the first area of ​​the image sensor.

[0092] According to any of the aspects herein, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm), and wherein a Cy5 or Cy5.5 fluorescent image is imaged onto the first area of ​​the image sensor.

[0093] According to any of the aspects herein, wherein an illumination device generates fluorescein excitation illumination comprising a blue spectral band with wavelengths between about 450 nanometers (nm) and about 500 nm, wherein a grayscale image is created based on the fluorescein excitation illumination imaged onto the first area of ​​the image sensor, and wherein a fluorescein fluorescence image is imaged onto the second area of ​​the image sensor.

[0094] According to at least one embodiment of the present disclosure, an endoscopic or exoscopic imaging system includes: an optical element configured to separate input light into at least a first spectrally different portion of output light and a second spectrally different portion of output light; an image sensor having a color filter array (CFA) disposed above a first region and a second region of the image sensor, the CFA configured to receive a first portion of the output light and a second portion of the output light incident thereon, and filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally different from each other, namely, a first spectral channel, a second spectral channel, and a third spectral channel collected by the first region of the image sensor, and a fourth spectral channel, a fifth spectral channel, and a sixth spectral channel collected by the second region of the image sensor; and an illumination device that illuminates a scene to be imaged by light comprising a first set of wavelengths, wherein the imaging system is configured to operate in one or more imaging modes, the one or more imaging modes comprising one or more of the following: a white light imaging mode, an ICG or OTL mode, an oxygenation or perfusion mode, a Cy5 or Cy5.5 mode, and a fluorescein mode.

[0095] According to any of the aspects of the present invention, light of a first wavelength is present in at least two of the first, second, and third spectral channels collected by the first portion of the image sensor and / or at least two of the fourth, fifth, and sixth spectral channels collected by the second portion of the image sensor.

[0096] According to any of the aspects of the present invention, light of the first wavelength is not present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first portion of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second portion of the image sensor.

[0097] Any aspect in combination with any one or more other aspects.

[0098] Any one or more of the features disclosed herein.

[0099] Any one or more features substantially as disclosed herein.

[0100] Any one or more features substantially disclosed herein in combination with any one or more other features substantially disclosed herein.

[0101] Any aspect / feature / embodiment of the various aspects / features / embodiments in combination with any one or more other aspects / features / embodiments.

[0102] The use of any one or more of the aspects or features disclosed herein.

[0103] It should be understood that any feature described herein may be claimed in combination with any other feature as described herein, whether or not such features are from the same described embodiment.

[0104] The phrases "at least one", "one or more", "or", and "and / or" are open expressions that are both conjunctions and disjuncts in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0105] The term "a" or "an" entity refers to one or more of that entity. Therefore, the terms "a" (or "an"), "one or more" and "at least one" can be used interchangeably herein. It should also be noted that the terms "including", "comprising" and "having" can be used interchangeably.

[0106] As used herein, the term "automatic" and variations thereof refer to any process or operation that is generally continuous or semi-continuous and that is performed without substantial human input. However, if input is received prior to the performance of the process or operation, then the process or operation may be automatic, even if the performance of the process or operation uses substantial or insubstantial human input. Human input is considered substantial if it affects the manner in which the process or operation will be performed. Human input that consents to the performance of the process or operation is not considered "substantial."

[0107] Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software (including firmware, resident software, microcode, etc.) embodiment, or an embodiment combining software and hardware aspects, which may all be generally referred to herein as a "circuit," "module," or "system." Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0108] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program used by or in connection with an instruction execution system, apparatus, or device.

[0109] A computer readable signal medium may include a propagated data signal in which a computer readable program code is embodied, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage device and that can convey, propagate, or transmit a program for use by or in connection with an instruction execution system, device, or apparatus. The program code embodied on a computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0110] As used herein, the terms "determine," "calculate," "compute," and variations thereof are used interchangeably and include any type of methodology, process, mathematical operation, or technique.

Claims

1. An endoscopic or exoscopic imaging device, comprising: an optical element configured to separate input light into at least a first spectrally distinct portion of output light and a second spectrally distinct portion of output light; and an image sensor having a color filter array (CFA), the image sensor being positioned to receive light from a first spectrally distinct portion of the output light at a first region of the image sensor and to receive light from a second spectrally distinct portion of the output light at a second region of the image sensor, The CFA is configured to receive a first portion of the output light and a second portion of the output light incident thereon, and filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally different from each other, namely, a first spectral channel, a second spectral channel, and a third spectral channel collected by the first area of ​​the image sensor, and a fourth spectral channel, a fifth spectral channel, and a sixth spectral channel collected by the second area of ​​the image sensor.

2. An imaging device according to claim 1, wherein light of a first wavelength is present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first part of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second part of the image sensor.

3. An imaging device according to claim 1, wherein light of a first wavelength is not present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first part of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second part of the image sensor.

4. The imaging device of claim 1, wherein the at least six resulting spectrally distinct spectral channels of the CFA include a combined red and infrared channel, a green channel, and a blue channel. 5 . The imaging device of claim 1 , wherein the at least six resulting spectrally distinct spectral channels of the CFA include a red channel, a green channel, a blue channel, and an infrared channel. 6 . The imaging device of claim 1 , further comprising a first spectral filter configured to filter out a certain wavelength range of the input light.

7. An imaging device according to claim 1, wherein the optical element is configured to separate the input light by a dichroic beam splitter, which reflects light with a wavelength shorter than 500 nanometers (nm), reflects light with a wavelength longer than 650nm, and transmits light with a wavelength between 500nm and 650nm.

8. The imaging device of claim 1 , wherein the optical element is configured to separate the input light by a dichroic beam splitter that reflects light having a wavelength between 500 nanometers (nm) and 650 nm, transmits light having a wavelength shorter than 500 nm, and transmits light having a wavelength longer than 650 nm.

9. The imaging device of claim 6, wherein the first spectral filter blocks a wavelength band corresponding to a fluorescent excitation signal.

10. The imaging device according to claim 1, further comprising: An illumination device illuminates a scene to be imaged with light comprising a first set of wavelengths.

11. The imaging device of claim 10, wherein the illumination device provides white light having a wavelength between about 450 nanometers (nm) and about 650 nm, and wherein a white light image is created based on a blue light channel and a green light channel of the first area of ​​the image sensor and based on a blue light channel, a green light channel, and a red light channel of the second area of ​​the image sensor.

12. An imaging device according to claim 10, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm) and an infrared spectral band centered at approximately 940 nm, and wherein an oxygenation or perfusion image is created based at least on an infrared light channel of the first region of the image sensor and also based on a red light channel of the second region of the image sensor.

13. The imaging device of claim 10, wherein the illumination device generates illumination having a wavelength between about 725 nanometers (nm) and about 800 nm, and wherein an ICG or OTL fluorescence image is imaged onto the first region of the image sensor.

14. The imaging device of claim 10, wherein the illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm), and wherein a Cy5 or Cy5.5 fluorescent image is imaged onto the first region of the image sensor.

15. The imaging device of claim 10, wherein the illumination device generates illumination comprising a blue spectral band having wavelengths between approximately 450 nanometers (nm) and approximately 500 nm, wherein a grayscale image is created based on the illumination imaged onto the first area of ​​the image sensor, and wherein a fluorescein fluorescence image is imaged onto the second area of ​​the image sensor.

16. The imaging device of claim 15, wherein the illumination includes spectral components having wavelengths longer than 650 nm, wherein the spectral components are imaged onto the first area of ​​the image sensor, and wherein a partial color image is created based on one or more color channels of the first area of ​​the image sensor.

17. An endoscopic or exoscopic imaging system, comprising: an optical element configured to separate input light into at least a first spectrally distinct portion of output light and a second spectrally distinct portion of output light that is directed to at least two discrete regions of the image sensor; processor; and a memory storing thereon instructions that, when processed by the processor, cause the processor to: detecting a first band, a second band, and a third band of light from a first area of ​​the image sensor via a color filter array, or CFA, the CFA configured to receive a first spectrally distinct portion of the output light and a second spectrally distinct portion of the light incident thereon and to filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally distinct from one another; as well as A fourth band, a fifth band, and a sixth band of light are detected by the CFA from a second area of ​​the image sensor.

18. The system of claim 17, wherein the at least six resulting spectrally distinct spectral channels include a combined red and infrared channel, a green channel, and a blue channel.

19. The system of claim 17, wherein the at least six resulting spectrally distinct spectral channels include a red light channel, a green light channel, a blue light channel, and an infrared light channel.

20. The system of claim 17, wherein the illumination device provides white light having a wavelength between about 450 nanometers (nm) and about 650 nm, and wherein a white light image is created based on a blue light channel and a green light channel of the first area of ​​the image sensor and based on a blue light channel, a green light channel, and a red light channel of the second area of ​​the image sensor.

21. The system of claim 17, wherein the illumination device generates illumination comprising a red spectral band centered at approximately 630 nanometers (nm) and an infrared spectral band centered at approximately 940 nm, and wherein an oxygenation or perfusion image is created based at least on a combined red and infrared light channel of the first region of the image sensor and also on a red light channel of the second region of the image sensor.

22. The system of claim 17, wherein an illumination device generates illumination having a wavelength between about 725 nanometers (nm) and about 800 nm, and wherein an ICG or OTL fluorescence image is imaged onto the first region of the image sensor.

23. The system of claim 17, wherein an illumination device generates illumination including a red spectral band centered at approximately 630 nanometers (nm), and wherein a Cy5 or Cy5.5 fluorescent image is imaged onto the first region of the image sensor.

24. The system of claim 17, wherein an illumination device generates fluorescein excitation illumination comprising a blue spectral band having wavelengths between approximately 450 nanometers (nm) and approximately 500 nm, wherein a grayscale image is created based on the fluorescein excitation illumination imaged onto the first area of ​​the image sensor, and wherein a fluorescein fluorescence image is imaged onto the second area of ​​the image sensor.

25. An endoscopic or exoscopic imaging system comprising: an optical element configured to separate input light into at least a first spectrally distinct portion of output light and a second spectrally distinct portion of output light; an image sensor having a color filter array (CFA) disposed over a first region and a second region of the image sensor, the CFA being configured to receive a first portion of the output light and a second portion of the output light incident thereon, and to filter the first portion of the output light and the second portion of the output light into at least six resulting spectral channels that are spectrally different from one another, namely, a first spectral channel, a second spectral channel, and a third spectral channel collected by the first region of the image sensor, and a fourth spectral channel, a fifth spectral channel, and a sixth spectral channel collected by the second region of the image sensor; and an illumination device for illuminating a scene to be imaged with light comprising a first set of wavelengths, The imaging system is configured to operate in one or more imaging modes, comprising one or more of the following: white light imaging mode, ICG or OTL mode, oxygenation or perfusion mode, Cy5 or Cy5.5 mode and fluorescein mode.

26. An imaging system according to claim 25, wherein light of a first wavelength is present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first portion of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second portion of the image sensor.

27. An imaging system according to claim 25, wherein light of a first wavelength is not present in at least two of the first spectral channel, the second spectral channel, and the third spectral channel collected by the first portion of the image sensor and / or at least two of the fourth spectral channel, the fifth spectral channel, and the sixth spectral channel collected by the second portion of the image sensor.

Images