Image rotation in endoscopic fluorescence imaging systems

Abstract

Image rotation in an endoscopic fluorescence imaging system is described. The invention provides a system comprising an emitter for emitting a pulse of electromagnetic radiation and an image sensor comprising an array of pixels for sensing reflected electromagnetic radiation. The system comprises a rotation sensor for detecting a rotation angle of a lumen relative to a handpiece of an endoscope. The system is such that at least a portion of the pulse of electromagnetic radiation emitted by the emitter comprises electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or about 795 nm to about 815 nm.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to digital imaging, and in particular to fluorescence imaging in light deficient environments. BACKGROUND

[0002] Advances in technology have provided advances in medical imaging capabilities. Endoscopes can be used to view inside the body and inspect the interior of organs or cavities of the body. Endoscopes are used to investigate symptoms in a patient, confirm a diagnosis, or provide medical treatment. Medical endoscopes can be used to view a variety of body systems and portions, such as the gastrointestinal tract, the respiratory tract, the urinary tract, the abdominal cavity, and the like. Endoscopes can also be used for surgical procedures, such as orthopedic surgery, procedures performed on joints or bones, procedures performed on the nervous system, procedures performed within the abdominal cavity, and the like.

[0003] In some cases of endoscopic imaging, it can be advantageous or necessary to view a color space. A digital color image includes at least three layers or“color channels” that cumulatively form an image with a range of hues. Each of the color channels measures the intensity and hue of a band of the light spectrum. Typically, a digital color image includes color channels for the red, green, and blue bands of the spectrum (this can be referred to as a red-green-blue or RGB image). Each of the red, green, and blue color channels includes luminance information for the red, green, or blue band of the spectrum. The luminance information of the separate red, green, and blue layers is combined to generate a color image. Because a color image is composed of separate layers, conventional digital camera image sensors include an array of color filters that allow red, green, and blue visible wavelengths of light to hit selected pixel sensors. Each individual pixel sensor element is sensitive to red, green, or blue wavelengths, and will only return image data for that wavelength. The image data from the total array of pixel sensors is combined to generate an RGB image. Having at least three different types of pixel sensors occupies a significant amount of physical space, such that a complete array of pixels cannot fit in the smaller distal end of an endoscope.

[0004] Because conventional image sensors cannot fit in the distal end of an endoscope, the image sensor is traditionally located in the handpiece unit of the endoscope, which is held by the endoscope operator and is not placed inside the body cavity. In such endoscopes, light is transmitted along the length of the endoscope from the handpiece unit to the distal end of the endoscope. This configuration has significant limitations. Endoscopes with this configuration are delicate, and can easily become misaligned or damaged when they are bumped or impacted during routine use. This can significantly degrade the quality of the image and require frequent repair or replacement of the endoscope.

[0005] Conventional endoscopes with image sensors placed in the handpiece unit are further limited to capturing only color images. However, in some implementations, it can be desirable to capture images with fluorescent image data in addition to color image data. Fluorescence is light emitted by a substance that has absorbed light or other electromagnetic radiation. Certain fluorescent substances "glow" or emit a different color visible to the human eye when they are subjected to ultraviolet light or other wavelengths of electromagnetic radiation. Certain fluorescent materials will stop glowing almost immediately when the source of radiation stops.

[0006] Fluorescence occurs when an orbital electron of a molecule, atom, or nanostructure is excited by light or other electromagnetic radiation and then relaxes to its ground state by emitting a photon from the excited state. The specific frequency of the excited orbital electron or the electromagnetic radiation emitted by the photon during relaxation depends on the atom, molecule, or nanostructure. Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy for chemical sensors, detecting biological processes or signals, and the like. Fluorescence can be used in biochemistry and medicine as a non-destructive means for tracking or analyzing biological molecules. Some fluorescent reagents or dyes can be configured to be able to attach to certain types of tissue, thereby causing attention to that type of tissue.

[0007] However, fluorescence imaging requires a specific emission of electromagnetic radiation and a specific imaging sensor capable of reading the specific relaxation wavelength of the specific fluorescent reagent. Different reagents or dyes are sensitive to different wavelengths of electromagnetic radiation and emit different wavelengths of electromagnetic radiation when fluorescing. Fluorescence imaging systems can be highly specialized and can be tuned for a certain reagent or dye. Such imaging systems are used for limited applications and cannot fluoresce more than one reagent or structure during a single imaging process. It is very expensive to use multiple different imaging systems each configured to fluoresce a different reagent. Additionally, it can be advantageous to administer multiple fluorescent reagents in a single imaging process and observe the multiple reagents in a single superimposed image.

[0008] In light of the foregoing, described herein are systems, methods, and apparatuses for fluorescence imaging in a light deficient environment. Such systems, methods, and apparatuses can provide multiple data sets for identifying key structures within the body and providing precise and valuable information about the body cavity. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present disclosure is described with reference to the following non-limiting and non-exhaustive specific embodiments, in which like elements are referred to with like numerals throughout the various figures described below. Aspects of the present disclosure are best understood from the following

[0010] Figure 1 is a schematic of a system for digital imaging in a light deficient environment with paired emitters and pixel arrays;

[0011] Figure 2 is a system for providing illumination to a light deficient environment for endoscopic imaging;

[0012] Figure 2A is a schematic of complementary system hardware;

[0013] Figures 3A to 3D is a graphical representation of the operation of an embodiment of the present disclosure combining the operation cycle of the sensor of

[0014] Figure 4A is a graphical representation of the operation of an embodiment of the electromagnetic emitter;

[0015] Figure 4B is a graphical representation of varying the duration and magnitude of the emitted electromagnetic pulses to provide exposure control;

[0016] Figure 5 is a graphical representation of the operation cycle of the sensor of Figures 3A to 4B combining the electromagnetic emitter and the emitted electromagnetic pulses, showing the imaging system during operation;

[0017] Figure 6A is a schematic of a process for recording a video with full-spectrum light over a time period from t(0) to t(l);

[0018] Figure 6B is a schematic of a process for recording a video by pulsing a partitioned-spectrum light over a time period from t(0) to t(l);

[0019] Figures 7A to 7E is a schematic showing a process for recording video frames of both full-spectrum light and partitioned-spectrum light over a time interval;

[0020] Figure 8 shows an endoscopic imaging system including a handpiece and an endoscope tube, where the handpiece and endoscope tube or capable of rotating relative to each other;

[0021] Figure 9 shows an embodiment of a rotation sensor for implementation within an endoscopic imaging system;

[0022] Figure 10 shows an embodiment of a rotation sensor for implementation within an endoscopic imaging system;

[0023] Figure 11 shows an embodiment of a rotation sensor for implementation within an endoscopic imaging system;

[0024] Figure 12A and Figure 12BEmbodiments of an endoscopic imaging system are shown that include an outer lumen, a distal lens, a prism, and an image sensor for enabling digital imaging of a wide field of view;

[0025] Figures 13A to 13C A light source having multiple emitters is shown;

[0026] Figure 14 A single optical fiber is shown that outputs via a diffuser at an output to illuminate a scene in a light deficient environment;

[0027] Figure 15 A portion of the electromagnetic spectrum that is split into multiple different sub-spectra that can be emitted by emitters of a light source is shown in accordance with the principles and teachings of the present disclosure;

[0028] Figure 16 A schematic diagram showing emission and readout timing for generating an image frame that includes multiple exposure frames produced by different partitions of pulsed light is shown;

[0029] Figure 17 An imaging system is shown that includes a single cut filter for filtering wavelengths of electromagnetic radiation;

[0030] Figure 18 An imaging system is shown that includes multiple cut filters for filtering wavelengths of electromagnetic radiation;

[0031] Figure 19A And Figure 19B Perspective and side views of embodiments of an imaging sensor having multiple pixel arrays for producing a three-dimensional image are shown, respectively, wherein the multiple pixel arrays and the image sensor are built on multiple substrates;

[0032] Figure 20A And Figure 20B Embodiments having multiple pixel arrays for producing a three-dimensional image are shown in accordance with the principles and teachings of the present disclosure; and

[0033] Figure 21A And Figure 21B Perspective and side views of embodiments of an imaging sensor built on multiple substrates are shown, respectively, wherein multiple pixel columns forming pixel arrays are located on a first substrate and multiple circuit columns are located on a second substrate, and electrical connections and communications between a column of pixels and its associated or corresponding column of circuitry are shown. DETAILED DESCRIPTION

[0034] Disclosed herein are systems, methods, and apparatuses for digital imaging that can be primarily adapted for medical applications such as medical endoscopic imaging. One embodiment of the present disclosure is an endoscopic system for fluorescence and color imaging in light deficient environments. Such methods, systems, and computer-based products disclosed herein provide imaging or diagnostic capabilities for medical robotic applications such as the use of robots for performing imaging procedures, surgical procedures, and the like.

[0035] Embodiments of the present disclosure are endoscopic imaging systems in which an image sensor is disposed in the distal tip of an endoscope. This configuration enables a number of advantages over conventional endoscopes as discussed herein. However, this configuration also introduces challenges for establishing the correct image orientation for the user. In some cases, and depending on user preference, the image orientation should reflect the orientation of the handpiece unit of the endoscope relative to the imaged scene. In one embodiment, the correct image orientation is established and maintained through post-processing in the image signal processing (ISP) pipeline. In this embodiment, a digital representation of the angle of the endoscope relative to the handpiece unit is continuously available to and adjusted by the ISP during operation.

[0036] Conventional endoscopes are designed such that the image sensor is placed at the proximal end of the device within the handpiece unit. This configuration requires the incident light to travel the length of the endoscope through precisely coupled optical elements. Precise optical elements are susceptible to misalignment during normal use, and this can result in image distortion or image loss. Embodiments of the present disclosure place the image sensor within the spatially constrained environment in the distal end of the endoscope itself. This provides greater optical simplicity compared to specific implementations known in the art. However, the acceptable solution to this approach is by no means simple and raises its own series of engineering challenges.

[0037] When the overall size of the image sensor is minimized such that the image sensor can fit within the distal tip of the endoscope, there can be a significant loss of image quality. The area of the pixel array of the image sensor can be reduced by reducing the number of pixels and / or reducing the sensing area of each individual pixel. Each of these modifications impacts the resolution, sensitivity, and dynamic range of the resulting image. Traditional endoscopic imaging systems aim to sense stable broadband illumination and provide color information through a segmented pixel array such as a Bayer pattern array. In view of the deficiencies associated with segmented pixel arrays, disclosed herein are alternative systems and methods that use a monochrome (which can be referred to as “color agnostic”) pixel array that does not include individual pixel filters. In the embodiments disclosed herein, color information is provided through the use of different wavelength electromagnetic radiation pulse emitters. The pulsed imaging systems disclosed herein can generate color images with fluorescence imaging data superimposed thereon.

[0038] In one embodiment, color information is determined by capturing independent exposure frames in response to different wavelength pulses of electromagnetic radiation. The alternative pulses can include red, green, and blue wavelengths for generating an RGB image frame composed of a red exposure frame, a green exposure frame, and a blue exposure frame. In alternative implementations, the alternative pulses can include luminance (“Y”) pulses, red chrominance (“Cr”) pulses, and blue chrominance (“Cb”) pulses of light for generating a YCbCr image frame composed of luminance data, red chrominance data, and blue chrominance data. The color image frame can also include data from a fluorescence exposure frame superimposed on the RGB or YCbCr image frame. The fluorescence pulses can include one or more pulses of electromagnetic radiation used to elicit a spectral response. In one embodiment, the fluorescence emission includes one or more of electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or about 795 nm to about 815 nm. Alternating the wavelengths of the pulsed electromagnetic radiation allows for the use of a full pixel array and avoids artifacts elicited by a Bayer pattern pixel array.

[0039] In some cases, it is desirable to generate endoscopic imaging with multiple data types or multiple images layered on one another. For example, it is desirable to generate a color (RGB or YCbCr) image that also includes fluorescence imaging data superimposed on the color image. Superimposed images of this nature can enable a medical practitioner or computer program to identify key body structures based on the fluorescence imaging data. Historically, this would require the use of multiple sensor systems including an image sensor for color imaging and one or more additional image sensors for fluorescence imaging. In such systems, the multiple image sensors would have multiple types of pixel sensors each sensitive to a different range of electromagnetic radiation. In systems known in the art, this includes three separate types of pixel sensors for generating color images, and additional pixel sensors for generating fluorescence image data at different wavelengths of the electromagnetic spectrum. These multiple different pixel sensors occupy an excessive amount of physical space and cannot be located at the distal tip of an endoscope. In systems known in the art, the one or more cameras are not placed at the distal tip of the endoscope, but rather in the endoscope handpiece or robotic unit. This elicits a number of drawbacks and results in an endoscope that is very fragile. When the fragile endoscope is bumped or impacted during use, the endoscope can be damaged and the image quality is reduced. In view of the foregoing, disclosed herein are systems, methods, and devices for endoscopic imaging in low light environments. The systems, methods, and devices disclosed herein provide a way to employ multiple imaging techniques in a single imaging session while allowing one or more image sensors to be disposed in the distal tip of an endoscope.

[0040] Fluorescent imaging

[0041] The systems, methods, and devices disclosed herein provide ways for generating fluorescence imaging data in a light deficient environment. The fluorescence imaging data can be used to identify certain materials, tissues, components, or processes within a body cavity or other light deficient environment. In certain embodiments, the fluorescence imaging can be provided to a medical practitioner or a computer implemented program to enable the identification of certain structures or tissues within the body. Such fluorescence imaging data can be overlaid on black and white or RGB images to provide additional information and context.

[0042] Fluorescence is light emitted by a substance that has absorbed light or other electromagnetic radiation. Certain fluorescent substances can "glow" or emit a different color visible to the human eye when the fluorescent substance is subjected to ultraviolet light or other wavelengths of electromagnetic radiation. Certain fluorescent materials will stop glowing almost immediately when the source of radiation stops.

[0043] Fluorescence occurs when an orbital electron of a molecule, atom, or nanostructure is excited by light or other electromagnetic radiation and then relaxes to its ground state by emitting a photon. The specific frequency of the excited orbital electron or the electromagnetic radiation emitted by the photon during relaxation depends on the specific atom, molecule, or nanostructure. In most cases, the light emitted by the substance has a longer wavelength and thus lower energy than the radiation absorbed by the substance. However, when the absorbed electromagnetic radiation is intense, one electron can absorb two photons. This two-photon absorption can result in the emission of radiation with a shorter wavelength and thus higher energy than the absorbed radiation. Additionally, the emitted radiation can also have the same wavelength as the absorbed radiation.

[0044] Fluorescence imaging has many practical applications, including mineralogy, geology, medicine, spectroscopy of chemical sensors, detecting biological processes or signals, etc. Fluorescence can be particularly useful in biochemistry and medicine as a non-destructive means for tracking or analyzing biological molecules. Biological molecules, including certain tissues or structures, can be tracked by analyzing the fluorescence emission of the biological molecules after being excited by certain wavelengths of electromagnetic radiation. However, relatively few cellular components are naturally fluorescent. In certain implementations, it can be advantageous to visualize certain tissues, structures, chemical processes, or biological processes that are not fluorescent in nature. In such implementations, a dye or agent can be administered to the body that can include molecules, proteins, or quantum dots with fluorescence properties. The agent or dye can then fluoresce after being excited by certain wavelengths of electromagnetic radiation. Different agents or dyes can include different molecules, proteins, and / or quantum dots that will fluoresce under certain wavelengths of electromagnetic radiation. Thus, it can be necessary to excite the agent or dye with a particular band of electromagnetic radiation to achieve fluorescence and identify the desired tissue, structure, or process within the body.

[0045] Fluorescence imaging can provide valuable information in the medical field that can be used for diagnostic purposes and / or can be visualized in real-time during a medical procedure. Certain agents or dyes can be administered to the body to cause certain tissues, structures, chemical processes, or biological processes to fluoresce. The fluorescence of the agents or dyes can highlight body structures, such as blood vessels, nerves, certain organs, etc. Additionally, the fluorescence of the agents or dyes can highlight conditions or diseases, such as cancerous cells or cells undergoing certain biological or chemical processes that can be associated with a condition or disease. Fluorescence imaging can be used in real-time by a medical practitioner or a computer program for distinguishing, for example, cancerous cells from non-cancerous cells during surgical tumor extraction. Fluorescence imaging can also be used as a non-destructive means for tracking and visualizing conditions within the body that are otherwise invisible to the human eye or indistinguishable in RGB images over time.

[0046] Systems, methods, and devices for generating fluorescence imaging data can be used in conjunction with agents or dyes. Certain agents or dyes are known to attach to certain types of tissues and fluoresce at specific wavelengths of the electromagnetic spectrum. In one implementation, a patient is administered an agent or dye that is configured to be able to fluoresce when activated by light of certain wavelengths. The agent or dye is excited and caused to fluoresce using the endoscopic imaging system disclosed herein. The fluorescence of the agent or dye is captured by the endoscopic imaging system to aid in the identification of tissues or structures in a body cavity. In one implementation, a patient is administered multiple agents or dyes, each configured to be able to fluoresce at a different wavelength and / or provide an indication of a different structure, tissue, chemical reaction, biological process, etc. In such implementations, the endoscopic imaging system emits each of the applicable wavelengths to cause each of the applicable agents or dyes to fluoresce. This can eliminate the need to perform a single imaging procedure for each of the multiple agents or dyes.

[0047] Imaging agents can enhance imaging capabilities in the pharmaceutical, medical, biotechnology, diagnostic, and medical procedure industries. Many imaging technologies such as X-ray, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and nuclear medicine primarily analyze anatomy and morphology and cannot detect changes at the molecular level. Fluorescent agents, dyes, and probes, including quantum dot nanoparticles and fluorescent proteins, assist medical imaging technologies by providing additional information about certain tissues, structures, chemical processes, and / or biological processes present within the imaged region. Imaging with fluorescent agents enables cell tracking and / or tracking of certain molecular biomarkers. Fluorescent agents can be used to image cancer, infections, inflammation, stem cell biology, and the like. Many fluorescent agents and dyes are being developed and applied to visualize and track biological processes in a non-destructive manner. Such fluorescent agents can be excited by electromagnetic radiation of certain wavelengths or wavelength bands. Similarly, these fluorescent agents can emit relaxation energy of certain wavelengths or wavelength bands when fluorescing, and the emitted relaxation energy can be read by a sensor to determine the location and / or boundaries of the agent or dye.

[0048] In one embodiment of the disclosure, an endoscopic imaging system pulses electromagnetic radiation to excite electrons in fluorescent agents or dyes. The endoscopic imaging system can pulse electromagnetic radiation of multiple different wavelengths during a single imaging session to cause multiple different agents or dyes to fluoresce. The endoscope includes an image sensor that is sensitive to the relaxation wavelengths of one or more agents or dyes. The imaging data generated by the image sensor can be used to identify the location and boundaries of one or more agents or dyes. The endoscopic system can also pulse electromagnetic radiation in the red, green, and blue bands of visible light so that the fluorescent imaging can be overlaid on an RGB video stream.

[0049] Pulsed imaging

[0050] Some implementations of the disclosure include various aspects of a sensor and system combination design that is capable of generating high definition images with reduced number of pixels in a constrained lighting environment. This is achieved by pulsing a single color wavelength frame by frame and using a controlled light source in combination with a high frame capture rate and specially designed corresponding monochrome sensor to switch or alternate between each frame at a single different color wavelength. Additionally, electromagnetic radiation outside the visible spectrum can be pulsed to enable generation of fluorescent images. The pixels can be color agnostic such that each pixel generates data for each electromagnetic radiation pulse, including pulses of red, green, and blue visible light wavelengths and other wavelengths useful for fluorescent imaging.

[0051] The system of the present disclosure is an endoscope system for use in light deficient environments. The system includes an endoscope including an image sensor, where the image sensor is configured to be able to sense reflected electromagnetic radiation for generating a plurality of exposure frames that can be combined to generate an RGB image frame with fluorescence data superimposed thereon. The system includes an emitter for emitting pulses of electromagnetic radiation. The system includes a controller (alternatively referred to as "control circuitry") in electrical communication with the image sensor and the emitter. The controller controls the duty cycle of the emitter in response to a signal corresponding to the duty cycle of the emitter. The image sensor includes a bidirectional pad that can send and receive information. The bidirectional pad of the image sensor operates in a frame period that is divided into three defined states, including a rolling readout state, a service line state, and a configuration state. The system includes an oscillator disposed in the controller and a frequency detector connected to the controller. The frequency detector controls the clock frequency of the image sensor in response to a signal from the controller corresponding to the oscillator frequency. The system causes clock signal data to be transmitted from the bidirectional pad of the image sensor to the controller during the service line phase and the configuration phase. The system causes the exposure frames to be synchronized without using an input clock or a data transfer clock.

[0052] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe these embodiments. It will, nevertheless, be understood that no limitation of the scope of the disclosure is intended by this specification. Any alterations and further modifications of the described features of this application, and any additional or additional applications of the principles of the present disclosure as described herein are contemplated as falling within the scope of the disclosure as defined by the appended claims.

[0053] Before the structures, systems and methods for producing images in light deficient environments are disclosed and described, it is to be understood that the disclosure is not limited to the particular structures, configurations, process steps, and materials disclosed herein as such structures, configurations, process steps and materials can vary somewhat. It is also to be understood that the terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting since the scope of the present application will be limited only by the appended claims and equivalents thereof.

[0054] In describing and claiming the subject matter of the present disclosure, the following terminology will be used in accordance with the definitions set out below.

[0055] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0056] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," "containing," "characterized by," and the like, are open-ended terms that are not to be construed as limiting in any way.

[0057] As used herein, the phrase "consisting of" and its grammatical equivalents exclude any element or step not specified in the claim.

[0058] As used herein, the phrase "consisting essentially of" and its grammatical equivalents limits the scope of the claim to the specified materials or steps and those that do not materially affect the basic and novel characteristic(s) of the claimed disclosure.

[0059] As used herein, the term "proximal" broadly refers to the concept of a portion that is close to a starting point.

[0060] As used herein, the term "distal" generally refers to the concept of a portion that is opposite to proximal, and thus depending on the context, it refers to a portion that is further from the starting point or the furthest portion.

[0061] As used herein, color sensors or multispectral sensors are those known to have an array of color filters (CFA) on them in order to filter the incident electromagnetic radiation into its individual components. In the visible range of the electromagnetic spectrum, such a CFA can be based on a Bayer pattern or a modified version thereof in order to separate the green, red, and blue spectral components in the light.

[0062] As used herein, monochrome sensors refer to imaging sensors without a filtering function. Because the pixels are color agnostic, their effective spatial resolution is significantly higher than the pixel color (typically filtered with a Bayer pattern) counterparts in traditional single-sensor cameras. Monochrome sensors can also have higher quantum efficiency due to fewer wasted incident photons between individual pixels.

[0063] As used herein, an emitter is a device capable of generating and emitting electromagnetic pulses. Various embodiments of an emitter can be configured to be capable of emitting pulses and have a very specific frequency or range of frequencies from within the entire electromagnetic spectrum. The pulses can include wavelengths from the visible range and the non-visible range. The emitter can cycle on and off to produce pulses, or can utilize a shutter mechanism to produce pulses. The emitter can have a variable power output level, or can be controlled with an auxiliary device such as an aperture or filter. The emitter can emit a broad spectrum or full spectrum of electromagnetic radiation that can be pulsed by color filtering or shuttering. The emitter can include multiple electromagnetic sources acting individually or in concert.

[0064] It should be noted that as used herein, the term "light" is both a particle and a wavelength, and is intended to mean electromagnetic radiation that can be detected by the pixel array 122, and can include wavelengths of electromagnetic radiation from the visible spectrum and the non-visible spectrum. The term "partition" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum, or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a source of light that is controllable with respect to the portion of the electromagnetic spectrum that is emitted, or the physical properties, emission intensity, or emission duration of the components that can be operated, or all of the above. The emitter can emit light in any of a dithered, diffuse, or collimated emission, and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy, and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0065] Referring now to the drawings, Figure 1 A schematic diagram of a system 100 for sequential pulsed imaging in a light deficient environment is shown. The system 100 can be deployed to generate an RGB image with fluorescent imaging data overlaid on the RGB image. The system 100 includes an emitter 102 and a pixel array 122. The emitter 102 pulses a partition of electromagnetic radiation in a light deficient environment 112, and the pixel array 122 senses instances of reflected electromagnetic radiation. The emitter 102 and the pixel array 122 operate sequentially such that one or more pulses of the partition of electromagnetic radiation produces image data sensed by the pixel array 122.

[0066] It should be noted that as used herein, the term "light" is both a particle and a wavelength, and is intended to mean electromagnetic radiation that can be detected by the pixel array 122, and can include wavelengths of electromagnetic radiation from the visible spectrum and the non-visible spectrum. The term "partition" as used herein refers to a predetermined range of wavelengths of the electromagnetic spectrum that is less than the entire spectrum, or in other words, wavelengths that make up a portion of the electromagnetic spectrum. As used herein, an emitter is a source of light that is controllable with respect to the portion of the electromagnetic spectrum that is emitted, or the physical properties, emission intensity, or emission duration of the components that can be operated, or all of the above. The emitter can emit light in any of a dithered, diffuse, or collimated emission, and can be controlled digitally or by analog methods or systems. As used herein, an electromagnetic emitter is a source of bursts of electromagnetic energy, and includes light sources such as lasers, LEDs, incandescent light, or any light source that can be digitally controlled.

[0067] The pixel array 122 of the image sensor can be electronically paired with the emitter 102 such that the emitter 102 and the pixel array 122 are synchronized during operation for both receiving the emission and for adjustments made within the system. The emitter 102 can be tuned to emit electromagnetic radiation in the form of laser light that can be pulsed to illuminate the light deficient environment 112. The emitter 102 can pulse at intervals corresponding to the operation and function of the pixel array 122. The emitter 102 can pulse light in multiple electromagnetic partitions such that the pixel array receives electromagnetic energy and produces a data set corresponding in time to each particular electromagnetic partition. For example, Figure 1 One implementation is shown in which the emitter 102 emits four different partitions of electromagnetic radiation including red 104, green 106, blue 108, and fluorescent excitation 110 wavelengths. The fluorescent excitation 110 wavelengths can include multiple different partitions of electromagnetic radiation selected to cause multiple fluorescent reagents present within the light deficient environment 112 to fluoresce. The fluorescent excitation 110 wavelengths can be selected to cause a particular fluorescent reagent present in the light deficient environment 112 to fluoresce.

[0068] In Figure 1 An alternative implementation not shown in FIG. 1 includes pulsed emission of light including luminance (“Y”) emission, red chrominance (“Cr”) emission, and blue chrominance (“Cb”) emission instead of pulsed red 104 emission, pulsed green 106 emission, and pulsed blue 108 emission. In one implementation, the controller or emitter 102 modulates the electromagnetic radiation pulses to provide luminance and / or chrominance information according to color transform coefficients that convert light energy from red, green, and blue light energy spaces to luminance, red chrominance, and blue chrominance light energy spaces. The pulsed emission of light can also include modulated blue chrominance (“Y+Cb”) pulses and / or modulated red chrominance (“Y+Cr”) pulses.

[0069] The light deficient environment 112 includes structures, tissues, and other elements that reflect a combination of red 114, green 116, and / or blue 118 light. Structures perceived as red 114 will reflect pulsed red 104 light. Reflection from red structures results in the pixel array 122 sensing red 105 after pulsed red 104 emission. The data sensed by the pixel array 122 produces a red exposure frame. Structures perceived as green 116 will reflect pulsed green 106 light. Reflection from green structures results in the pixel array 122 sensing green 107 after pulsed green 106 emission. The data sensed by the pixel array 122 produces a green exposure frame. Structures perceived as blue 118 will reflect pulsed blue 108 light. Reflection from blue structures results in the pixel array 122 sensing blue 109 after pulsed blue 108 emission. The data sensed by the pixel array 122 produces a blue exposure frame.

[0070] When the structure is a combination of colors, the structure will reflect a combination of the pulsed red 104 emission, the pulsed green 106 emission, and / or the pulsed blue 108 emission. For example, a structure perceived as purple will reflect light from the pulsed red 104 emission and the pulsed blue 108 emission. The resulting data sensed by the pixel array 122 will indicate that light was reflected in the same area after the pulsed red 104 emission and the pulsed blue 108 emission. When the resulting red exposure frame and blue exposure frame are combined to form an RGB image frame, the RGB image frame will indicate that the structure is purple.

[0071] In embodiments where the light deficient environment 112 includes a fluorescent reagent or fluorescent dye or includes one or more fluorescent structures, tissues, or other elements, the pulsing scheme can include emission of certain fluorescent excitation wavelengths. Certain fluorescent excitation wavelengths can be selected to cause known fluorescent reagents, fluorescent dyes, or other structures to fluoresce. The fluorescent structures will be sensitive to the fluorescent excitation wavelengths and will emit a fluorescent relaxation wavelength. After the fluorescent excitation wavelength is emitted, the fluorescent relaxation wavelength will be sensed by the pixel array 122. The data sensed by the pixel array 122 results in a fluorescent exposure frame. The fluorescent exposure frame can be combined with a plurality of other exposure frames to form an image frame. The data in the fluorescent exposure frame can be overlaid on an RGB image frame that includes data from red exposure frames, green exposure frames, and blue exposure frames.

[0072] In embodiments where the light deficient environment 112 includes structures, tissues, or other materials that emit a spectral response to certain partitions of the electromagnetic spectrum, the pulsing scheme can include emission of hyperspectral partitions of electromagnetic radiation to elicit a spectral response from structures, tissues, or other materials present in the light deficient environment 112. The spectral response includes emission or reflection of certain wavelengths of electromagnetic radiation. The spectral response can be sensed by the pixel array 122 and results in a hyperspectral exposure frame. The hyperspectral exposure frame can be combined with a plurality of other exposure frames to form an image frame. The data in the hyperspectral exposure frame can be overlaid on an RGB image frame that includes data from red exposure frames, green exposure frames, and blue exposure frames.

[0073] In one embodiment, the pulsing scheme includes emission of a laser mapping pattern or a tool tracking pattern. After the laser mapping pattern or tool tracking pattern is emitted, the reflected electromagnetic radiation sensed by the pixel array 122 results in a laser mapping exposure frame. The data in the laser mapping exposure frame can be provided to a corresponding system to identify, for example, distances between tools present in the light deficient environment 112, a three-dimensional surface topography of a scene in the light deficient environment 112, distances, sizes, or locations of structures or objects within the scene, etc. This data can be overlaid on an RGB image frame or otherwise provided to a user of the system.

[0074] The emitter 102 can be a laser emitter capable of emitting pulsed red 104 light for generating sensed red 105 data to identify red 114 elements within the light deficient environment 112. The emitter 102 can also be capable of emitting pulsed green 106 light for generating sensed green 107 data to identify green 116 elements within the light deficient environment. The emitter 102 can also be capable of emitting pulsed blue 108 light for generating sensed blue 109 data to identify blue 118 elements within the light deficient environment. The emitter 102 can also be capable of emitting pulsed electromagnetic radiation of a fluorescent excitation 110 wavelength for identifying fluorescent reagents 120 within the light deficient environment 112. Fluorescent reagents 120 are identified by exciting them with pulsed fluorescent excitation 110 light and then sensing a fluorescent relaxation 111 wavelength of the particular fluorescent reagent 120 (by the pixel array 122). The emitter 102 can emit pulsed red 104, pulsed green 106, pulsed blue 108, and pulsed fluorescent excitation 110 wavelengths in any desired order.

[0075] The pixel array 122 senses reflected electromagnetic radiation. Each of the sensed red 105, sensed green 107, sensed blue 109, and sensed fluorescent relaxation 111 data can be referred to as an "exposure frame." Each exposure frame is assigned a particular color partition or wavelength partition, with the assignment being based on the timing of the pulsed color or wavelength partition from the emitter 102. The combination of an exposure frame and the assigned particular color or wavelength partition can be referred to as a data set. Any given data set can be assigned a color based on prior information about the emitter, even if the pixels 122 are not color-specific.

[0076] For example, during operation, after pulsed red 104 light is pulsed in the light deficient environment 112, the pixel array 122 senses reflected electromagnetic radiation. The reflected electromagnetic radiation produces an exposure frame, and the exposure frame is categorized as sensed red 105 data because it corresponds in time to the pulsed red 104 light. The combination of the exposure frame and its indication that it corresponds in time to the pulsed red 104 light is a "data set." This process is repeated for each partition of electromagnetic radiation emitted by the emitter 102. The data created by the pixel array 122 includes sensed red 105 exposure frames that identify red 114 components in the light deficient environment and correspond in time to the pulsed red 104 light. The data also includes sensed green 107 exposure frames that identify green 116 components in the light deficient environment and correspond in time to the pulsed green 106 light. The data also includes sensed blue 109 exposure frames that identify blue 118 components in the light deficient environment and correspond in time to the pulsed blue 108 light. The data also includes sensed fluorescent relaxation 111 exposure frames that identify fluorescent reagents 120 and correspond in time to the pulsed electromagnetic radiation of the fluorescent excitation 110 wavelength.

[0077] In one embodiment, the three data sets representing the RED, GREEN, and BLUE electromagnetic pulses are combined to form a single image frame. Thus, the information in the red, green, and blue exposure frames is combined to form a single RGB image frame. One or more additional data sets representing other wavelength partitions can be overlaid on the single RGB image frame. The one or more additional data sets can represent, for example, fluorescence imaging in response to pulse excitation 110 wavelengths between 770 nm and 790 nm and between 795 nm and 815 nm.

[0078] It should be understood that the present disclosure is not limited to any particular color combination or any particular electromagnetic partition, and that any color combination or any electromagnetic partition can be used in place of RED, GREEN, and BLUE, such as cyan, magenta, and yellow; ultraviolet; infrared; any combination of the foregoing or any other color combination, including all visible wavelengths and non-visible wavelengths, without departing from the scope of the present disclosure. In the figure, the light deficient environment 112 to be imaged includes red 114, green 116, and blue 118 portions, and also includes a fluorescent reagent 120. As shown, the reflected light from the electromagnetic pulses contains only data for portions of the object having a particular color corresponding to the color partition of the pulse. These individual color (or color interval) data sets can then be used to reconstruct an image by combining the data sets at 126. The information in each of the multiple exposure frames (i.e., multiple data sets) can be combined by a controller 124, control unit, camera control unit, image sensor, image signal processing pipeline, or some other computing resource that can be configured to be able to process the multiple exposure frames and combine the data sets at 126. The data sets can be combined to generate a single image frame either within the endoscope unit itself or by some other processing resource off-site.

[0079] Figure 2is a system 200 for providing illumination to a light deficient environment such as for endoscopic imaging. The system 200 can be used in conjunction with any of the systems, methods, or devices disclosed herein. The system 200 includes an emitter 202, a controller 204, a jumper waveguide 206, a waveguide connector 208, a lumen waveguide 210, a lumen 212, and an image sensor 214 with accompanying optical components such as lenses. The emitter 202, which can generally be referred to as a "light source," generates light that travels through the jumper waveguide 206 and the lumen waveguide 210 to illuminate a scene at a distal end of the lumen 212. The emitter 202 can be used to emit electromagnetic energy of any wavelength, including visible wavelengths, infrared, ultraviolet, hyperspectral, fluorescence excitation, laser mapping pulse schemes, or other wavelengths. The lumen 212 can be inserted into a patient's body for imaging, such as during a procedure or examination. Light is output as shown by dashed lines 216. The image sensor 214 can be used to capture the scene illuminated by the light and display the scene to a physician or some other medical personnel. The controller 204 can provide control signals to the emitter 202 to control when illumination is provided to a scene. In one embodiment, the emitter 202 and the controller 204 are located within a camera control unit (CCU) or external control console to which an endoscope is connected. If the image sensor 214 includes a CMOS sensor, light can be periodically provided to the scene in a series of illumination pulses between readout periods of the image sensor 214 during so-called blanking periods. Thus, light can be pulsed in a controlled manner to avoid reading out image pixels in the pixel array of the image sensor 214.

[0080] In one embodiment, the lumen waveguide 210 includes one or more optical fibers. These optical fibers can be made of a low cost material such as plastic in view of disposal of the lumen waveguide 210 and / or other portions of the endoscope. In one embodiment, the lumen waveguide 210 is a single glass fiber with a diameter of 500 microns. The jumper waveguide 206 can be permanently attached to the emitter 202. For example, the jumper waveguide 206 can receive light from an emitter within the emitter 202 and provide the light to the lumen waveguide 210 at the location of the connector 208. In one embodiment, the jumper waveguide 106 includes one or more glass optical fibers. The jumper waveguide can include any other type of waveguide for directing light to the lumen waveguide 210. The connector 208 can selectively couple the jumper waveguide 206 to the lumen waveguide 210 and allow light within the jumper waveguide 206 to pass through the lumen waveguide 210. In one embodiment, the lumen waveguide 210 is directly coupled to the light source without any intervening jumper waveguide 206.

[0081] The image sensor 214 includes an array of pixels. In one embodiment, the image sensor 214 includes two or more arrays of pixels for generating three-dimensional images. The image sensor 214 can constitute two or more image sensors each having an independent array of pixels and can be operated independently of one another. The array of pixels of the image sensor 214 includes active pixels and optical black (“OB”) pixels or optically blind pixels. The active pixels can be transparent, “color-agnostic” pixels capable of sensing imaging data for any wavelength of electromagnetic radiation. The optical black pixels are read during a blanking period of the array of pixels when the array of pixels is “reset” or calibrated. In one embodiment, light is pulsed during the blanking period of the array of pixels when the optical black pixels are read. After the optical black pixels are read, the active pixels are read during a readout period of the array of pixels. The active pixels can be charged by electromagnetic radiation pulsed during the blanking period so that the active pixels are ready to be read by the image sensor during the readout period of the array of pixels.

[0082] Figure 2A complementary system hardware such as a special or general purpose computer. Implementations within the scope of the disclosure can also include physical and other non-transitory computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, implementations of the disclosure can include at least two distinct kinds of computer-readable media: computer storage media (devices) and transmission media.

[0083] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0084] A "network" is one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. In one specific embodiment, the sensors and camera control unit can be networked to communicate with each other, as well as with other components connected through the network to which they are connected. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmission media can include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0085] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received by way of network or data link can be buffered in RAM within a network interface module (e.g., a "NIC"), and then eventually transferred to computer system RAM and / or to non-volatile computer storage media (devices) at a computer system. RAM can also include solid state drives (SSDs or PCIx-based real-time memory tiering storage devices such as FusionIO). Thus, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0086] Computer-executable instructions comprise, for example, instructions and data which, when executed at a general purpose computer, a special purpose computer, or a special purpose processing device cause the computing device to perform a certain function or group of functions. The computer executable instructions can be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0087] Those skilled in the art will appreciate that the present disclosure can be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, hand-held devices, game devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above-mentioned computing devices can be provided by or located within an entity. The present disclosure can also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0088] In addition, where appropriate, functions described herein can be performed in one or more of hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, components can be referred to by different names. This document does not intend to distinguish between components that differ in name but not function.

[0089] Figure 2A FIG. 25 is a block diagram illustrating an example computing device 250. The computing device 250 can be used to execute various programs, such as those discussed herein. The computing device 250 can be used as a server, a client, or any other computing entity. The computing device 250 can perform various monitoring functions discussed herein, and can execute one or more applications, such as the applications described herein. The computing device 250 can be any of a variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, a camera control unit, a tablet computer, and the like.

[0090] The computing device 250 includes one or more processors 252, one or more memory devices 254, one or more interfaces 256, one or more mass storage devices 258, one or more input / output (I / O) devices 260, and a display device 280, all of which are coupled to a bus 262. The processor 252 includes one or more processors or controllers that execute instructions stored in the memory device 254 and / or the mass storage device 258. The processor 252 can also include various types of computer-readable media, such as cache memory.

[0091] Memory devices 254 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 264) and / or nonvolatile memory (e.g., read-only memory (ROM) 266). Memory devices 254 can also include rewritable ROM, such as flash memory.

[0092] Mass storage devices 258 include various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As Figure 2 As shown in FIG. 2B, a particular mass storage device is a hard disk drive 274. Various drives can also be included in mass storage devices 258 to enable reading from and / or writing to various computer- readable media. Mass storage devices 258 include removable media 276 and / or non-removable media.

[0093] I / O devices 260 include various devices that enable input and output of data and / or other information to and from computing device 250. Exemplary I / O devices 260 include digital imaging devices, electromagnetic sensors and transmitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

[0094] Display devices 280 include any type of device capable of displaying information to one or more users of computing device 250. Examples of display devices 280 include monitors, display terminals, video projection devices, and the like.

[0095] Interfaces 256 include various interfaces that enable computing device 250 to interact with other systems, devices, or computing environments. Exemplary interfaces 256 can include any number of different network interfaces 270, such as interfaces connecting local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interfaces include a user interface 268 and a peripheral device interface 272. Interfaces 256 can also include one or more user interface elements 268. Interfaces 256 can also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mice, trackpads, etc.), keyboards, and the like.

[0096] Bus 262 enables communications among the processor 252, the memory devices 254, the interfaces 256, the mass storage devices 258, and the I / O devices 260, and with other devices or components coupled to bus 262. Bus 262 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, and the like.

[0097] For illustrative purposes, the programs and other executable program devices shown herein are discrete blocks, although it is understood that such programs and devices can reside at various times in different memory devices of computing device 250, and are executed by processor 252. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and / or firmware. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to carry out one or more of the systems and procedures described herein.

[0098] Figure 3A An operational cycle of a sensor used in a rolling readout mode or during sensor readout 300 is shown. A frame readout period can begin at and can be represented by the vertical line 310. The readout period 302 is represented by the diagonal or slanted line. The sensor can read out row by row, with the top of the downward sloping edge being the top sensor row 312 and the bottom of the downward sloping edge being the bottom sensor row 314. The time between the last row readout and the next readout period can be referred to as the blanking period 316. It should be noted that some of the sensor pixel rows can be covered with a light shield (e.g., a metallic coating or any other substantially black layer of another material type). These covered pixel rows can be referred to as optical black rows 318 and 320. The optical black rows 318 and 320 can be used as inputs to a correction algorithm. As shown, these optical black rows 318 and 320 can be located on the top of the pixel array or the bottom of the pixel array or both the top and bottom of the pixel array. Figure 3A

[0099] Figure 3B A method of controlling the amount of electromagnetic radiation (e.g., light) that is exposed to a pixel so as to be integrated or accumulated by the pixel is shown. It should be understood that a photon is the basic particle of electromagnetic radiation. Photons are integrated, absorbed, or accumulated by each pixel and converted to an electrical charge or current. An electronic shutter or rolling shutter (shown by the dotted line 322) can be used to start the integration time by resetting the pixel. Light will then be integrated until the next readout period. The position of the electronic shutter 322 can be moved between two readout periods 302 in order to control the pixel saturation for a given amount of light. It should be noted that this technique allows the integration time to be constant between two different rows, but introduces a delay as one moves from the top row to the bottom row.

[0100] Figure 3C A case where the electronic shutter 322 has been removed is shown. In this configuration, the integration of incident light can begin during the readout period 302 and can end at the next readout period 302, which also defines the beginning of the next integration.

[0101] Figure 3D ​A configuration is shown without electronic shutter 322 but with a controlled and pulsed light 330 during the blanking period 316. This ensures that all rows see the same light emanating from the same light pulse 330. In other words, each row will start its integration in a dark environment, which can be located at the optical black trailing row 320 of the readout frame (m) for maximum light pulse width, and will receive the light pass and will end its integration in a dark environment, which can be located at the optical black leading row 318 of the next subsequent readout frame (m+1) for maximum light pulse width. In, for example Figure 3D The image generated by the light pulse will be available only during the frame (m+1) readout without interfering with frame (m) and frame (m+2). It should be noted that the condition for having the light pulse read out in only one frame and not interfering with adjacent frames is to fire the given light pulse during the blanking period 316. Because the optical black rows 318, 320 are not sensitive to light, the optical black trailing row 320 time of frame (m) and the optical black leading row 318 time of frame (m+1) can be added to the blanking period 316 to determine the maximum range of the firing time of the light pulse 330.

[0102] As shown in Figure 3A The sensor can cycle multiple times to receive data for each pulsed color or wavelength (e.g., red, green, blue, or other wavelengths on the electromagnetic spectrum) as shown in

[0103] Figure 4A The operation of an embodiment of an electromagnetic emitter is shown graphically. The emitter can be timed to correspond to the cycling of the sensor, such that electromagnetic radiation is emitted during the sensor operation cycle and / or a portion of the sensor operation cycle. Figure 4A Pulse 1 at 402, pulse 2 at 404, and pulse 3 at 406 are shown. In an embodiment, the emitter can pulse during the readout period 302 of the sensor operation cycle. In an embodiment, the emitter can pulse during the blanking period 316 of the sensor operation cycle. In an embodiment, the emitter can pulse for a duration that is during a portion of two or more sensor operation cycles. In an embodiment, the emitter can start the pulse during the blanking period 316 or during the optical black portion 320 of the readout period 302 and end the pulse during the readout period 302 or the optical black portion 318 of the readout period 302 of the next subsequent cycle. It should be understood that any combination of the above is intended to fall within the scope of the present disclosure so long as the pulse of the emitter and the cycling of the sensor correspond.

[0104] Figure 4B The duration and magnitude of the emitted electromagnetic pulses (e.g., pulse 1 at 412, pulse 2 at 414, pulse 3 at 416) are represented graphically to control the exposure. An emitter with a fixed output magnitude can be pulsed at longer time intervals to provide more electromagnetic energy to the pixels, or the emitter can be pulsed at shorter time intervals to provide less electromagnetic energy. Whether longer or shorter time intervals are needed depends on the operating conditions. Figure 3D and Figure 4A The pulses are of a certain interval during any of the cycles described to provide the required electromagnetic energy to the pixel array. An emitter with a fixed output magnitude can be pulsed at longer time intervals to provide more electromagnetic energy to the pixels, or the emitter can be pulsed at shorter time intervals to provide less electromagnetic energy. Whether longer or shorter time intervals are needed depends on the operating conditions.

[0105] The magnitude of the emission itself can be increased compared to adjusting the time interval of the emitter pulse fixed output magnitude to provide more electromagnetic energy to the pixels. Similarly, decreasing the magnitude of the pulse can provide less electromagnetic energy to the pixels. It should be noted that embodiments of the system can have the ability to adjust both the magnitude and the duration if needed. Additionally, the sensor can be adjusted to increase its sensitivity and duration as needed for optimal image quality. Figure 4B The magnitude and duration of the pulses are shown to change. In the illustration, pulse 1 at 412 has a higher magnitude or intensity than pulse 2 at 414 or pulse 3 at 416. Additionally, pulse 1 at 412 has a shorter duration than pulse 2 at 414 or pulse 3 at 416 such that the electromagnetic energy provided by the pulse is shown by the area under the pulse shown in the illustration. In the illustration, pulse 2 at 414 has a relatively low magnitude or intensity and a longer duration when compared to pulse 1 at 412 or pulse 3 at 416. Finally, in the illustration, pulse 3 at 416 has an intermediate magnitude or intensity and duration when compared to pulse 1 at 412 and pulse 2 at 414.

[0106] Figure 5 is a graphical representation of the operation cycle, electromagnetic emitter, and emitted electromagnetic pulses to display embodiments of the present disclosure of an imaging system in accordance with the principles and teachings of the present disclosure Figures 3A to 3D and Figure 4A , Figure 4B The dashed lines in the figure represent the electromagnetic radiation pulses (from the electromagnetic emitter 306) that are emitted during the blanking period 316 of the image sensor 302. The electromagnetic radiation pulses are emitted primarily during the blanking period 316 of the image sensor 302, but can overlap with the readout period 302 of the image sensor 302. Figure 5 The dashed lines in the figure represent the electromagnetic radiation pulses (from the electromagnetic emitter 306) that are emitted during the blanking period 316 of the image sensor 302. The electromagnetic radiation pulses are emitted primarily during the blanking period 316 of the image sensor 302, but can overlap with the readout period 302 of the image sensor 302. Figure 4A The dashed lines in the figure represent the electromagnetic radiation pulses (from the electromagnetic emitter 306) that are emitted during the blanking period 316 of the image sensor 302. The electromagnetic radiation pulses are emitted primarily during the blanking period 316 of the image sensor 302, but can overlap with the readout period 302 of the image sensor 302.

[0107] An exposure frame includes data read by the pixel array of the image sensor during a readout period 302. The exposure frame can be combined with an indication of what type of pulse was emitted by the emitter prior to the readout period 302. The combination of the exposure frame and the indication of the type of pulse can be referred to as a data set. Multiple exposure frames can be combined to generate a black and white or RGB color image. Additionally, hyperspectral imaging data, fluorescence imaging data, and / or laser mapping imaging data can be overlaid on the black and white or RGB image.

[0108] In one embodiment, the exposure frame is data sensed by the pixel array during a readout period 302 that occurs after a blanking period 316. The emission of electromagnetic radiation is emitted during the blanking period 316. In one embodiment, a portion of the emission of electromagnetic radiation is overlaid with the readout period 302. The blanking period 316 occurs when the optical black pixels of the pixel array are read, and the readout period 302 occurs when the active pixels of the pixel array are read. The blanking period 316 can be overlaid with the readout period 302.

[0109] Figure 6A and Figure 6B A process for recording image frames is shown. Multiple image frames can be strung together to generate a video stream. A single image frame can include data from multiple exposure frames, where an exposure frame is data sensed by the pixel array after an emission of electromagnetic radiation. Figure 6A A conventional process is shown that is typically implemented with a color image sensor that has a color filter array (CFA) to filter out certain wavelengths of light for each pixel. Figure 6B is the process disclosed herein, and can be implemented with a monochrome “color agnostic” image sensor that is capable of receiving electromagnetic radiation of all wavelengths.

[0110] Figure 6A The process shown occurs from time t(0) to time t(l). The process first emits white light 602 and senses white light 604. At 606, an image is processed and displayed based on the sensing at 604.

[0111] Figure 6BThe illustrated process occurs from time t(0) to time t(l). The process first emits green light 612 and senses reflected electromagnetic radiation 614 after emitting green light 612. The process continues to emit red light 616 and senses reflected electromagnetic radiation 618 after emitting red light 616. The process continues to emit blue light 620 and senses reflected electromagnetic radiation 622 after emitting blue light 620. The process continues with one or more emissions of a fluorescent excitation wavelength 624 and senses reflected electromagnetic energy 626 after each of the one or more emissions of the fluorescent excitation wavelength of electromagnetic radiation 624. At 628, an image is processed and displayed based on each of the sensed reflected electromagnetic energy instances 614, 618, 622, and 626.

[0112] Figure 6B The illustrated method provides higher resolution images and provides a means for generating RGB images that also include fluorescent imaging data. When using partitioned spectral light (as Figure 6B illustrated), the sensor can be made sensitive to electromagnetic energy of all wavelengths. In Figure 6B the illustrated process, the monochromatic pixel array is instructed to sense electromagnetic energy from a predetermined partition of full-spectrum electromagnetic energy in each cycle. Thus, to form an image, the sensor only needs to cycle through multiple different partitions within the full-spectrum light. The final image is combined based on the multiple cycles. Because the image from each color partition frame cycle has higher resolution (compared to a CFA pixel array), the resulting image formed when the partitioned light frames are combined also has higher resolution. In other words, because every pixel within the array (not every other pixel, as in a sensor with a CFA) is sensing the energy value of a given pulse and a given scene, a higher resolution image is formed for each scene only a time period apart.

[0113] As can be seen graphically between times t(0) and t(l) in the embodiments illustrated in Figure 6A and Figure 6B , the sensor for the partitioned spectral system in Figure 6B cycles at least four times per each of the full-spectrum systems in Figure 6A In one embodiment, the display device (LCD panel) operates at a speed of 50-60 frames per second. In such an embodiment, the partitioned light system in Figure 6B can operate at a rate of 200-240 frames per second to maintain continuity and smoothness of the displayed video. In other embodiments, there can be different capture and display frame rates. Furthermore, the average capture rate can be any multiple of the display rate.

[0114] In one embodiment, it can be desirable that not all partitions are represented equally within the system frame rate. In other words, not all light sources must be pulsed with the same regularity in order to emphasize and de-emphasize aspects of the recorded scene as desired by the user. It should also be understood that the non-visible and visible partitions of the electromagnetic spectrum can be pulsed together within the system, where their respective data values are stitched into the video output for display to the user.

[0115] Embodiments can include the following pulsed cycle pattern:

[0116] i. green pulse;

[0117] ii. red pulse;

[0118] iii. blue pulse;

[0119] iv. green pulse;

[0120] v. red pulse;

[0121] vi. blue pulse;

[0122] vii. fluorescent excitation pulse;

[0123] viii. (repeat)

[0124] Embodiments can include the following pulsed cycle pattern:

[0125] i. brightness pulse;

[0126] ii. red chrominance pulse;

[0127] iii. brightness pulse;

[0128] iv. blue chrominance pulse;

[0129] v. fluorescent excitation pulse;

[0130] vi. (repeat)

[0131] Embodiments can include the following pulsed cycle pattern:

[0132] i. brightness pulse;

[0133] ii. red chrominance pulse;

[0134] iii. brightness pulse;

[0135] iv. blue chrominance pulse;

[0136] v. brightness pulse;

[0137] vi. red chrominance pulse;

[0138] vii. a brightness pulse;

[0139] viii. a blue chrominance pulse;

[0140] ix. a fluorescence excitation pulse;

[0141] x. (repeat)

[0142] As can be seen in this example, the fluorescence excitation partition can pulse at a different rate than the other partition pulses. Doing so can emphasize a certain aspect of the scene, where fluorescence imaging data is only overlaid with other data in the video output to make the desired emphasis. It should be noted that adding a fluorescence partition on top of the red, green, and blue partitions does not necessarily require a sequenced system to operate at four times the rate of a full-spectrum non-sequential system, as each partition does not have to be equally represented in the pulse pattern. As seen in this embodiment, adding a partition pulse that is represented less in the pulse pattern (fluorescence excitation in the above example) will result in less than a 20% increase in the cycling speed of the sensor to accommodate the irregular partition sampling.

[0143] In various embodiments, the pulse cycling pattern can also include any of the following wavelengths in any suitable order. Such wavelengths can be particularly suitable for exciting a fluorescence reagent to produce fluorescence imaging data by sensing a relaxation emission of the fluorescence reagent based on a relaxation emission of the fluorescence reagent:

[0144] i. 770 ± 20 nm;

[0145] ii. 770 ± 10 nm;

[0146] iii. 770 ± 5 nm;

[0147] iv. 790 ± 20 nm;

[0148] v. 790 ± 10 nm;

[0149] vi. 790 ± 5 nm;

[0150] vii. 795 ± 20 nm;

[0151] viii. 795 ± 10 nm;

[0152] ix. 795 ± 5 nm;

[0153] x. 815 ± 20 nm;

[0154] xi. 815 ± 10 nm;

[0155] xii. 815 ± 5 nm;

[0156] xiii. 770 nm to 790 nm; and / or

[0157] xiv. 795 nm to 815 nm.

[0158] The partition cycles can be divided to accommodate or approximate various imaging and video standards. In one embodiment, the partition cycle includes pulses of electromagnetic energy in the red, green, and blue spectra as best shown in Figures 7A to 7D the timing relationship between the emission of pulses of electromagnetic radiation by the emitters and the readout of the pixel array is shown in further detail in Figures 7A to 7D

[0159] In one embodiment, the emitters emit one or more hyperspectral emissions to elicit a spectral response. The hyperspectral emissions include one or more of electromagnetic radiation having a wavelength of about 513-545 nm, about 565-585 nm, and / or about 900-1000 nm. In such embodiments, the coherent light source 802 includes at least one laser emitter for the 513-545 nm partition, at least one laser emitter for the 565-585 partition, and at least one laser emitter for the 900-1000 nm partition. It will be appreciated that additional hyperspectral emissions for eliciting a spectral response can be emitted without departing from the scope of the present disclosure.

[0160] In one embodiment, the emitters emit one or more fluorescence excitation emissions to cause a reagent to fluoresce. The fluorescence excitation emissions include one or more of electromagnetic radiation having a wavelength of about 460-470 nm, 529-537 nm, 633-643 nm, 775-785 nm, 800-810 nm, 970-980 nm, 575-579 nm, 519-527 nm, 770-790 nm, and / or 795-815 nm. In such embodiments, the coherent light source 802 can include at least one laser emitter for each of the aforementioned partitions of electromagnetic radiation. It will be appreciated that additional fluorescence excitation emissions for causing a reagent to fluoresce can be emitted without departing from the scope of the present disclosure.

[0161] In Figure 7A different light intensities have been achieved by modulating the light pulse width or duration within the working range shown by the vertical gray dashed lines. Figure 7A The general timing relationship between the mixing of pulses of three wavelengths within a four-frame cycle and the readout cycle of a pixel array of an image sensor is shown. In one embodiment, there are three monochromatic pulsed light sources under the control of a controller. A periodic sequence of monochromatic red, monochromatic green, and monochromatic blue exposure frames are captured, for example, with an R-G-B-G pulse pattern, and combined into an sRGB image frame by an image signal processor chain.

[0162] In Figure 7B ​In this process, different light intensities are achieved by modulating the power of the optical power or the power of the electromagnetic transmitter (which can be a laser or an LED transmitter), while keeping the pulse width or duration constant.

[0163] Figure 7C This illustrates a scenario where both optical power and pulse width are modulated for greater flexibility. Partition cycling can utilize cyan, magenta, yellow (CMY), infrared, ultraviolet, hyperspectral, and fluorescence, employing invisible pulse sources mixed with visible pulse sources, and any other color space required to generate the image, or approximating any currently known or yet-to-be-developed desired video standard. It should also be understood that the system can switch between operating color spaces to provide the desired image output quality.

[0164] Using the color space green-blue-green-red (e.g.) Figure 7D In the implementation shown, it may be desirable to pulse the luminance component more frequently than the chromaticity component, because users are generally more sensitive to differences in light intensity than to differences in light color. Examples of such implementations include... Figure 7D The monochrome image sensor shown utilizes this principle. Figure 7D In this configuration, green, which contains the most luminance information, can pulse more frequently or have greater intensity in the (GBGRGBGR…) scheme to obtain luminance data. This configuration will create a video stream with noticeably more detail, without creating and transmitting imperceptible data.

[0165] In one implementation, all three light sources are pulsed synchronously with light energy modulated to provide pure luminance information within the same exposure frame. The light energy can be modulated according to color conversion coefficients from the RGB color space to the YCbCr color space. It should be understood that the color conversion can be implemented according to any suitable standard such as ITU-R BT.709HD, ITU-R BT.601, ITU-R BT.2020, or any other suitable standard or formula. This conversion can be performed according to the ITU-R BT.709HD standard as follows:

[0166]

[0167] In addition to modulating luminance information, full-color images also require red and blue chrominance components. However, algorithms applied to the luminance component cannot be directly applied to the chrominance component because the algorithm is signed, as reflected in the fact that some RGB coefficients are negative. In one implementation, the luminance is increased to such that all final pulse energies are positive. As long as the color fusion processes in the image signal processor know the composition of the chrominance exposure frames, they can decode them by subtracting an appropriate amount of luminance from adjacent frames. The pulse energy ratio is given by the following formula:

[0168] Y = 0.183 R + 0.614 G + 0.062 B

[0169] Cb = λ Y - 0.101 R - 0.339 G + 0.439 B

[0170] Cr = δ Y + 0.439 R - 0.399 G - 0.040 B

[0171] where

[0172]

[0173]

[0174] If the lambda factor is equal to 0.552, the red and green components are cancelled out. In this case, the blue chrominance information can be provided with pure blue light. Similarly, if the delta factor is equal to 0.650, the blue and green components are cancelled out, and the red chrominance information can be provided with pure red light. This embodiment is a convenient approximation of digital frame reconstruction.

[0175] In an embodiment where white balancing is performed in the illumination domain, in addition to the white balancing modulation, a modulation is applied.

[0176] In one embodiment, the pulses that replicate the weaker partitions can be used to produce an output that has been adjusted for the weaker pulses. For example, blue lasers are considered weak with respect to the sensitivity of the silicon-based pixels, and it is difficult to produce blue light as compared to red or green light, so the blue lasers can be pulsed more frequently during the frame cycle to compensate for the weakness of the light. These additional pulses can be done continuously over time, or by pulsing multiple lasers simultaneously to produce the desired compensation effect. It should be noted that by pulsing during a blanking period (time during which the sensor does not read out the pixel array), the sensor is not sensitive to the differences / mismatches between lasers of the same kind, and simply accumulates the light for the desired output. In another embodiment, the range of maximum light pulses can be different from frame to frame. This is shown in Figure 7E , where the light pulses are different from frame to frame. The sensor can be constructed to be able to program different blanking periods in a repeating pattern of 2 frames or 3 frames or 4 frames or n frames.

[0177] In Figure 7E , four different light pulses are shown, and pulse 1 can be repeated, for example, after pulse 4 and can have a pattern of four frames with different blanking periods. This technique can be used to place the partition with the maximum power on the smallest blanking period, and thus allow the weakest partition to have a wider pulse on one of the subsequent frames without increasing the readout speed. The reconstructed frame can still have a regular pattern from frame to frame, as it is composed of many pulse frames.

[0178] Figure 8 An embodiment of a system 800 for endoscopic imaging is illustrated. The system includes a handheld component 802 and an endoscope tube 804, which are freely rotatable relative to each other, such as... Figure 8 As shown. System 800 may also include a cable 806 attached to the handheld device 802 and an optical fiber 810 attached to the endoscope tube 804. System 800 includes an image sensor 808, which may be located within the handheld device 802 or at the distal end of the endoscope tube 804, such as... Figure 8 As shown in the image.

[0179] The handheld device 802 is configured to be held by a user or robotic system during system 800 use. The endoscope tube 804 is configured to be inserted into the environment, allowing the image sensor 808 to capture exposure frames of that environment. In a medical implementation, for example, the endoscope tube 804 is inserted into a body cavity while the handheld device 802 is held and operated by a physician. The system 800 can be electrically communicated with a display, allowing the user to view a video stream depicting images captured in real time by the image sensor 808.

[0180] In some cases, it is desirable for the horizontal line of the image on the video stream to remain constant during operation, even when the handheld device 802 is rotated. Alternatively, the user may expect the image orientation in the video stream to change when the handheld device 802 is rotated. In each case, some post-processing may be performed on the image captured by the image sensor 808 to ensure that the image orientation reflects the orientation of the handheld device 802 relative to the scene.

[0181] In system 800, image sensor 808 is rigidly coupled to a lens stack, and each of image sensor 808 and lens stacks is rigidly coupled to endoscope tube 804. Endoscope tube 804 may consist of a single tube. The orientation of the image captured by the image sensor can be rotated during post-processing in an image signal processing pipeline (ISP). The ISP calculates a digital representation of the angle of endoscope tube 804 relative to a handheld device 802 that is continuously available during operation.

[0182] In Figure 8 In different alternative embodiments of the illustrated system 800, the image sensor is rigidly coupled to a handheld device, such that rotating the handheld device also rotates the lens stack at the distal end of the endoscope tube. This can be achieved by combining two concentric tubes. This alternative embodiment allows the distal prism to rotate, and this changes the user's viewing angle while the image sensor remains in a constant position. This allows the device to be used in the same way as expected by users or operators experienced in using conventional rigid endoscope systems. The user or operator can rotate the external cavity, thereby changing the viewing angle while the image sensor remains in a fixed position. The video stream then maintains a constant horizontal line. The prism can rotate, while the image sensor does not, so that the user does not lose orientation.

[0183] Figure 9 Embodiments of a rotation sensing system 900 are shown in use in conjunction with an endoscope imaging system such as the systems 200, 800 shown in Figure 2 and Figure 8 Embodiments of a rotation sensing system 900 are shown in use in conjunction with an endoscope imaging system such as the systems 200, 800 shown in

[0184] The rotation sensor 918 can be any suitable rotation sensor known in the art. In one embodiment, the rotation sensor 918 is a rotation-detecting Hall effect sensor. In such embodiments, the rotation sensor 918 can be located within the handpiece 802 of the endoscope. Additionally, in such embodiments, the rotation sensing system 900 can also include a radially polarized magnetic ring 912. The rotation sensor 918 generates a voltage 914, and this voltage 914 can be used to detect the rotation angle of the radially polarized magnetic ring 912. Thus, the voltage 914 generated by the rotation sensor 918 can be used to calculate the rotation angle of the handpiece 802 relative to the endoscope tube 804 or lumen.

[0185] The voltage 914 is fed to an analog-to-digital converter (ADC) 916. Alternatively, the ADC 916 can be a digitizer. The digital number representing the voltage 914 is transmitted to an image signal processing (ISP) pipeline or camera processing pipeline. In one embodiment, the ISP pipeline and / or camera processing pipeline is integrated in a controller in electrical communication with the transmitter and image sensor. In alternative embodiments, the ISP pipeline and / or camera processing pipeline is independent of the controller.

[0186] In one embodiment, the ISP pipeline (or alternatively, the camera processing pipeline) calculates an image rotation transform. The image rotation transform can be applied to an exposure frame captured by the image sensor. In one embodiment, the image rotation transform is applied to an image frame (such as an RGB or YCbCr image frame) that includes data from multiple exposure frames, and it should be understood that this embodiment encompasses the concept of applying the image rotation transform to an exposure frame. The image rotation transform is calculated based on the rotation angle detected by the rotation sensor 918. The ISP pipeline can then rotate the exposure frame inversely relative to the rotation angle to maintain a constant image horizon for the scene being imaged by the image sensor.

[0187] In one embodiment, computing the image rotation transform includes identifying integer (x,y) coordinates of pixel data in an exposure frame (or image frame) captured by pixels of a pixel array. This can be computed for each of a plurality of pixels in the pixel array. The computation also includes applying a rotation kernel to the integer (x,y) coordinates to transform the integer (x,y) coordinates to real number pixel coordinates. Again, this can be performed for each of a plurality of pixels in the pixel array. The computation also includes truncating the real number pixel coordinates to integer values. If there is blank pixel data (i.e., pixels that do not produce any data) in the exposure frame, values can be assigned to the blank pixel data using nearby filled pixel data. The assigned values can be computed using one or more of nearest neighbor substitution, bilinear interpolation, or bicubic interpolation.

[0188] In one embodiment, computing the image rotation transform includes identifying integer (x,y) coordinates of pixel data in an exposure frame (or image frame) captured by pixels of a pixel array. This can be computed for each of a plurality of pixels in the pixel array or each of a plurality of “padding pixels” in the pixel array that generate “padding data” that influences the exposure frame. The computation also includes applying an inverse rotation kernel to the integer (x,y) coordinates to transform the integer (x,y) coordinates to real number (x,y) coordinates. Again, this can be performed for each of a plurality of pixels in the pixel array. The computation also includes estimating a pixel value at the real number (x,y) coordinates using data from one or more nearest integer coordinate locations. The estimation can be performed using one or more of nearest neighbor substitution, bilinear interpolation, or bicubic interpolation.

[0189] Figure 10 An alternative embodiment of a rotation sensing system 1000 is shown in use in conjunction with an endoscope imaging system such as the systems 200, 800 shown in Figure 2 and Figure 8 The rotation sensing system 1000 includes a potentiometer 1020. The potentiometer 1020 includes a carbon track or carbon wire 1022 that can be rigidly attached to the endoscope tube 804. The potentiometer 1020 can also include a wiper 1024 that can be rigidly attached to the handpiece 802. The resistance between the potentiometer 1020 and the wiper 1024 can be measured to compute the angle of rotation of the endoscope tube 804 relative to the handpiece 802. In one embodiment, a possible dispenser arrangement is used for the voltage at the wiper 1024 to provide an angle measurement. In one embodiment, the potentiometer 1020 also includes a carbon wire disposed within the lumen of the endoscope.

[0190] The rotation angle of the endoscope tube 804 relative to the handheld device 802 can be calculated based on the resistance between the potentiometer 1020 and the wiper 1024. Image rotation transformation can be calculated based on the rotation angle. This can be based on the combination of... Figure 9 The same steps discussed are used to calculate the image rotation transformation.

[0191] Figure 11 It shows the use of endoscopic imaging systems such as Figure 2 and Figure 8 The illustration shows an embodiment of a rotation sensing system 1100 used in conjunction with systems 200 and 800. The rotation sensing system 1100 includes a light source 1126 and a photosensor 1128. The light source 1126 may include any suitable light source, such as a light-emitting diode (LED) light source. The photosensor 1128 may be a photodiode or a phototransistor. In one embodiment, the light source 1126 and the photosensor 1128 are integrated into or attached to a handheld device 802. The rotation sensing system 1100 may also include a continuous loop 1130 or a disc.

[0192] In one embodiment, the continuous annular band 1130 is rigidly attached to the endoscope tube 804. The continuous annular band 1130 may have different degrees of reflectivity, such as... Figure 11 As shown. Different levels of reflectivity can be achieved using a series of mirrors with different levels of reflectivity. Each mirror in this series can have the same size, such that the continuous ring 1130 includes portions with varying reflectivity at regular intervals. The reflectivity of the continuous ring 1130 can vary linearly with the angle of the endoscope tube 804 relative to the handpiece 802. In such embodiments, the rotation angle of the endoscope tube 804 relative to the handpiece 802 is measured based on the amount of light reflected from the light source 1126 from the continuous ring 1130 and returning to the photodetector 1128.

[0193] Image rotation transformations can be calculated based on the rotation angle. This can be based on a combination of... Figure 9 The same steps discussed are used to calculate the image rotation transformation.

[0194] Figure 12A and Figure 12B An embodiment of a system 1200 for endoscopic imaging is shown. System 1200 includes a fixed handpiece 802 having a cylindrical opening at its distal end. System 1200 includes an endoscope tube 804. The cylindrical opening of the endoscope tube 804 is confined in the axial direction but allows rotation about the axis. (Except relative to...) Figure 8In addition to the components mentioned, the system 1200 also includes an interface component 1202 that can be affixed to the handpiece 802, a rotating sleeve 1204, a rotating column 1206, support electronics and circuitry 1208 for the image sensor 808, a sensor harness 1210, a lens stack 1212, and a lens holder 1214. The lens stack 1212 includes a distal prism that is distal to the endoscope tube 804.

[0195] Figure 12B An embodiment is shown in which the combination of the rotating sleeve 1204 and the rotating column 1206 are used to axially constrain the endoscope tube 804. There can be additional material between the interface component 1202 and the rotating sleeve 1204 to increase or decrease friction, thus achieving a torque that is low enough to be ergonomic and pleasant to feel, but also high enough to prevent accidental rotation.

[0196] In one embodiment, the rotating column 1206 allows the user to rotate the endoscope tube 804 in a manner similar to rotating a conventional rigid endoscope. When the rotating column 1206 is rotated, the entire endoscope assembly is also rotated, including the distal imaging sensor 808 and the attached lens stack 1212. As can be seen, the angle of view determined by the distal prism is changed and the user is allowed to obtain a broader or different view of the scene.

[0197] For each embodiment of the system shown in Figures 8 to 12B For each embodiment of the system shown in

[0198] The digital angle information can be used in the image signal processing (ISP) pipeline, where the digital angle information is periodically sampled (e.g., every exposure frame and / or every image frame) and appropriately quantized to, for example, 5° or 10° units. To prevent the final image from rapidly oscillating in angle between adjacent angles, some degree of hysteresis can be required. One approach is to only allow the image transform if the same quantized angle is consistently observed over the previous n samples, where n will be adjusted to the user’s satisfaction.

[0199] The image plane transformation by an image rotation by an angle Θ can be computed based on the following transformations:

[0200] x2 = (X1 - x0)cosΘ - (Y1 - y0)sinΘ + x0

[0201] y2 = (Y1 - y0)cosΘ + (X1 - x0)sinΘ + y0

[0202] where (X1, Y1) is the initial integer pixel coordinate, (x2, y2) is the final real pixel coordinate, and (x0, y0) marks the axis of rotation. In general, x2 and y2 are not integers unless Θ is a multiple of 90°. The (x2, y2) values can be truncated or rounded to integer coordinates (X2, Y2) to fill the pixel positions in the final image buffer by:

[0203] X2 = int(x2)

[0204] Y2 = int(y2)

[0205] In some cases, this method results in multiple candidate cases and void (i.e., blank) pixels. The void pixels can be filled by nearest-neighbor substitution or by interpolation (e.g., bilinear or bicubic), which requires an occupancy survey of their locations.

[0206] In one embodiment, an inverse rotation transform is applied to the final integer pixel positions to obtain real coordinates in the initial plane, as follows:

[0207] x1 = (X2 - x0) cos Θ + (Y2 - y0) sin Θ + x0

[0208] y1 = (Y2 - y0) cos Θ - (X2 - x0) sin Θ + y0

[0209] An interpolated image content estimate can be computed, since the pixel data in this plane is known to be all integer coordinates. This interpolation can likewise be a bilinear or bicubic interpolation. A bilinear interpolation requires only knowledge of the four closest pixels (two in each dimension). These pixels are identified as (X a ,Y a ), (X a ,Y b ), (X b ,Y a ), and (X b ,Y b ), where:

[0210] X a = int(x1); X b = 1 + int(x1)

[0211] Y a = int(y1); Y b = 1 + int(y1)

[0212] The convolution kernel is described by:

[0213]

[0214] where

[0215] a = xi - x a

[0216] b = yi - y a

[0217] In pixels.

[0218] Figures 13A to 13C Each illustrates a light source 1300 having multiple emitters. The light sources 1300 can be collectively referred to herein as “emitters.” The multiple emitters include a first emitter 1302, a second emitter 1304, and a third emitter 1306. Additional emitters can be included, as discussed further below. The emitters 1302, 1304, and 1306 can include one or more laser generators that emit light having different wavelengths. For example, the first emitter 1302 can emit a wavelength consistent with a blue laser, the second emitter 1304 can emit a wavelength consistent with a green laser, and the third emitter 1306 can emit a wavelength consistent with a red laser. For example, the first emitter 1302 can include one or more blue lasers, the second emitter 1304 can include one or more green lasers, and the third emitter 1306 can include one or more red lasers. The lasers 1302, 1304, 1306 emit laser beams toward a collection region 1308, which can be a waveguide, a lens, or a location of other optical components that provide light to a waveguide (such as the jumper waveguide 206 or the internal cavity waveguide 210 of FIG. 2) or other optical components. Figure 2 The jumper waveguide 206 or the internal cavity waveguide 210 of FIG. 2, for example.

[0219] In one implementation, the emitters 1302, 1304, and 1306 emit electromagnetic radiation at hyperspectral wavelengths. Certain hyperspectral wavelengths can penetrate tissue and enable a medical practitioner to “see through” tissue in the foreground to identify chemical processes, structures, compounds, biological processes, etc. located behind the tissue in the foreground. The hyperspectral wavelengths can be specifically selected to identify particular diseases, tissue conditions, biological processes, chemical processes, tissue types, etc. that are known to have certain spectral responses.

[0220] In implementations in which an agent or dye has been administered to a patient that aids in identifying certain tissues, structures, chemical reactions, biological processes, etc., the emitters 1302, 1304, and 1306 can emit wavelengths that are used to cause the agent or dye to fluoresce. Such wavelengths can be determined based on the agent or dye administered to the patient. In such implementations, the emitters can need to be highly precise in order to emit the required wavelengths to cause certain agents or dyes to fluoresce or activate.

[0221] In one implementation, emitters 1302, 1304, and 1306 emit a laser mapping pattern for mapping the topography of a scene and / or for calculating the size and distance between objects in a scene. In one embodiment, an endoscopic imaging system is used in conjunction with multiple tools, such as a scalpel, a retractor, a clamp, etc. In such an embodiment, each of emitters 1302, 1304, and 1306 can emit a laser mapping pattern such that the laser mapping pattern is projected onto each tool individually. In such an embodiment, the laser mapping data for each of the tools can be analyzed to identify the distance between the tools and other objects in the scene.

[0222] In Figure 13B embodiments, emitters 1302, 1304, 1306 each deliver laser light to collection region 1308 at different angles. The variation in angles can result in a variation in the location of the electromagnetic energy in the output waveguide. For example, if the light enters a fiber bundle (glass or plastic) immediately at collection region 1308, the varying angles can result in different amounts of light entering different fibers. For example, the angles can result in a variation in intensity across collection region 1308. In addition, the light from the different emitters can not be mixed uniformly, so some fibers can receive different amounts of different colors of light. The variation in color or intensity of the light in different fibers can result in a non-optimal illumination of the scene. For example, the variation in delivered light or light intensity can result at the scene and the captured image.

[0223] In one embodiment, an intervening optical element can be placed between the fiber bundle and emitters 1302, 1304, 1306 to mix the different colors (wavelengths) of light before entering the fibers or other waveguides. Exemplary intervening optical elements include a diffuser, a mixing rod, one or more lenses, or other optical components for mixing the light so that a given fiber receives the same amount of each color (wavelength) of light. For example, each fiber in the fiber bundle can have the same color. This mixing can result in the same color in each fiber, but, in some embodiments, can still result in different total brightness delivered to different fibers. In one embodiment, the intervening optical element can also spread or uniformly spread the light over the collection region so that each fiber carries the same total amount of light (e.g., the light can spread out in a top hat profile). A diffuser or mixing rod can result in a loss of light.

[0224] Although collection region 1308 is represented in Figure 13A as a physical component, collection region 1308 can simply be the region where light from emitters 1302, 1304, and 1306 is delivered. In some cases, collection region 1308 can include optical components, such as a diffuser, a mixing rod, a lens, or any other intervening optical component between emitters 1302, 1304, 1306 and the output waveguide.

[0225] Figure 13C An embodiment of a light source 1300 is shown with emitters 1302, 1304, 1306 providing light to a collection area 1308 at the same or approximately the same angle. The light is provided at an angle substantially normal to the collection area 1308. The light source 1300 includes a plurality of dichroic mirrors, including a first dichroic mirror 1310, a second dichroic mirror 1312, and a third dichroic mirror 1314. The dichroic mirrors 1310, 1312, 1314 include mirrors that reflect light of a first wavelength but transmit light of a second wavelength (or are transparent to it). For example, the third dichroic mirror 1314 can reflect blue laser light provided by the third emitter, while being transparent to the red and green light provided by the first emitter 1302 and the second emitter 1304, respectively. The second dichroic mirror 1312 can be transparent to light from the first emitter 1302, but reflective to light from the second emitter 1304. If other colors or wavelengths are included, the dichroic mirrors can be selected to reflect light corresponding to at least one emitter and be transparent to the other emitters. For example, the third dichroic mirror 1314 reflects light from the third emitter 1306, but is transparent to emitters behind it, such as the first emitter 1302 and the second emitter 1304. In embodiments where there are tens or hundreds of emitters, each dichroic mirror can reflect the corresponding emitter and the emitters in front of it, while being transparent to the emitters behind it. This can allow tens or hundreds of emitters to emit electromagnetic energy to the collection area 1308 at substantially the same angle.

[0226] Because these dichroic mirrors allow other wavelengths to be transmitted or pass through, each of these wavelengths can reach the collection area 1308 from the same angle and / or at the same center point or focal point. Providing light from the same angle and / or the same focal / center point can significantly improve reception and color mixing at the collection area 1308. For example, a particular fiber can receive different colors in the same proportion as they are transmitted / reflected by the emitters 1302, 1304, 1306 and the mirrors 1310, 1312, 1314. In contrast to embodiments of Figure 13B Light mixing can be significantly improved at the collection area compared to embodiments of

[0227] Figure 13CAn embodiment of a light source 1300 with emitters 1302, 1304, 1306 that also provide light to the collection region 1308 at the same or approximately the same angle is shown. However, the light incident on the collection region 1308 is vertically offset. The angle 1316 indicates the angle of the vertical offset. In one embodiment, the laser emitters 1302, 1304, 1306 can have a Gaussian cross-sectional intensity profile. As previously mentioned, improved distribution of light energy between the fibers can be achieved by forming a more flat or top-hat shaped intensity profile. In one embodiment, as the angle 1316 increases, the intensity across the collection region 1308 approaches a top-hat profile. For example, by increasing the angle 1316 until the profile is sufficiently flat, the top-hat profile can even approximate a non-flat output beam. The top-hat profile can also be achieved using one or more lenses, diffusers, mixing rods, or any other intervening optical components between the emitters 1302, 1304, 1306 and the output waveguide, fiber, or bundle of optical fibers.

[0228] Figure 14 is a schematic diagram showing a single optical fiber 1402 outputting at the output via a diffuser 1404. In one embodiment, the optical fiber 1402 has a diameter of 500 microns, a numerical aperture of 0.65, and emits a light cone 1406 of about 70 or 80 degrees without the diffuser 1404. With the diffuser 1404, the light cone 1406 can have an angle of about 110 or 120 degrees. The light cone 1406 can be the majority of the place where all the light reaches and is uniformly distributed. The diffuser 1404 can allow for a more uniform distribution of electromagnetic energy of the scene observed by the image sensor.

[0229] In one embodiment, the lumen waveguide 210 includes a single plastic or glass optical fiber of about 500 microns. Plastic fibers are less expensive, and their width can allow the fiber to carry a sufficient amount of light to the scene, but have coupling, diffusing, or other losses. For example, a smaller fiber can not be able to carry as much light or power as a larger fiber. The lumen waveguide 210 can include a single or multiple optical fibers. The lumen waveguide 210 can receive light directly from a light source or via a jumper waveguide. A diffuser can be used to widen the light output 206 to obtain a desired field of view of the image sensor 214 or other optical components.

[0230] While in Figures 13A to 13CThree emitters are shown, but in some embodiments, a number of emitters ranging from one to hundreds or more can be used. The emitters can have different wavelengths or spectrums of light that they emit, and these lights can be used to continuously cover a desired portion of the electromagnetic spectrum (e.g., the visible spectrum as well as the infrared and ultraviolet spectrums). The emitters can be configured to be able to emit visible light such as red, green, and blue light, and can also be configured to be able to emit hyperspectral emissions of electromagnetic radiation, fluorescence excitation wavelengths for causing reagents to fluoresce, and / or laser mapping patterns for calculating parameters and distances between objects in a scene.

[0231] Figure 15 A portion of the electromagnetic spectrum 1500 is shown that is divided into twenty different sub-spectrums. The number of sub-spectrums is merely exemplary. In at least one embodiment, the spectrum 1500 can be divided into hundreds of sub-spectrums, each having a small waveband. The spectrum can extend from the infrared spectrum 1502, through the visible spectrum 1504, and into the ultraviolet spectrum 1506. The sub-spectrums each have a waveband 1508 that covers a portion of the spectrum 1500. Each waveband can be defined by an upper wavelength and a lower wavelength.

[0232] Hyperspectral imaging includes imaging information from across the electromagnetic spectrum 1500. A hyperspectral pulse of electromagnetic radiation can include multiple sub-pulses across one or more portions of the electromagnetic spectrum 1500 or the entire electromagnetic spectrum 1500. A hyperspectral pulse of electromagnetic radiation can include a single wavelength bin of electromagnetic radiation. The resulting hyperspectral exposure frame includes information sensed by the pixel array after the hyperspectral pulse of electromagnetic radiation. Thus, a hyperspectral exposure frame can include data for any suitable binning of the electromagnetic spectrum 1500, and can include multiple exposure frames for multiple binning of the electromagnetic spectrum 1500. In one embodiment, the hyperspectral exposure frame includes multiple hyperspectral exposure frames such that the combined hyperspectral exposure frames include data for the entire electromagnetic spectrum 1500.

[0233] In one embodiment, at least one emitter, such as a laser emitter, is included in the light source, such as light source 202, 1300, for each sub-spectrum to provide complete and continuous coverage of the entire optical spectrum 1500. For example, a light source for providing coverage of the illustrated sub-spectra can include at least 20 different emitters, one for each sub-spectrum. In one embodiment, each emitter covers a 40 nanometer spectrum. For example, one emitter can emit light within a 500 nm to 540 nm waveband, while another emitter can emit light within a 540 nm to 580 nm waveband. In another embodiment, the emitters can cover other sized wavebands, depending on the types of emitters available or the imaging needs. For example, a plurality of emitters can include a first emitter covering a 500 nm to 540 nm waveband, a second emitter covering a 540 nm to 640 nm waveband, and a third emitter covering a 640 nm to 650 nm waveband. Each emitter can cover a different segment of the electromagnetic spectrum ranging from far infrared, mid infrared, near infrared, visible light, near ultraviolet, and / or far ultraviolet. In some cases, multiple emitters of the same type or wavelength can be included to provide sufficient output power for imaging. The number of emitters needed for a particular waveband can depend on the sensitivity of the monochromatic sensor to the waveband and / or the power output capability of the emitters in that waveband.

[0234] The waveband width and coverage provided by the emitters can be selected to provide any desired combination of the optical spectrum. For example, continuous coverage of the optical spectrum using very small waveband widths (e.g., 10 nm or less) can allow for highly selective hyperspectral imaging and / or fluorescence imaging. This waveband width can allow for selective emission of excitation wavelengths for one or more specific fluorescent reagents. Additionally, this waveband width can allow for selective emission of certain segments of hyperspectral electromagnetic radiation for identifying specific structures, chemical processes, tissues, biological processes, etc. Because the wavelengths come from emitters that can be selectively activated, extreme flexibility in causing one or more specific fluorescent reagents to fluoresce during an examination can be achieved. Additionally, extreme flexibility in identifying one or more objects or processes through hyperspectral imaging can be achieved. Thus, more fluorescence information and / or hyperspectral information can be achieved in less time and within a single examination, which would otherwise require multiple examinations, delays due to application of dyes or stains, etc.

[0235] Figure 16is a schematic diagram showing a timing diagram 1600 for generating emission and readout of images. The solid lines represent readout (peaks 1602) and blanking periods (valleys) for capturing a series of exposure frames 1604-1614. The series of exposure frames 1604-1614 can include a series of repeated exposure frames that can be used to generate laser mapping, hyperspectral, and / or fluorescence data that can be overlaid on an RGB video stream. In one embodiment, a single image frame includes information from multiple exposure frames, with one exposure frame including red image data, another exposure frame including green image data, and another exposure frame including blue image data. Additionally, a single image frame can include one or more of hyperspectral image data, fluorescence image data, and laser mapping data. The multiple exposure frames are combined to produce a single image frame. The single image frame is an RGB image with hyperspectral imaging data. The series of exposure frames includes a first exposure frame 1604, a second exposure frame 1606, a third exposure frame 1608, a fourth exposure frame 1610, a fifth exposure frame 1612, and an Nth exposure frame 1626.

[0236] Additionally, the hyperspectral image data, fluorescence image data, and laser mapping data can be used in combination to identify key tissues or structures and further measure the dimensions of those key tissues or structures. For example, the hyperspectral image data can be provided to a corresponding system to identify certain key structures in the body, such as nerves, ureters, blood vessels, cancerous tissue, etc. The location and identification of the key structures can be received from the corresponding system and also used to generate a topography of the key structures using the laser mapping data. For example, the corresponding system determines the location of a cancerous tumor based on the hyperspectral imaging data. Since the location of the cancerous tumor is known based on the hyperspectral imaging data, the topography and distance of the cancerous tumor can then be calculated based on the laser mapping data. This example can also apply when a cancerous tumor or other structure is identified based on fluorescence imaging data.

[0237] In one embodiment, each exposure frame is generated based on at least one pulse of electromagnetic energy. The pulses of electromagnetic energy are reflected and detected by the image sensor and then read out in a subsequent readout (1602). Thus, each blanking period and readout results in an image frame for a particular spectrum of electromagnetic energy. For example, the first exposure frame 1604 can be generated based on the spectrum of a first one or more pulses 1616, the second exposure frame 1606 can be generated based on the spectrum of a second one or more pulses 1618, the third exposure frame 1608 can be generated based on the spectrum of a third one or more pulses 1620, the fourth exposure frame 1610 can be generated based on the spectrum of a fourth one or more pulses 1622, the fifth exposure frame 1612 can be generated based on the spectrum of a fifth one or more pulses 2424, and the Nth exposure frame 1626 can be generated based on the spectrum of an Nth one or more pulses 1626.

[0238] Pulses 1616-1626 can include energy from a single emitter or a combination from two or more emitters. For example, the spectra included in a single readout cycle or within multiple exposure frames 1604-1614 can be selected for a desired examination or detection of a particular tissue or condition. According to one embodiment, one or more pulses can include visible spectrum light for generating an RGB or black and white image, while one or more additional pulses are emitted to sense a spectral response of electromagnetic radiation for hyperspectral wavelengths. For example, pulse 1616 can include red light, pulse 1618 can include blue light, and pulse 1620 can include green light, while the remaining pulses 1622-1626 can include wavelengths and spectra for detecting a particular tissue type, causing a reagent to fluoresce, and / or mapping the topography of a scene. As another example, the pulses of a single readout cycle include spectra (e.g., different segments of the electromagnetic spectrum) generated by multiple different emitters that can be used to detect a particular tissue type. For example, if the combination of wavelengths causes a pixel to have a value that exceeds or falls below a threshold, the pixel can be classified as corresponding to a particular type of tissue. Each frame can be used to further narrow down the type of tissue present at that pixel (e.g., as well as each pixel in the image) to provide a very specialized classification of the tissue and / or a state (diseased / healthy) of the tissue based on the spectral response of the tissue and / or whether a fluorescent reagent is present in the tissue.

[0239] Multiple frames 1604-1614 are shown as having different lengths of readout cycles and having pulses of different lengths or intensities. The blanking period, pulse length or intensity, etc. can be selected based on the sensitivity of the monochromatic sensor to particular wavelengths, the power output capability of the emitters, and / or the carrying capacity of the waveguide.

[0240] In one embodiment, a dual image sensor can be used to obtain a three- dimensional image or video feed. Three-dimensional examination can allow for an improved understanding of the three-dimensional structure of an examination region and a mapping of different tissue or material types within the region.

[0241] In one example implementation, a patient is provided with a fluorescent reagent and the fluorescent reagent is configured to be able to attach to cancer cells. The fluorescent reagent is known to fluoresce when irradiated by a particular partitioned electromagnetic radiation. The relaxation wavelength of the fluorescent reagent is also known. In this example implementation, the patient is imaged using an endoscopic imaging system as described herein. The endoscopic imaging system pulses partitions of red, green, and blue wavelengths of light to generate an RGB video stream of the interior of the patient's body. Additionally, the endoscopic imaging system pulses electromagnetic radiation of an excitation wavelength for the fluorescent reagent administered to the patient. In an example, the patient has cancer cells and the fluorescent reagent has adhered to the cancer cells. When the endoscopic imaging system pulses the excitation wavelength for the fluorescent reagent, the fluorescent reagent will fluoresce and emit the relaxation wavelength. If cancer cells are present in the scene imaged by the endoscopic imaging system, then the fluorescent reagent will also be present in the scene and emit its relaxation wavelength after fluorescing due to the emission of the excitation wavelength. The endoscopic imaging system senses the relaxation wavelength of the fluorescent reagent and thereby senses the presence of the fluorescent reagent in the scene. Because the fluorescent reagent is known to attach to cancer cells, the presence of the fluorescent reagent is further indicative of the presence of cancer cells within the scene. The endoscopic imaging system thereby identifies the location of the cancer cells within the scene. The endoscopic imaging system can also emit a laser mapping pulse scheme for generating a topography of the scene and calculating the dimensions of objects within the scene. The location of the cancer cells, as identified by the fluorescent imaging data, can be combined with the topography and dimension information calculated based on the laser mapping data. Thus, the precise location, size, dimensions, and topography of the cancer cells can be identified. This information can be provided to a medical practitioner to aid in the excision of the cancer cells. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to excise the cancer cells.

[0242] In another example implementation, a patient is imaged using an endoscopic imaging system to identify quantitative diagnostic information about the patient's tissue pathology. In an example, the patient is suspected of or known to have a disease that can be tracked using hyperspectral imaging to observe the progression of the disease in the patient's tissue. The endoscopic imaging system pulses a partition of red, green, and blue wavelengths of light to generate an RGB video stream of the inside of the patient's body. Additionally, the endoscopic imaging system pulses one or more hyperspectral wavelengths of light that allow the system to "see through" some tissue and generate an image of the tissue affected by the disease. The endoscopic imaging system senses the reflected hyperspectral electromagnetic radiation to generate hyperspectral imaging data of the diseased tissue and thereby identify the location of the diseased tissue within the patient's body. The endoscopic imaging system can also emit a laser mapping pulse scheme used to generate a topography of the scene and calculate the dimensions of objects within the scene. The location of the diseased tissue, as identified by the hyperspectral imaging data, can be combined with the topography and dimensional information calculated with the laser mapping data. Thus, the precise location, size, dimensions, and topography of the diseased tissue can be identified. This information can be provided to a medical practitioner to assist in resecting, imaging, or studying the diseased tissue. Additionally, this information can be provided to a robotic surgical system to enable the surgical system to resect the diseased tissue.

[0243] Figure 17 is a schematic diagram of an imaging system 1700 with a single cut filter. The system 1700 includes an endoscope 1706 or other suitable imaging device with a light source 1708 for a light deficient environment. The endoscope 1706 includes an image sensor 1704 and a filter 1702 that is used to filter out unwanted wavelengths of light or other electromagnetic radiation before reaching the image sensor 1704. The light source 1708 transmits light that can illuminate a surface 1712 in a light deficient environment such as a body cavity. Light 1710 reflects off the surface 1712 and passes through the filter 1702 before hitting the image sensor 1704.

[0244] The filter 1702 can be used in implementations where a fluorescent reagent or dye has been applied. In such embodiments, the light source 1708 emits an excitation wavelength for use in causing the fluorescent reagent or dye to fluoresce. Typically, the relaxation wavelength emitted by the fluorescent reagent or dye will have a different wavelength than the excitation wavelength. The filter 1702 can be selected to filter out the excitation wavelength and only allow the relaxation wavelength to pass through the filter and be sensed by the image sensor 1704.

[0245] In one embodiment, the optical filter 1702 is configured to filter out excitation wavelengths of electromagnetic radiation that cause a reagent or dye to fluoresce, such that only the expected relaxation wavelengths of the fluorescing reagent or dye are allowed to pass through the optical filter 1702 and reach the image sensor 1704. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 795 nm and 815 nm. In one embodiment, the optical filter 1702 filters out fluorescent reagent excitation wavelengths of at least between 770 nm and 790 nm and between 795 nm and 815 nm. In these embodiments, the optical filter 1702 filters out the excitation wavelengths of the reagent and only allows the image sensor 1704 to read the relaxation wavelengths of the fluorescent reagent. The image sensor 1704 can be a wavelength-agnostic image sensor, and the optical filter 1702 can be configured to allow the image sensor 1704 to receive only the relaxation wavelengths of the fluorescent reagent and not the emission excitation wavelengths of the reagent. The data determined by the image sensor 1704 can then indicate the presence of a key body structure, tissue, biological process, or chemical process determined by the location of the reagent or dye.

[0246] The optical filter 1702 can also be used in implementations where no fluorescent reagent or dye has been applied. The optical filter 1702 can be selected to allow wavelengths corresponding to a desired spectral response to pass through and be read by the image sensor 1704. The image sensor 1704 can be a monochromatic image sensor, such that pixels of the captured image above or below a threshold can be characterized as corresponding to certain spectral responses or fluorescent emissions. The spectral responses or fluorescent emissions determined by the pixels captured by the image sensor 1704 can indicate the presence of certain body tissues or structures, certain conditions, certain chemical processes, etc.

[0247] Figure 18 is a schematic diagram of an imaging system 1800 having multiple cut-off filters. The system 1800 includes a scope 1806 or other suitable imaging device having a light source 1808 for a light deficient environment. The scope 1806 includes an image sensor 1804 and two optical filters 1802a, 1802b. It should be understood that in alternative embodiments, the system 1800 can include any number of optical filters, and the number of optical filters and the type of optical filters can be selected for certain purposes, such as for gathering imaging information of particular body tissues, body conditions, chemical processes, etc. The optical filters 1802a, 1802b are configured to prevent the image sensor 1804 from sensing light or other electromagnetic radiation of unwanted wavelengths. The optical filters 1802a, 1802b can be configured to filter out unwanted wavelengths from the white light or other electromagnetic radiation that can be emitted by the light source 1808.

[0248] With respect toFigure 17 Further describing the invention, filters 1802a and 1802b can be used in embodiments where a fluorescent reagent or dye has been applied. Filters 1802a and 1802b can be configured to block the emission excitation wavelength of the reagent or dye and allow the image sensor 1804 to read only the relaxation wavelength of the reagent or dye. Furthermore, filters 1802a and 1802b can be used in embodiments where a fluorescent reagent or dye has not been applied. In such embodiments, filters 1802a and 1802b can be selected to allow wavelengths corresponding to the desired spectral response to pass through and be read by the image sensor 1804.

[0249] Multiple filters 1802a, 1802b can each be configured to filter out wavelengths of different ranges in the electromagnetic spectrum. For example, one filter can be configured to filter out wavelengths longer than the desired wavelength range, and additional filters can be configured to filter out wavelengths shorter than the desired wavelength range. A combination of two or more filters can result in only certain wavelengths or wavelength bands being read by the image sensor 1804.

[0250] In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 513 nm and 545 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 565 nm and 585 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 900 nm and 1000 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 417 nm and 475 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 520 nm and 545 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 617 nm and 645 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 760 nm and 795 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 795 nm and 815 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 370 nm and 420 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are customized such that electromagnetic radiation between 600 nm and 670 nm contacts the image sensor 1804. In one embodiment, the optical filters 1802a, 1802b are configured to allow only certain fluorescent relaxation emissions to pass through the optical filters 1802a, 1802b and contact the image sensor 1804. In one embodiment, the first optical filter blocks electromagnetic radiation having a wavelength of about 770 nm to about 790 nm, and the second optical filter blocks electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0251] In one embodiment, the system 1800 includes multiple image sensors 1804, and can specifically include two image sensors for generating a three-dimensional image. The image sensors 1804 can be color / wavelength agnostic, and configured to read electromagnetic radiation of any wavelength reflected from the surface 1812. In one embodiment, the image sensors 1804 are each color-dependent or wavelength-dependent, and configured to read electromagnetic radiation of a specific wavelength reflected from the surface 1812 and returned to the image sensors 1804. Alternatively, the image sensors 1804 can include a single image sensor having multiple different pixel sensors configured to read different wavelengths or colors of light, such as a Bayer filter color filter array. Alternatively, the image sensors 1804 can include one or more color agnostic image sensors that can be configured to read different wavelengths of electromagnetic radiation according to a pulse schedule, such as those shown in FIGS. 1-3. Figures 5 to 7E

[0252] Figure 19A and Figure 19B respectively show perspective and side views of an implementation of an imaging sensor 1900 having multiple pixel arrays for producing a three-dimensional image. The three-dimensional image sensor can be built on multiple substrates and can include multiple pixel arrays and other associated circuitry, with multiple columns of pixels 1904a forming a first pixel array and multiple columns of pixels 1904b forming a second pixel array located on respective substrates 1902a and 1902b, respectively, and multiple columns of circuitry 1908a and 1908b located on a separate substrate 1906. Also shown are electrical connections and communication between the columns of pixels and associated or corresponding columns of circuitry.

[0253] The multiple pixel arrays can sense information simultaneously, and information from the multiple pixel arrays can be combined to generate a three-dimensional image. In one embodiment, an endoscopic imaging system includes two or more pixel arrays that can be deployed to generate three-dimensional imaging. The endoscopic imaging system can include an emitter to emit a pulse of electromagnetic radiation during a blanking period of the pixel arrays. The pixel arrays can be synchronized such that optical black pixels are read for both or more pixel arrays simultaneously (i.e., the blanking period occurs). The emitter can emit a pulse of electromagnetic radiation to charge each of the two or more pixel arrays. The two or more pixel arrays can read their respective charged pixels simultaneously, such that the readout periods of the two or more pixel arrays occur simultaneously or approximately simultaneously. In one embodiment, the endoscopic imaging system includes multiple emitters, each emitter being individually synchronized with one or more of the multiple pixel arrays. Information from the multiple pixel arrays can be combined to generate three-dimensional image frames and video streams.

[0254] Figure 20A and​Figure 20B Perspective and side views of an implementation of a monolithic sensor 2000 having multiple pixel arrays for producing a three-dimensional image are shown, respectively, in accordance with the teachings and principles of the present disclosure. Such an implementation can be desirable for three-dimensional image capture, where the two pixel arrays 2002 and 2004 can be offset during use. In another implementation, the first pixel array 2002 and the second pixel array 2004 can be dedicated to receiving electromagnetic radiation of a predetermined wavelength range, where the first pixel array is dedicated to electromagnetic radiation of a different wavelength range than the second pixel array.

[0255] Figure 21A and Figure 21B Perspective and side views of an implementation of an imaging sensor 2100 built on multiple substrates are shown, respectively. As shown, multiple pixel columns 2104 forming a pixel array are located on a first substrate 2102 and multiple circuit columns 2108 are located on a second substrate 2106. Electrical connections and communication between one pixel column and its associated or corresponding circuit column are also shown in the figure. In one implementation, the image sensor can have a pixel array separate from all or most of the supporting circuitry, while it can otherwise be fabricated with its pixel array and supporting circuitry on a single, monolithic substrate / chip. The present disclosure can use at least two substrates / chips that will be stacked together using three-dimensional stacking technology. A first 2102 of the two substrates / chips can be processed using image CMOS technology. The first substrate / chip 2102 can consist of only a pixel array, or can consist of a pixel array surrounded by limited circuitry. The second or subsequent substrate / chip 2106 can be processed using any technology, not necessarily from image CMOS technology. The second substrate / chip 1306 can be, but is not limited to, a high-density digital technology for integrating various and multiple functions into very limited space or area on the substrate / chip, or a mixed-mode or analog technology for integrating, for example, precise analog functions, or an RF technology for enabling wireless capabilities, or a MEMS (Micro-Electro-Mechanical System) for integrating MEMS devices. The image CMOS substrate / chip 2102 can be stacked with the second or subsequent substrate / chip 2106 using any three-dimensional technology. The second substrate / chip 2106 can support the vast majority or most of the circuitry that would otherwise be implemented in the first image CMOS chip 2102 (if implemented on a monolithic substrate / chip) as peripheral circuitry, and thus increases the overall system area while keeping the pixel array size constant and optimized as much as possible. Electrical connections between the two substrates / chips can be done through interconnects, which can be bond wires, lugs, and / or TSVs (Through Silicon Vias).

[0256] It should be understood that the teachings and principles of the present disclosure can be employed in a reusable device platform, a limited use device platform, a repositionable use device platform, or a single use / disposable device platform without departing from the scope of the present disclosure. It should be understood that in a reusable device platform, an end user is responsible for cleaning and sterilization of the device. In a limited use device platform, a device can be used a specified number of times before becoming inoperable. A typical new device has been sterilized prior to delivery, and if to be used for other purposes, should be cleaned and sterilized by the end user prior to other uses. In a repositionable use device platform, a third party can reprocess (e.g., clean, package, and sterilize) single use devices for additional uses at a lower cost than new units. In a single use / disposable device platform, the device is delivered to the operating room in a sterile manner and can only be used once before being disposed of.

[0257] Examples

[0258] The following embodiments relate to preferred features of additional embodiments:

[0259] Example 1 is a system. The system includes an emitter to emit pulses of electromagnetic radiation and an image sensor including an array of pixels to sense reflected electromagnetic radiation, wherein the image sensor is disposed within a lumen of an endoscope. The system includes a rotation sensor to detect a rotation angle of the lumen relative to a handpiece of the endoscope. The system includes a controller in electronic communication with the emitter and the image sensor. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter includes electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or about 795 nm to about 815 nm.

[0260] Example 2 is the system of Example 1, wherein the lumen is rotatable about an axis of the endoscope and relative to the handpiece.

[0261] Example 3 is the system of any of Examples 1-2, further comprising an image signal processing pipeline to perform a rotation transform on an exposure frame captured by the image sensor based on the rotation angle detected by the rotation sensor; wherein the image signal processing pipeline rotates the exposure frame inversely to the rotation angle to maintain a consistent image horizon of a scene being imaged by the image sensor.

[0262] Example 4 is the system of any of Examples 1-3, wherein the rotation sensor is a rotation detection Hall effect sensor and is located in the handpiece of the endoscope.

[0263] Example 5 is the system of any of Examples 1-4, further comprising a radially polarized magnetic ring band, and wherein the rotation sensor generates a voltage for detecting an angle of the radially polarized magnetic ring band.

[0264] Example 6 is the system of any of Examples 1-5, wherein the rotation sensor generates a voltage for detecting the rotation angle of the inner lumen relative to the handpiece.

[0265] Example 7 is the system of any of Examples 1-6, wherein the rotation sensor is a potentiometer comprising a carbon wire, wherein the carbon wire is disposed within the inner lumen of the endoscope.

[0266] Example 8 is the system of any of Examples 1-7, wherein the rotation sensor comprises a light source and a photodiode rotating relative to a gradient disk, and wherein the photodiode detects electromagnetic energy emitted by the light source and reflected from the gradient disk.

[0267] Example 9 is the system of any of Examples 1-8, further comprising an image signal processing pipeline for performing a rotation transform on an exposure frame captured by the image sensor based on the rotation angle detected by the rotation sensor, wherein the image rotation transform comprises: identifying integer (x,y) coordinates of pixel data in the exposure frame captured by pixels of the pixel array; applying a rotation kernel to the integer (x,y) coordinates to transform the integer (x,y) coordinates to real number pixel coordinates; truncating the real number pixel coordinates to integer values; and assigning values to blank pixel data in the exposure frame using nearby fill pixel data from the exposure frame.

[0268] Example 10 is the system of any of Examples 1-9, further comprising an image signal processing pipeline for performing a rotation transform on an exposure frame captured by the image sensor based on the rotation angle detected by the rotation sensor, wherein the image rotation transform comprises: identifying integer (x,y) coordinates of pixel data in the exposure frame captured by pixels of the pixel array; applying an inverse rotation kernel to the integer (x,y) coordinates to transform the integer (x,y) coordinates to real number (x,y) coordinates; and estimating a pixel value at the real number (x,y) coordinates using data from one or more nearest distance integer coordinate locations; wherein the estimating is performed using one of nearest neighbor substitution, bilinear interpolation, or bicubic interpolation.

[0269] Example 11 is the system of any of Examples 1-10, wherein the image sensor is configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to a pulse of electromagnetic radiation emitted by the emitter.

[0270] Example 12 is the system of any of Examples 1-11, wherein the array of pixels of the image sensor senses reflected electromagnetic radiation during a readout period of the array of pixels to generate the plurality of exposure frames, wherein the readout period is a duration of time when active pixels in the array of pixels are read.

[0271] Example 13 is the system of any of Examples 1-12, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are hyperspectral wavelengths for eliciting a spectral response, wherein the hyperspectral wavelengths include one or more of: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm, and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm, and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0272] Example 14 is the system of any of Examples 1-13, wherein the emitter is configured to emit a plurality of electromagnetic radiation sub-pulses during a pulse duration, the plurality of electromagnetic radiation sub-pulses having a sub-duration shorter than the pulse duration.

[0273] Example 15 is the system of any of Examples 1-14, wherein one or more of the pulses of electromagnetic radiation emitted by the emitter include electromagnetic radiation emitted at two or more wavelengths simultaneously as a single pulse or a single sub-pulse.

[0274] Example 16 is the system of any of Examples 1-15, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter are hyperspectral emissions resulting in hyperspectral exposure frames produced by the image sensor, and wherein the controller is configured to provide the hyperspectral exposure frames to a corresponding hyperspectral system that determines a location of a key tissue structure within a scene based on the hyperspectral exposure frames.

[0275] Example 17 is the system of any of Examples 1-16, wherein the hyperspectral emissions include: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0276] Example 18 is the system of any of Examples 1-17, wherein the controller is further configured to: receive the location of the critical tissue structure from the corresponding hyperspectral system; generate an overlay frame including the location of the critical tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

[0277] Example 19 is the system of any of Examples 1-18, wherein sensing electromagnetic radiation reflected by the pixel array includes generating a laser mapping exposure frame by sensing reflected electromagnetic radiation resulting from the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topography of the scene and / or dimensions of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive a topography and / or dimensions of the critical tissue structure from the corresponding laser mapping system.

[0278] Example 20 is the system of any of Examples 1-19, wherein the critical structure includes one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0279] Example 21 is the system of any of Examples 1-20, wherein at least a portion of the electromagnetic radiation pulses emitted by the emitter are the fluorescence excitation wavelengths that result in a fluorescence exposure frame produced by the image sensor, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding fluorescence system that determines a location of a critical tissue structure within the scene based on the fluorescence exposure frame.

[0280] Example 22 is the system of any of Examples 1-21, wherein the fluorescence excitation emissions include one or more of: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0281] Example 23 is the system of any of Examples 1-22, wherein the controller is further configured to: receive the location of the critical tissue structure from the corresponding fluorescence system; generate an overlay frame comprising the location of the critical tissue structure; and combine the overlay frame with a color image frame depicting the scene to indicate the location of the critical tissue structure within the scene.

[0282] Example 24 is the system of any of Examples 1-23, wherein sensing the reflected electromagnetic radiation by the pixel array comprises generating a laser mapping exposure frame by sensing reflected electromagnetic radiation produced by the emitter pulsing the laser mapping pattern, and wherein the controller is further configured to: provide the laser mapping exposure frame to a corresponding laser mapping system that determines a topography of the scene and / or a size of one or more objects within the scene; provide the location of the critical tissue structure to the corresponding laser mapping system; and receive a topography and / or size of the critical tissue structure from the corresponding laser mapping system.

[0283] Example 25 is the system of any of Examples 1-24, wherein the critical structure comprises one or more of a nerve, a ureter, a blood vessel, an artery, a blood flow, or a tumor.

[0284] Example 26 is the system of any of Examples 1-25, wherein the controller is configured to synchronize timing of the pulses of electromagnetic radiation during a blanking period of the image sensor, wherein the blanking period corresponds to a time between a readout of a last row of active pixels in the pixel array and a start of a next subsequent readout of active pixels in the pixel array.

[0285] Example 27 is the system of any of Examples 1-26, wherein two or more pulses of electromagnetic radiation emitted by the emitter result in two or more instances of reflected electromagnetic radiation that are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

[0286] Example 28 is the system of any of Examples 1-27, wherein the image sensor comprises a first image sensor and a second image sensor such that the image sensor is capable of generating a three-dimensional image.

[0287] Example 29 is the system of any of Examples 1-28, wherein the emitter is configured to repeatedly emit a sequence of pulses of electromagnetic radiation so as to be sufficient to generate a video stream comprising a plurality of image frames, wherein each image frame in the video stream comprises data from a plurality of exposure frames, and wherein each of the exposure frames corresponds to a pulse of electromagnetic radiation.

[0288] Example 30 is the system of any of Examples 1-29, wherein the pulses of electromagnetic radiation are emitted in a pattern of different wavelengths of electromagnetic radiation, and wherein the emitter repeats the pattern of different wavelengths of electromagnetic radiation.

[0289] Example 31 is the system of any of Examples 1-30, wherein at least a portion of the pulses of electromagnetic radiation comprise a red wavelength, a green wavelength, a blue wavelength, and a hyperspectral wavelength, such that reflected electromagnetic radiation corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the hyperspectral wavelength sensed by the pixel array is processable to generate a red-green-blue (RGB) image frame comprising a superposition of hyperspectral imaging data, wherein the hyperspectral wavelength of electromagnetic radiation comprises: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0290] Example 32 is the system of any of Examples 1-31, wherein at least a portion of the pulses of electromagnetic radiation comprise a luminance emission, a red chrominance emission, a blue chrominance emission, and a hyperspectral emission, such that reflected electromagnetic radiation corresponding to each of the luminance emission, the red chrominance emission, the blue chrominance emission, and the hyperspectral emission sensed by the pixel array is processable to generate a YCbCr image frame comprising a superposition of hyperspectral imaging data, wherein the hyperspectral emission of electromagnetic radiation comprises: the electromagnetic radiation having a wavelength of about 513 nm to about 545 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm; or the electromagnetic radiation having a wavelength of about 565 nm to about 585 nm and the electromagnetic radiation having a wavelength of about 900 nm to about 1000 nm.

[0291] Example 33 is the system of any of Examples 1-32, wherein the internal lumen of the endoscope is rotatable about an axis of the endoscope and relative to the handpiece of the endoscope.

[0292] Example 34 is the system of any of Examples 1-33, wherein the endoscope comprises a proximal portion and a distal portion, wherein the lumen and the image sensor are disposed at the distal portion and the handpiece is disposed at the proximal portion.

[0293] Example 35 is the system of any of Examples 1-34, wherein the controller comprises an image signal processing pipeline to perform a rotational transform on an exposure frame captured by the image sensor based on the rotational angle detected by the rotational sensor, wherein the image signal processing pipeline is configured to rotate the exposure frame inversely to the rotational angle detected by the rotational sensor to maintain a constant image horizon.

[0294] Example 36 is the system of any of Examples 1-35, wherein an orientation of a rotated exposure frame is rotationally different from an orientation of the lumen relative to the handpiece.

[0295] Example 37 is the system of any of Examples 1-36, wherein the array of pixels is a two-dimensional array of individual pixels, each individual pixel capable of detecting electromagnetic radiation of any wavelength.

[0296] Example 38 is the system of any of Examples 1-37, wherein assigning the value to the blank pixel data in the exposure frame is performed by bilinear interpolation.

[0297] Example 39 is the system of any of Examples 1-38, wherein assigning the value to the blank pixel data in the exposure frame is performed by bicubic interpolation.

[0298] Example 40 is the system of any of Examples 1-39, wherein assigning the value to the blank pixel data in the exposure frame is performed by nearest neighbor substitution.

[0299] Example 41 is the system of any of Examples 1-40, further comprising a filter that filters electromagnetic radiation having a wavelength of about 770 nm to about 790 nm.

[0300] Example 42 is the system of any of Examples 1-41, further comprising a filter that filters electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0301] Example 43 is the system of any one of Examples 1-42, wherein at least a portion of the pulses of electromagnetic radiation emitted by the emitter is a fluorescence excitation wavelength for causing a reagent to fluoresce, wherein the fluorescence excitation wavelength comprises one or more of: the electromagnetic radiation having a wavelength of about 770 nm to about 790 nm; or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0302] Example 44 is the system of any one of Examples 1-43, wherein at least a portion of the pulses of electromagnetic radiation comprises a red wavelength, a green wavelength, a blue wavelength, and a fluorescence excitation wavelength, such that reflected electromagnetic radiation corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the fluorescence excitation wavelength sensed by the pixel array can be processed to generate a red-green-blue (RGB) image frame comprising an overlay of fluorescence imaging data, wherein the fluorescence wavelength of electromagnetic radiation comprises: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0303] Example 45 is the system of any one of Examples 1-44, wherein at least a portion of the pulses of electromagnetic radiation comprises a luminance emission, a red chroma emission, a blue chroma emission, and a fluorescence excitation emission, such that reflected electromagnetic radiation corresponding to each of the luminance emission, the red chroma emission, the blue chroma emission, and the fluorescence excitation emission sensed by the pixel array can be processed to generate a YCbCr image frame comprising an overlay of fluorescence imaging data, wherein the fluorescence wavelength of electromagnetic radiation comprises: electromagnetic radiation having a wavelength of about 770 nm to about 790 nm and / or electromagnetic radiation having a wavelength of about 795 nm to about 815 nm.

[0304] It should be appreciated that various features disclosed herein provide significant advantages over the state of the art. The following claims are in the nature of a benchmark to those features.

[0305] In the above disclosure of embodiments, various features of the disclosure have been centralized into single embodiments for the purpose of simplifying the disclosure. The methods of the disclosure should not be understood to embody an intent that the disclosure claimed requires more features than are expressly recited in each claim. Rather, the inventive aspects fail to embody all features of the single embodiments disclosed above.

[0306] It should be appreciated that any feature of the above-described arrangements, embodiments, and implementations can be combined in a single embodiment comprising a combination of features from any of the disclosed arrangements, embodiments, and implementations.

[0307] It is to be understood that the above-referenced arrangements are merely illustrative of the application of the principles of this disclosure. Numerous modifications and alternative arrangements can be devised by those skilled in the art without departing from the spirit and scope of the present disclosure and the appended claims are intended to cover such modifications and arrangements.

[0308] Thus, although there have been described specific embodiments of the disclosure, it is manifestly intended that the disclosure be used in any other form, structure, or arrangement not specifically shown and described herein. Many modifications and variations will readlize themselves in light of the above teachings. The disclosure is to be understood based on the words of the claims, which are to be interpreted in the light of this disclosure and changing or relating the prior art, and therefore should the claims be read also to cover any and all coming equivalents.

[0309] Also, the functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) can be programmed to perform one or more of the systems and processes described herein. Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, components can be referred to by different names in different patents and documents throughout the industry. Reference to such components herein is not intended to limit the scope of the application or the patent claiming here.

[0310] The specific embodiments were chosen and described in order to provide a thorough and complete disclosure of the application. No undertaking is made to exhaustively describe all embodiments of the disclosure and additional modifications will occur to those skilled in the art in view of the foregoing teachings. It is also noted that, as used herein, "comprising" means "including, but not limited to," and "comprises" or "comprising" is not used in a restrictive sense whenever it appears.

[0311] In addition, while a particular feature can have been disclosed with respect to only one of several implementations, other implementations can include the feature. It is, therefore, intended that such features be within the scope of the present disclosure, and in particular within the scope of the methods claimed, and additionally within the scope of any claims that can be presented in the future.

Claims

1. A system for endoscopic imaging, the system comprising: A transmitter for emitting multiple electromagnetic radiation pulses; An image sensor comprising an array of pixels for sensing reflected electromagnetic radiation, wherein the image sensor is disposed within the cavity of an endoscope and configured to generate a plurality of exposure frames, wherein each of the plurality of exposure frames corresponds to one of a plurality of electromagnetic radiation pulses emitted by the transmitter, wherein the plurality of exposure frames includes color exposure frames and fluorescence exposure frames. A rotation sensor for detecting the rotation angle of the lumen relative to the endoscope's handheld device; and The controller is in electronic communication with the transmitter and the image sensor; Wherein, at least a portion of the electromagnetic radiation pulse emitted by the emitter is fluorescence excitation emission for causing the reagent to fluoresce, wherein the fluorescence excitation emission includes one or more of the following: Electromagnetic radiation with a wavelength of approximately 770±20 nm; or Electromagnetic radiation with a wavelength of approximately 795±20nm.

2. The system according to claim 1, wherein, The cavity is rotatable about the axis of the endoscope and relative to the handpiece.

3. The system according to claim 1, further comprising: An image signal processing pipeline for performing rotation transformation on the plurality of exposure frames captured by the image sensor based on the rotation angle detected by the rotation sensor; The image signal processing pipeline rotates the exposure frame in the opposite direction to the rotation angle to maintain a consistent image horizon for the scene being imaged by the image sensor.

4. The system according to claim 1, wherein, The rotation sensor is a rotation detection Hall effect sensor and is located in the handheld component of the endoscope.

5. The system of claim 4, further comprising radially polarized magnetic rings, wherein, The rotation sensor generates a voltage for detecting the angle of the radially polarized magnetic ring.

6. The system according to claim 1, wherein, The rotation sensor generates a voltage for detecting the rotation angle of the cavity relative to the handheld device.

7. The system according to claim 1, wherein, The rotation sensor is a potentiometer comprising a carbon wire, wherein the carbon wire is disposed within the cavity of the endoscope.

8. The system according to claim 1, wherein, The rotation sensor includes a light source and a photodiode that rotates relative to a gradient disk, wherein the photodiode detects electromagnetic energy emitted by the light source and reflected from the gradient disk.

9. The system of claim 1, further comprising an image signal processing pipeline, the image signal processing pipeline being configured to perform a rotation transformation on the plurality of exposure frames captured by the image sensor based on the rotation angle detected by the rotation sensor, wherein, Calculating the image rotation transformation includes: Identify the integer (x, y) coordinates of pixel data in the exposure frame captured by the pixels of the pixel array; Apply a rotation kernel to the integer (x, y) coordinates to transform the integer (x, y) coordinates into real pixel coordinates; Truncate the real pixel coordinates to integer values; and Values ​​are assigned to the blank pixel data in the exposure frame using fill pixel data values ​​from the vicinity of the exposure frame.

10. The system of claim 1, further comprising an image signal processing pipeline, the image signal processing pipeline being configured to perform a rotation transformation on the plurality of exposure frames captured by the image sensor based on the rotation angle detected by the rotation sensor, wherein, Calculating the image rotation transformation includes: Identify the integer (x, y) coordinates of pixel data in the exposure frame captured by the pixels of the pixel array; Applying a reverse rotation kernel to the integer (x, y) coordinates transforms the integer (x, y) coordinates into real (x, y) coordinates; and The pixel value at the real (x, y) coordinates is estimated using data from one or more of the nearest integer coordinate locations; The estimation is performed using one of nearest neighbor permutation, bilinear interpolation, or bicubic interpolation.

11. The system according to claim 1, wherein, During the readout period of the pixel array of the image sensor, the pixel array senses reflected electromagnetic radiation to generate the plurality of exposure frames, wherein the readout period is the duration during which an effective pixel in the pixel array is read.

12. The system according to claim 1, wherein, The transmitter is configured to emit multiple electromagnetic radiation sub-pulses during the pulse duration, the multiple electromagnetic radiation sub-pulses having a sub-duration shorter than the pulse duration.

13. The system according to claim 1, wherein, One or more of the plurality of electromagnetic radiation pulses emitted by the transmitter include electromagnetic radiation emitted simultaneously at two or more wavelengths as a single pulse or a single sub-pulse.

14. The system according to claim 1, wherein, The emission of fluorescence excitation results in the fluorescence exposure frame generated by the image sensor, and wherein the controller is configured to provide the fluorescence exposure frame to a corresponding system that determines the location of the tissue structure within the scene based on the fluorescence exposure frame.

15. The system according to claim 14, wherein, The controller is further configured to be able to: Receive the location of the organizational structure from the corresponding system; Generate a stacked frame that includes the location of the tissue structure; as well as The overlay frame is combined with a color image frame depicting the scene to indicate the location of the tissue structure within the scene.

16. The system according to claim 15, wherein, The structures include one or more of nerves, ureters, blood vessels, arteries, blood flow, or tumors.

17. The system according to claim 1, wherein, The controller is configured to synchronize the timing of the plurality of electromagnetic radiation pulses during the blanking period of the image sensor, wherein the blanking period corresponds to the time between the readout of the last row of valid pixels in the pixel array and the start of the next subsequent readout of valid pixels in the pixel array.

18. The system according to claim 1, wherein, Two or more of the plurality of electromagnetic radiation pulses emitted by the transmitter result in two or more instances of reflected electromagnetic radiation, which are sensed by the pixel array to generate two or more exposure frames that are combined to form an image frame.

19. The system according to claim 1, wherein, The image sensor includes a first image sensor and a second image sensor, enabling the image sensor to generate three-dimensional images.

20. The system according to claim 1, wherein, The transmitter is configured to repeatedly emit a sequence of electromagnetic radiation pulses sufficient to generate a video stream comprising multiple image frames, wherein each image frame in the video stream includes data from multiple exposure frames, and wherein each of the multiple exposure frames corresponds to an electromagnetic radiation pulse.

21. The system according to claim 1, wherein, The plurality of electromagnetic radiation pulses are emitted in patterns of electromagnetic radiation at different wavelengths, wherein the transmitter repeats the patterns of electromagnetic radiation at different wavelengths.

22. The system according to claim 1, wherein, At least a portion of the plurality of electromagnetic radiation pulses includes red wavelength, green wavelength, blue wavelength, and fluorescence excitation wavelength. This allows the reflected electromagnetic radiation sensed by the pixel array, corresponding to each of the red wavelength, the green wavelength, the blue wavelength, and the fluorescence excitation wavelength, to be processed to generate a red-green-blue (RGB) image frame that includes superimposed fluorescence imaging data.

23. The system according to claim 1, wherein, At least a portion of the plurality of electromagnetic radiation pulses includes luminance emission, red chromaticity emission, blue chromaticity emission, and fluorescence excitation emission, such that electromagnetic radiation reflected by the pixel array corresponding to each of the luminance emission, the red chromaticity emission, the blue chromaticity emission, and the fluorescence excitation emission can be processed to generate a YCbCr image frame that includes superimposed fluorescence imaging data.

24. The system of claim 1 further includes a filter that filters electromagnetic radiation having a wavelength of about 770 nm ± 20 nm.

25. The system of claim 1 further includes a filter that filters electromagnetic radiation having a wavelength of about 795 nm ± 20 nm.

Images