Method and system for timing enhanced image sensors

Abstract

It provides a combination of conventional and advanced imaging for medical devices. [Solution] The stereoscopic image acquisition device includes a first image sensor 321L, a second image sensor 321R, a first frame timer 325L, and a second frame timer 325R. The first and second frame timers are different frame timers. The first image sensor includes a first set of pixel rows. The second image sensor includes a second set of pixel rows. The first and second image sensors may be separate devices or different areas of sensor regions within an integrated circuit. The first frame timer is coupled to the first image sensor and provides an image acquisition timing signal to the first image sensor. The second frame timer is coupled to the second image sensor and provides an image acquisition timing signal to the second image sensor.

Description

Technical Field

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 732,718, filed on September 18, 2018, entitled "METHOD AND SYSTEM FOR ENHANCED IMAGE SENSOR TIMING", the contents of which are hereby incorporated by reference in their entirety.

[0002] Aspects of the present invention relate to imaging of medical devices, more specifically, to combinations of normal imaging and advanced imaging.

Background Art

[0003] Imaging devices are used in minimally invasive surgery. Various imaging modalities (visible scene, fluorescence scene, infrared scene, hyperspectral scene) are implemented using the imaging device. However, each imaging modality utilizes one or more parameters that are different from one or more corresponding parameters of other imaging modalities, such as exposure time. This makes it difficult to use an image sensor configured for one imaging modality for a different imaging modality.

[0004] The problem of using a single image sensor for different imaging modalities becomes even more complex when a stereoscopic imaging system is used and the image sensor is optimized to capture a visible color scene. As is known, an image sensor includes pixels that capture and integrate light over time. To maximize the area of the chip available for the pixels, other circuits on the image sensor are minimized.

[0005] For example, in a stereoscopic complementary metal-oxide-semiconductor (CMOS) sensor integrated circuit, the sensor area is divided into two regions: the first region contains pixels that capture the left scene, and the second region contains pixels that capture the right scene. Both regions of the sensor area have pixels arranged in rows and columns. There are reset lines and row selection lines associated with each row of the sensor area, and read lines associated with each pixel in each row of the sensor area. To minimize the logic required for the integrated circuit, a common frame timer logic circuit is used to drive the reset lines and row selection lines of both sensor areas.

[0006] Figure 1 is a timing diagram for a CMOS sensor integrated circuit that captures frames of pixel data using a rolling shutter. The timing diagram is the same for both channels of the stereoscopic image acquisition device. In Figure 1, the Nth frame 101 is captured, followed by the N+1th frame 102, and then the N+2nd frame 103. The N+1th frame 102 is sometimes referred to as frame 102.

[0007] In this example, the capture of line zero in N+1 frames 102 is considered. (A pixel line and a pixel row are the same thing.) The capture of each line of pixels in frame 102 is the same as the capture of line 0. Similarly, each frame is captured in the same way as frame 102. Not all lines are captured simultaneously; for example, the image capture device does not have a mechanical shutter to prevent light from reaching the pixels after a predetermined time. Rather, each pixel row is read sequentially. This is shown by the diagonal 102_S of frame 102. The round dots at the right end of each horizontal line indicate that a row selection line is active and the value of each pixel in the row can be read on the reading line for that row.

[0008] To allow the pixels to re-accumulate charge over a known time interval, a signal on the reset line of each pixel in row 0 is activated, setting each pixel to a known state.

[0009] Following an active reset signal, the pixel accumulates charge corresponding to the light incident on it until a signal on the row zero selection line becomes active, and then the charge accumulated in the pixel becomes available on the reading line associated with the row. Each row in the frame is read in the same way. Once all rows have been read, a blank (BLNK) row is read to define the frame. The blank row reduces noise in the captured frame by ensuring that the power supply load remains constant. [Overview of the project]

[0010] The video viewing capability of the device is enhanced by incorporating an enhanced frame timer into the device to increase sensitivity to both visible scenes and alternative modality scenes. For example, a stereoscopic image acquisition device includes a first image sensor, a second image sensor, a first frame timer, and a second frame timer. The first and second frame timers are different frame timers. The first image sensor includes a first set of pixel rows. The second image sensor includes a second set of pixel rows. The first and second image sensors may be separate devices or different areas of sensor regions within an integrated circuit. The first frame timer is coupled to the first image sensor and provides an image acquisition timing signal to the first image sensor. The second frame timer is coupled to the second image sensor and provides an image acquisition timing signal to the second image sensor.

[0011] Dual frame timers offer several advantages. For example, one frame timer can be configured to provide a signal to one of the image sensors, thereby allowing this image sensor to capture frames at a normal video rate. The other frame timer can be configured to provide a signal to the other image sensor, thereby allowing the other image sensor to capture the scene at a slower rate than the normal video rate. This improves the signal-to-noise ratio because the other image sensor can integrate the available light for a longer period of time. Specifically, in one embodiment, the first frame timer is configured to provide an image acquisition timing signal for sequentially capturing N frames at the first image sensor. The second frame timer is configured to provide an image acquisition timing signal for capturing one frame at the second image sensor for every N frames captured at the first image sensor. Thus, each frame captured by the second image sensor integrates the incident light for a longer period of time than the first image sensor. This can also be achieved if a first frame timer is configured to expose each row of a first plurality of pixel rows over a first exposure time, and a second frame timer is configured to expose each row of a second plurality of active pixels over a second exposure time, where the first exposure time is different from the second exposure time.

[0012] An improved signal-to-noise ratio can also be achieved through multiple pixel binning. In this embodiment, the first image sensor of the stereoscopic image acquisition device includes, for example, a Bayer color filter array spanning a first plurality of pixel rows. Each position in the first plurality of pixel rows of the first image sensor contains a set of Bayer pixels. The first frame timer circuit is configured to combine each set of Bayer pixels into a row to form a single output pixel.

[0013] In one embodiment, multiple pixel binning is used in combination with a longer exposure time for one of the image acquisition sensors, sometimes called an image sensor. For example, the first image sensor of a stereoscopic image acquisition device includes a Bayer color filter array across a first set of pixel rows. Each position in the first set of pixel rows of the first image sensor contains a set of Bayer pixels. A first frame timer circuit is configured to combine each set of Bayer pixels into a row to form a single output pixel. The first frame timer is also configured to expose each row of the first set of active pixel rows over a first exposure time. A second frame timer is configured to expose each row of a second set of pixel rows over a second exposure time. The first exposure time is different from the second exposure time. This is advantageous, for example, when it is desired to overlay an augmented scene, such as a fluorescent scene, onto a monochrome scene of a surgical site.

[0014] In one embodiment, a first plurality of pixel rows includes a plurality of pixel cells. Each of the plurality of pixel cells includes a plurality of pixels. In this embodiment, the first image sensor also includes a visible light color filter array including a plurality of different individual visible light color filters and an alternative light filter array including a plurality of individual alternative light filters. One of the plurality of individual alternative light filters covers both a first set of pixels in a first pixel cell of the plurality of pixel cells and a second set of pixels in a second pixel cell of the plurality of pixel cells. The first pixel cell is adjacent to the second pixel cell. Each of the plurality of different individual visible light color filters covers different pixels in the first and second sets of pixels. Pixels covered by individual visible light color filters of the plurality of different individual color filters are different from pixels covered by individual alternative light filters.

[0015] In this embodiment, the first frame timer is configured to simultaneously reset pixels in first and second pixel cells covered by one of several different individual visible light color filters. The frame timer is also configured to simultaneously read a first pixel in the first pixel cell covered by one of several different individual visible light color filters and a second pixel in the second pixel cell covered by one of several different individual visible light color filters.

[0016] The first frame timer is also configured to simultaneously read a first pixel from a set of pixels within a first pixel cell of a plurality of pixel cells, and a second pixel from a set of pixels within a second pixel cell of a plurality of pixel cells. In this embodiment, the image acquisition device is configured to bin the read first pixel and the read second pixel.

[0017] In another embodiment, the first image sensor further includes a plurality of alternative light-filtered pixel cells and a plurality of interleaved visible light color-filtered cells.

[0018] In yet another embodiment, the image acquisition device includes an image sensor. The image sensor includes a plurality of pixel rows and a visible light color filter array. The visible light color filter array includes a plurality of different individual visible light color filters. The plurality of pixel rows include a plurality of pixel cells. Each of the plurality of pixel cells includes a plurality of pixels. Each of the plurality of pixels in the pixel cell is covered by a different color filter from the plurality of different individual visible light color filters. A frame timer is coupled to the image sensor and provides the image acquisition timing signal to the image sensor. The frame timer is configured to combine the plurality of pixels in the pixel cell to form a single output pixel.

[0019] In a further embodiment, the image acquisition device includes an image sensor having a plurality of pixel rows, a visible light color filter array, and an alternative light filter array. The plurality of pixel rows include a plurality of pixel cells. Each of the plurality of pixel cells includes a plurality of pixels. The visible light color filter array includes a plurality of different individual visible light color filters. The alternative light filter array includes a plurality of individual alternative light filters. One of the plurality of individual alternative light filters covers both a first set of pixels in a first pixel cell of the plurality of pixel cells and a second set of pixels in a second pixel cell of the plurality of pixel cells. The first pixel cell is adjacent to the second pixel cell. Each of the plurality of different individual visible light color filters covers different pixels in the first and second sets of pixels. The pixels covered by each individual visible light color filter of the plurality of individual visible light color filters are different from the pixels covered by each individual alternative light filter.

[0020] The image acquisition device also includes a frame timer coupled to the image sensor, which provides the image acquisition timing signal to the image sensor. For example, the frame timer is configured to simultaneously reset pixels in first and second pixel cells covered by one of several different individual visible light color filters. The frame timer is also configured to simultaneously read a first pixel in the first pixel cell covered by one of several different individual visible light color filters and a second pixel in the second pixel cell covered by one of several different individual visible light color filters.

[0021] The first method includes the step of using a signal from a first frame timer to expose each row of a first plurality of pixel rows of a first image sensor of a stereoscopic image capturing device over a first exposure time. This method also includes the step of using a signal from a second frame timer to expose each row of a second plurality of pixel rows of a second image sensor of the stereoscopic image capturing device over a second exposure time, where the first exposure time is different from the second exposure time.

[0022] Another method includes the step of outputting a single output pixel from a position within an image sensor that includes a plurality of Bayer pixels. The step of outputting is the step of combining a plurality of Bayer pixels at that position using a signal by a frame timer to form a single output pixel.

Brief Description of the Drawings

[0023] [Figure 1] It is a timing diagram for capturing a scene using a rolling shutter. [Figure 2] It is a diagram of a computer-assisted surgery system including an enhanced frame timer that enables implementation of an alternative rolling shutter image capture sequence. [Figure 3] It is a more detailed diagram of a part of a computer-assisted surgery system including a stereoscopic image capturing device, where each channel of the stereoscopic image capturing device has its own frame timer. [Figure 4] It is a timing diagram of one aspect of capturing a scene at different exposure times using a rolling shutter in the stereoscopic image capturing device of FIG. 3. [Figure 5] It is a more detailed timing diagram of one aspect of capturing a scene at different exposure times using a rolling shutter in the stereoscopic image capturing device of FIG. 3. [Figure 6] It is a generalized diagram showing that N frames are captured in one channel of the stereoscopic image capturing device of FIG. 3 while only a single frame is captured in the other channel of the stereoscopic image capturing device of FIG. 3. [Figure 7]A more detailed view of a part of a computer-assisted surgery system including an image capture device having a single image sensor and a single frame timer. [Figure 8A] A diagram of a frame timer and a part of the pixel array of an image sensor, where the image sensor includes a Bayer color filter array including a set of Bayer pixels, sometimes also called a plurality of Bayer pixels, at each position of the pixel array. [Figure 8B] A diagram of a frame timer and a part of the pixel array of an image sensor, where the image sensor includes a visible light color filter array and an alternative light filter array. [Figure 8C] Another example diagram of a frame timer and a part of the pixel array of an image sensor, where the image sensor includes a visible light color filter array and an alternative light filter array. [Figure 9A] A timing diagram of a plurality of pixel binning at the position of the pixel array of FIG. 8A. [Figure 9B] A timing diagram of the image capture device of FIG. 8B. [Figure 9C] A timing diagram of the read and reset sequence of the non-binned pixels of rows 0 and 1 of the image capture device of FIG. 8C. [Figure 9D] A timing diagram of the read and reset sequence of the hyper-spectral pixels binned in four directions of rows 0 and 1 of the image capture device of FIG. 8C. [Figure 9E] A timing diagram of the read and reset sequence of the non-binned pixels of rows 2 and 3 of the image capture device of FIG. 8C. [Figure 9F] A timing diagram of the read and reset sequence of the hyper-spectral pixels binned in four directions of rows 2 and 3 of the image capture device of FIG. 8C. [Figure 10] A diagram showing some combinations that can be obtained using the stereoscopic image capture device of FIG. 3, including dual frame timer logic circuits and various timing sequences. [Figure 11]This figure shows various combinations of frame timer timing sequences that can be implemented using the image acquisition device shown in Figure 7. In the figure, the first digit of the element's reference number indicates the figure in which the element first appears with that single-digit figure number. The first two digits of the element's reference number indicate the figure in which the element first appears with that two-digit figure number. [Modes for carrying out the invention]

[0024] Aspects of the present invention enhance the video capture and video viewing functions of a surgical device, such as the da Vinci® surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, by incorporating an enhanced frame timer and increasing the sensitivity of both visible scenes and alternative modality scenes used to identify clinically relevant tissues or other aspects during surgery. (da Vinci® is commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.) (This is a registered trademark of Inc.) In this specification, a computer-assisted surgical system is used as an example, but aspects of the present invention can be used in any device or system utilizing an alternative imaging modality.

[0025] The enhanced frame timer 122 (Figure 2) utilizes a new type of pixel control sequence, which in one aspect is implemented with low overhead in digital logic. These control sequences are designed to enhance the sensitivity of alternative imaging modes (hyperspectral, fluorescence, high dynamic range, etc.).

[0026] A typical complementary metal-oxide-semiconductor (CMOS) sensor frame timer in an image acquisition system uses a set of state machines to control signals for resetting, transferring, and selecting rows in the image sensor pixel array. These state machines typically output a simple series of pulses, enabling adjustments to shutter width and scene flips, etc. A typical frame time circuit is designed around the specific pixel cell used (e.g., a four-way shared pixel cell), but access to low-level timing signals is not permitted in conventional imaging applications. In a typical frame timer circuit, the user can only select values ​​from a limited set of parameters, such as shutter time and row time or frame rate settings, and timing changes for specific high dynamic range (HDR) modes. While many alternative timing sequences are possible with the four-way shared pixel cell referred to here, typical consumer-designed conventional frame timers treat these shared pixel cells as a non-shared array, simply scanning pixels row by row and column by column. In one embodiment, the enhanced frame timer 222 of the image acquisition system 220 includes enhanced fixed logic that enables the generation of more pulse sequences in the reset, transfer, and row selection lines of the image sensor 221 than was possible with previous CMOS sensor frame timers.

[0027] In another embodiment, the enhanced frame timer 222 is implemented with a soft frame timer, where a pulse sequence is downloaded to memory, such as a RAM block, and the enhanced frame timer 222 reads the pulse sequence and generates signals for reset, transfer, and row selection lines of the image sensor 221. This has the advantage that new sequences can be added after the silicon for the image sensor 221, including the frame timer 222, has been released.

[0028] Therefore, aspects of the present invention provide new flexibility to the enhanced frame timer 222 associated with the image sensor 221. This flexibility allows exposure of advanced imaging modes (hyperspectral, fluorescence, etc.) to be separated from the exposure used for visible light imaging by the image sensor 221. This allows for various trade-offs, such as slowing down the frame rate of advanced imaging data, to improve sensitivity.

[0029] Another way the Enhanced Frame Timer 122 improves advanced imaging performance is through on-chip binning of pixels covered by a single filter element. On-chip binning provides noise reduction compared to individual sampling and binning in the digital domain. Typical image sensors bin pixels in either a monochrome sensor or a Bayer pattern. However, the hyperspectral filters available for mounting on the image sensor 221 are larger than the pixel size of a typical image sensor. As a result, the Enhanced Frame Timer 222 selects pixel cells to enable binning that is adjusted not only to the pixel size of the image sensor 221 but also to the desired filter pixel size.

[0030] In a stereo image sensor, the two active regions on the image sensor 221 are typically read synchronously, which minimizes artifacts in the displayed 3D video. However, when the stereo image sensor is used in combination with white light imaging and advanced imaging, the enhanced frame timer 222 allows for different exposures in the two active regions while combining the captured pixel data into a single stream for transmission.

[0031] The enhanced frame timer 122 also enables more sensitive and advanced imaging by covering different lines of the active region of the image sensor 221, or one of the active regions of a stereo image sensor, with a different filter material. For example, one active region may be set for visible imaging, and the other active region may be set for fluorescence imaging without a filter.

[0032] In most cases, it is desirable to acquire advanced imaging data simultaneously with the visible light scene. The enhanced frame timer 222 utilizes a method for interlacing different exposure settings for each color or row of a single image sensor with a conventional video image. The alternative pixel timing sequence used in the advanced imaging mode can also be used for high dynamic range visible light imaging, for example, by exposing the green pixels of a typical Bayer pattern differently.

[0033] Finally, the same frame timer enhancements used to enable advanced imaging can be applied to standard Bayer patterns using four-directional shared pixel cells, which provide a simple means of exposing different colors in different amounts to improve noise performance in conventional imaging.

[0034] Figure 2 is a high-level schematic diagram of a computer-assisted surgical system 200, such as the da Vinci® surgical system. In this example, a surgeon using a surgical console 210 remotely controls an endoscope 201 using a robotic manipulator arm 213. The surgeon can also operate surgical instruments attached to other robotic manipulator arms. Other components, cables, etc., related to the computer-assisted surgical system 200 exist but are not shown in Figure 2 to avoid prejudice to this disclosure. Further information regarding computer-assisted surgical systems can be found, for example, in U.S. Patent Application Publication No. 2008-0065105 (filed June 13, 2007; disclosing “Minimally Invasive Surgical System”) and U.S. Patent No. 6,331,181 (filed December 18, 2001; “Surgical Robotic Tools, Data Architecture, These documents can be found in the disclosures of “and Use” and both are incorporated herein by reference.

[0035] An illumination system (not shown) is coupled to the endoscope 201 or is instead contained within the endoscope 201. In one embodiment, the illumination system provides white light illumination, or a combination of white light illumination and alternative imaging mode illumination, such as hyperspectral illumination. In one embodiment, all or part of this light is coupled to at least one illumination path within the endoscope 201. In another embodiment, the illumination source is located at or near the tip of the endoscope 201. In one embodiment, both visible white light illumination and alternative imaging mode illumination are constant during the surgical procedure. In another embodiment, the visible illumination is constant over time, but the spectrum of the alternative imaging mode illumination changes over time.

[0036] In this embodiment, light from the endoscope 201 illuminates the tissue 203 of the patient 211. In one embodiment, the endoscope 201 is a stereoscopic endoscope including two light channels, e.g., a left light channel and a right light channel, which transmit light from the tissue 203 to an image sensor 221, and the image sensor 221 includes two sensing regions, one of which captures the left scene and the other captures the right scene. In another embodiment, the endoscope 201 is a monoscopic endoscope including a single light channel for transmitting light from the tissue to the image sensor 221, in which case the image sensor 221 includes a single sensing region.

[0037] As fully explained below, for both types of endoscopes, reflected white light is captured by the image acquisition system 220 as a visible light frame. The visible light frame contains visible scenes, such as scenes of tissue, and is sometimes referred to as a visible frame. Non-visible light and / or emitted light reflected from the tissue is captured by the image acquisition system 220 as an augmented light frame. The augmented light frame contains, for example, a hyperspectral scene of tissue 203 or other features within the field of view of the endoscope 201, or fluorescence from tissue 203. In another embodiment, the augmented light frame contains pixels that are exposed differently, which can be used to generate a high dynamic range scene. The augmented light frame is sometimes referred to as an augmented frame.

[0038] In one embodiment, the camera of the image acquisition system 220 is mounted on the proximal end of the endoscope 201. In another embodiment, the camera is mounted on the proximal end of the endoscope 201, where the camera includes at least a frame timer and an image sensor. Here, the proximal end means closer to the surgical site, and the proximal end means further away from the surgical site. In one embodiment, the camera acquires visible and augmented frames through the same front-end optical system. This is in contrast to systems that utilize a special front-end optical system to acquire, for example, hyperspectral frames.

[0039] Figure 3 is a more detailed illustration of an example embodiment of the computer-assisted surgical system 200 of Figure 2. In the embodiment of Figure 3, the computer-assisted surgical system 200 includes an illuminator which is a combined light source 310. The combined light source 310 includes a visible light illuminator 311, for example, a white light source, and a light-enhancing illuminator 312. The specific embodiments of the illuminators 311 and 312 are not important as long as the combined light source 310 has the capabilities that will be described more fully below.

[0040] In this embodiment, the combined light source 310 is used in conjunction with at least one illumination path within the stereoscopic endoscope 201 to illuminate the tissue 203. In one embodiment, the combined light source 310 has at least two operating modes: a normal observation mode and an enhanced observation mode.

[0041] In normal observation mode, the visible light illuminator 311 provides illumination that illuminates the tissue 203 with white light. The light-enhancing illuminator 312 is not used in normal observation mode.

[0042] In the enhanced observation mode, the visible light illuminator 311 provides illumination that illuminates the tissue 203 with white light. In one embodiment, the light-enhancing illuminator 312 provides illumination that illuminates the tissue 203 with hyperspectral light, such as near-infrared spectrum light, or light that excites fluorescence.

[0043] The use of near-infrared light as an example of hyperspectral illumination is illustrative and not intended to limit the scope to this particular embodiment. In consideration of this disclosure, a person familiar with the art may select hyperspectral illumination that makes non-prominent features of the captured visible frame more prominent in the captured enhanced frame.

[0044] In one embodiment, the visible light illuminator 311 includes light sources for each of the different visible color illumination components. In the case of a red-green-blue embodiment, in one example, the light sources are lasers: a red laser, two green lasers, and a blue laser. In one embodiment, the light from the visible light illuminator 311 has a spectrum formed such that the light appears to the human eye as having a purple hue. See International Publication No. 2015 / 142800, which is incorporated herein by reference.

[0045] The use of a laser in the visible light illuminator 311 is illustrative and not intended to limit it. The visible light illuminator 311 can also be implemented using, for example, multiple light-emitting diode (LED) sources instead of a laser. Alternatively, the visible light illuminator 311 can use a xenon lamp including an elliptical back reflector and a bandpass filter coating to produce broadband white illumination for visible scenes. The use of a xenon lamp is also illustrative and not intended to limit it. For example, a high-pressure mercury arc lamp, another arc lamp, or another broadband light source may be used.

[0046] The embodiment of the light-enhancing illuminator 312 depends on the light spectrum of interest. Typically, one or more laser modules, one or more light-emitting diodes, are used as the light-enhancing illuminator 312.

[0047] In normal and enhanced observation modes, light from the visible light illuminator 311, or light from both the visible light illuminator 311 and the light-enhancing illuminator 312, is directed to the connector 316. The connector 316 provides light to the illumination path of the stereoscopic endoscope 201, which then directs this light to the tissue 203. Each of the illumination paths of the connector 316 and the stereoscopic endoscope 201 can be implemented using, for example, a bundle of optical fibers, a single rigid or flexible rod, or optical fibers.

[0048] Light from the surgical site 203 (Figure 3) is transmitted to cameras 320L and 320R via stereoscopic light channels within the endoscope 201, such as the left and right light channels, or the first and second light channels. The use of two separate cameras 320L and 320R is for the purpose of facilitating explanation and discussion, and should not be interpreted as requiring two separate cameras or two separate image acquisition units. The components of cameras 320L and 320R can be combined into a single unit.

[0049] As fully explained below, the left camera 320L includes a left image sensor 321L. The left image sensor 321L captures light received from the left channel of the stereoscopic endoscope 302 as the left frame 322L. Similarly, the right camera 320R includes a right image sensor 321R. The right image sensor 321R captures light received from the right channel of the stereoscopic endoscope 302 as the right frame 322R. The left image sensor 321L and the right image sensor 321R may be separate sensors or different active regions of a single sensor. The use of left and right is intended to facilitate the distinction between the first and second sensors.

[0050] Camera 320L includes a first frame timer circuit 325L, sometimes called frame timer 325L, which in this embodiment is coupled to the left camera control unit 330L and the left image sensor 321L. Camera 320R includes a second frame timer circuit 325R, sometimes called frame timer 325R, which in this embodiment is coupled to the right camera control unit 330R and the right image sensor 321R. Using individual frame timers for each image sensor enhances imaging capabilities compared to a configuration using a common frame timer for all image sensors. Using separate frame timers 325L and 325R allows for the separation of exposure for advanced imaging modes (hyperspectral, fluorescence, etc.) captured by one image sensor from the exposure used for visible light imaging by the other image sensor. This allows for improved sensitivity through various trade-offs, such as slowing down the frame rate of advanced imaging data. Another way to improve advanced imaging performance using individual frame timers is to bin pixels covered by a single filter element on-chip. On-chip binning reduces noise compared to individual sampling and binning in the digital domain.

[0051] Camera 320L is coupled to the stereoscopic display 351 of the surgical console 210 by the left camera control unit 330L and the image processing module 340. The image processing module 340 is part of the image processing system 130. Camera 320R is coupled to the stereoscopic display 351 of the surgical console 210 by the right camera control unit 330R and the image processing module 340. Camera control units 330L and 330R receive signals from the system processing control module 362. The system processing control module 362 represents various controllers within the system 300.

[0052] The display mode selection switch 352 provides a signal to the user interface 361, which then passes the selected display mode to the system processing control module 362. Various controllers within the system processing control module 362 constitute the lighting controller 315, the left and right camera control units 330L and 330R to acquire the desired scene, and other elements within the image processing module 340 necessary to process the acquired scene, thereby presenting the scene requested by the surgeon on the stereoscopic display 351. The image processing module 340 implements an image processing pipeline equivalent to a known image processing pipeline.

[0053] The video output on the stereoscopic display 351 can be toggled between normal observation mode and enhanced observation mode by, for example, a foot switch, a double-click of a master grip used to control surgical instruments, voice control, and other switching methods. The toggle for switching observation modes is represented in Figure 3 as the display mode selection switch 352.

[0054] The central controller 360 and the system processing control module 362 are similar to those in conventional systems, except for the embodiments described below in full. Although described as the central controller 360, it should be understood that the central controller 360 can actually be implemented by any number of modules, each module may contain any combination of components. Each module and each component may include hardware, software running on a processor, firmware, or any combination of these three.

[0055] Furthermore, as described herein, the functions and operations of the central controller 360 and the system processing control module 362 may be performed by a single module or may be divided among different modules or even among different components of a module. When divided among different modules or components, the modules or components may be centralized in one location or distributed throughout the system 200 for the purpose of distributed processing. Thus, the central controller 360 and the system processing control module 362 should not be interpreted as requiring a single physical entity, as in some embodiments both are distributed throughout the system 200.

[0056] Further information relating to computer-assisted surgical systems can be found, for example, in U.S. Patent Application No. 11 / 762,165 (filed June 23, 2007; disclosing a "Minimally Invasive Surgical System"), U.S. Patent No. 6,837,883 (filed October 5, 2001; disclosing an "Arm Cart for Telerobotic Surgical System"), and U.S. Patent No. 6,331,181 (filed December 28, 2001; disclosing "Surgical Robotic Tools, Data Architecture, and Use"), all of which are incorporated herein by reference.

[0057] In Figure 3, the cameras 320L, 320R, and the combined light source 310 are shown as being external to the endoscope 201. However, in one embodiment, the cameras 320L, 320R, and the light source 310 are included in the tip of the endoscope 201 and are adjacent to the tissue 203. Also, the left image sensor 321L and the right image sensor 321R may be different active regions of the sensor area of ​​an integrated circuit chip including the left frame timer circuit 325L and the right frame timer circuit 325R.

[0058] The system controller 320 (Figure 3) is shown as a unified structure for illustrative purposes and ease of understanding. This is merely illustrative and not intended to be limiting. The various components of the system controller 320 can still perform the functions described, even when arranged separately.

[0059] Stereoscopic image acquisition using alternative frame timing In some embodiments, a first scene captured by the left image sensor 321L is presented to the left-eye viewer of the stereoscopic display 351, and a second scene captured by the right image sensor 321R is presented to the right-eye viewer of the stereoscopic display 351. For example, a normal color scene of the surgical site is presented to the user's left eye, and an augmented scene of the surgical site is presented to the user's right eye.

[0060] Typically, an enhanced scene captured by one image sensor will have significantly lower intensity than a color scene captured by the other image sensor. Previously, this intensity difference was compensated for by digitally processing the captured scenes. Unfortunately, this can introduce noise, for example, caused by amplifying low signal levels.

[0061] In this embodiment, frame timers 325L and 325R are configured to read data from the left image sensor 321L and data from the right image sensor 321R at different rates. For example, as shown in Figure 4, a visible color scene, i.e., a reflected white light scene, is captured by the left image sensor 321L at a normal rate, e.g., 60 frames per second. An enhanced scene, e.g., a fluorescent scene or a hyperspectral scene, is captured by the right image sensor 321R at a slower rate than the normal rate, e.g., 30 frames per second. Figure 4 shows embodiments of a rolling shutter using frame timer 325L for the left image sensor 321L and embodiments of a rolling shutter using frame timer 325R for the right image sensor 321R.

[0062] In this example, it is assumed that the left image sensor 321L and the right image sensor 321R each have (m+2) rows of pixels, i.e., (m+1) active rows and dummy rows. Thus, the active rows are numbered from 0 to m.

[0063] The frame timer 325L repeatedly provides signals on the transmit, reset, and select lines, causing the image sensor 321L to capture frames 401L, 402L, 403L, 404L, and 405L in the same way and at the same time. In this example, the capture of frame 402L, and in particular the capture of row zero (0) in frame 402L, is considered. The capture of each pixel row in frame 402L is the same as row zero.

[0064] As previously noted, with a rolling shutter, not all active rows of the image sensor 321L are captured simultaneously. For example, camera 320L does not have a mechanical shutter that prevents light from reaching the pixels after a predetermined time. Rather, each pixel row is read sequentially. This is indicated by the diagonal 402L-S of frame 402L. The diagonal 402L-S represents the rolling shutter for the image sensor 321L to capture frame 402L. Frames 401L, 403L, 404L, and 405L have equivalent rolling shutters 401L-S, 403L-S, 404L-S, and 405L-S, respectively.

[0065] A signal on the row's reset line is activated so that each pixel in the row can re-accumulate charge. The squares at the left end of each horizontal line in Figure 4 indicate that the signal on the reset line for that row is activated. Thus, square 402L-0-RST represents the activation of the reset signal for row 0 in frame 402L, thereby setting each pixel in row 0 to a known state and initiating the accumulation of charge corresponding to the light incident on that pixel.

[0066] The round dots at the right end of each horizontal line in Figure 4 indicate that the signal on the row selection line for that row is activated, and the value of each pixel in that row is read. Once each pixel in a row is read, the shutter for that row is effectively closed. Thus, the dot 402L-0-SLCT indicates that the row selection signal for row zero in frame 402L is activated, thereby reading the value of each pixel in row zero.

[0067] The time 402L-0-EXP, from when the pixels of row 0 in frame 402L are set to a known state until the row selection line for row 0 becomes active and the pixel value of row 0 is read, is the exposure time for that row. In this way, the frame timer 325L can control the exposure time of rows in a frame by controlling the time interval from when the row selection signal for a row in the previous frame is activated until the reset signal for a row in the current frame becomes active.

[0068] Once all active rows in a frame have been read, a dummy row from image sensor 321L is read. The time intervals used for reading the dummy row are 401L-BLNK for frame 401L, 402L-BLNK for frame 402L, 403L-BLNK for frame 403L, 404L-BLNK for frame 404L, and 405L-BLNK for frame 405. Blanking is a typical feature of video timing, but blanking is useful for processing and display, and it is not necessary to perform any blanking and dummy row reading in order to use any of the pixel timing sequences described here.

[0069] The operation of frame timer 325R regarding resetting and reading rows of pixels is equivalent to that of frame timer 325L described above, but various signals are activated at a slower rate. Frame 401R is captured at the same time interval as frames 402L and 403L, while frame 402R is captured at the same time interval as frames 404L and 405L.

[0070] Line 401R-S represents the rolling shutter for frame 401R. Square 402R-0-RST indicates that the reset signal for row 0 in frame 401R is activated, thereby setting each pixel in row 0 to a known state and initiating the accumulation of charge corresponding to the light incident on that pixel. Dot 402R-0-SLCT indicates that the row selection signal for row 0 in frame 401R is activated, thereby reading the value of each pixel in row 0. The time 401R-0-EXP, from when the pixels in row 0 in frame 401R are set to a known state until the row selection line for row 0 is activated and the pixel value for row 0 is read, is the exposure time for that row.

[0071] Once all active rows in frame 401R have been read, a dummy row from image sensor 321L is read. The time interval used for reading the dummy row is 401R-BLNK for frame 401R.

[0072] Thus, Figure 4 shows that frames from the left image sensor 321L are read at the normal rate, while frames from the right image sensor 321R are read at half the rate. This allows the right image sensor 321R to integrate incident light over a longer period, which improves the signal-to-noise ratio compared to when frames are captured at the normal rate with the right image sensor 321R and the captured signals are then digitally amplified.

[0073] Figure 5 is a more detailed timing diagram of the reset and selection signals generated by frame timers 325L and 325R. Note that the timing diagram is for frames of interest to show the different exposure times of the two image sensors. Figure 5 does not include all the signals from the frames in Figure 4.

[0074] The reference codes for the pulses in Figure 5 are the same as the corresponding reference codes in Figure 4. However, Figure 5 has some additional reference codes. The key points regarding the reference codes in Figures 4 and 5 are as follows: xxxy-s-name, Here, xxx is the reference numeral for the frame in Figure 4. In this example, y represents the channel (right or left), 's' is the row number, ranging from 0 to m for the active row, and 'D' for a dummy row. Then, Name, RST = Reset Row, SLCT = Select Row, EXP = Exposure Time.

[0075] The frame timer 325L sequentially generates active row reset signals 401L-0-RST to 401L-m-RST for each row from 0 to m of the image sensor 321L. Following the exposure time for each row, the frame timer 325L sequentially generates active row selection signals 401L-0-SLCT to 401L-m-SLCT for each row from 0 to m of the image sensor 321L.

[0076] After each active row is reset, the frame timer 325L generates an active dummy row reset signal 401L-D-RST for the dummy rows of the image sensor 321L, and after the exposure time, the frame timer 325L generates an active row selection signal 401L-D-SLCT for the dummy rows of the image sensor 321L. After generating the dummy row signals, the frame timer 325L continues to generate row reset signals and row selection signals for each subsequent frame captured by the image sensor 321L.

[0077] The operation of frame timer 325R differs from the operation of frame timer 325L. Frame timer 325L sequentially generates active row reset signals 401R-0-RST to 401L-m-RST for each row from 0 to m of the image sensor 321R. After that, frame timer 325R generates active reset signals for dummy rows until it is time to stop generating active row reset signals or to start acquiring the next frame.

[0078] After the image sensor 321R has finished acquiring the preceding frame, the frame timer 325R generates a dummy row selection signal 401R-D-SLCT until the exposure time 401R-0-EXP for the 0th row of the image sensor 321R has elapsed. Thereafter, the frame timer 325R sequentially generates active row selection signals 401R-0-SLCT to 401L-m-SLCT for each row from 0 to m of the image sensor 321R.

[0079] In this example, the exposure time for the frame captured by image sensor 321R is twice as long as the exposure time for the frame captured by image sensor 321L. However, this approach of using two image sensors to capture scenes with different exposures can be generalized as shown in Figure 6.

[0080] In Figure 6, the frame timer 325L is configured to sequentially capture N frames (from frame 0 to frame (N-1)) from the image sensor 321L, while the frame timer 325R captures one frame (frame 0) from the image sensor 321R. Here, in one embodiment, N is a positive number greater than zero. Thus, the exposure time of the frames captured by the image sensor 321R is N times the exposure time of the frames captured by the image sensor. Figure 5 shows the case where N is 2.

[0081] Pixel binning The pixel binning configuration described below can be implemented in the stereoscopic computer-assisted surgical system 200 of Figure 3 or the monoscopic system 700 of Figure 7. In Figure 7, the image sensor 321, image 322, and camera control unit 330 are equivalent to the image sensors 321R, 321L, frames 322R, 322L, and camera control units 330R, 330L, so a description of these elements will not be repeated here. Similarly, the image processing module 740, the surgical console 714 including the display 751, the central controller 760, and the system processing control module 762 are equivalent to the corresponding elements in Figure 3 for either the left or right channel of Figure 3. The endoscope 701 is similar to the endoscope 302, except that it has only a single optical channel that transmits light from the tissue 203 to the camera 720. Thus, the monoscopic system 700 is equivalent to the system in Figure 3 with one of the left or right channels of Figure 3 removed. Therefore, since the explanation would be a repetition of the explanation of the elements in Figure 3, we will not provide further details.

[0082] Multiple pixel binning using Bayer color filter arrays Figure 8A shows a typical portion of a Bayer color filter on a CMOS image sensor including a novel frame timer 825A and a four-way shared pixel cell. Thus, Figure 8A is an example of a partial image acquisition unit having an image sensor 821A including a Bayer color filter array and a frame timer 825A. The image sensor 821A and frame timer 825A are examples of image sensor 321L and frame timer 325L, image sensor 321R and frame timer 325R, or image sensor 321 and frame timer 325.

[0083] Each position within the image sensor contains multiple pixels. In Figure 8, only four positions (0,0), (0,1), (1,0), and (1,1) are shown. Each position contains four pixels connected to a shared column line, which constitutes a four-directional shared pixel cell. Other positions of the image sensor 821A, not shown, are arranged in a corresponding manner.

[0084] In this example, each pixel is covered by a filter of a Bayer color filter array. As is known, in a Bayer color filter array, 50% of the filters are green filters, 25% are red filters R, and 25% are blue filters B. In this example, for the sake of discussion, the green filters are divided into a first green filter Gr and a second green filter Gb. In this example, the two green filters use the same filter dye and allow the same wavelength range to pass through. There is a one-to-one correspondence between the filters of the Bayer color filter array and the pixels of the image sensor 821A, meaning that in this embodiment, each pixel of the image sensor 821A is covered by a different filter of the Bayer color filter array. Although a Bayer color filter array is used as an example, the color filter array does not have to be in this particular configuration. Color filter arrays with different colors or different ratios of different colors can also be used in the applications described herein.

[0085] Pixels covered by a red filter R are called red pixels R. Pixels covered by a first green filter Gr are called first green pixels Gr. Pixels covered by a second green filter Gb are called second green pixels Gb. Pixels covered by a blue filter B are called blue pixels B. Thus, in Figure 8A, each position includes a red pixel, first and second green pixels, and a blue pixel. Also, in Figures 8A and 8B, rows are shown as extending vertically, and columns as extending horizontally. This is for the sake of simplicity and should not be interpreted as restricting the rows and columns of the image sensor to a particular direction. The configuration described more fully below operates the same regardless of the row and column direction.

[0086] Each row driver of the image sensor 821A is connected to multiple different pixel rows. The first transmit line Tx_1 connects the row driver to each red pixel in the second row connected to the row driver. The second transmit line Tx_2 connects the row driver to each second green pixel in the second row connected to the row driver. The third transmit line Tx_3 connects the row driver to each first green pixel in the first row connected to the row driver. The fourth transmit line Tx_4 connects the row driver to each blue pixel in the first row connected to the row driver.

[0087] The RESET line connects the row driver to the SHARED shared column driver at each position of the two rows associated with the row driver. The SELECT line connects the row driver to the SHARED shared column driver at each position of the two rows associated with the row driver. In one embodiment, each SHARED shared column driver is a single floating diffuse charge storage node.

[0088] The frame timer 825A is connected to each row driver of the image sensor 821A by multiple lines. In this example, the multiple lines include 21 lines.

[0089] Ten of the 21 lines are row address lines ROW_ADDR<9,0>. Row address lines ROW_ADDR<9,0> transmit the address of the row being accessed by the frame timer 825A.

[0090] Three of the 21 lines are the row selection line ROW_SELECT, the reset set line RST_SET, and the reset clear line RST_CLR. An active signal on the row selection line ROW_SELECT drives an active signal on the selection line SELECT to the row driver addressed by the address of the row address line ROW_ADDR<9,0>. An active signal on the reset set line RST_SET drives an active signal on the reset line RESET to the row driver addressed by the address of the row address line ROW_ADDR<9,0>.

[0091] An active signal on the reset clear line RST_CLR causes the row driver, addressed by the address on the row address line ROW_ADDR<9,0>, to drive an inactive signal on the reset line RESET.

[0092] Four of the 21 lines are transmit set lines TX_SET<4,1>, and the other four lines are transmit clear lines TX_CLR<4,1>. Each transmit set line TX_SET<4,1> is coupled via a row driver to one of the following distinct lines: the first transmit line Tx_1, the second transmit line Tx_2, the third transmit line Tx_3, and the fourth transmit line Tx_4. For example, transmit set line TX_SET(1) is coupled to the first transmit line Tx_1, transmit set line TX_SET(2) is coupled to the second transmit line Tx_2, and so on. Similarly, each transmit set line TX_CLR<4,1> is coupled via a row driver to one of the following distinct lines: the first transmit line Tx_1, the second transmit line Tx_2, the third transmit line Tx_3, and the fourth transmit line Tx_4. For example, the transmit clear line TX_CLR(1) is coupled to the first transmit line Tx_1, the transmit clear line TX_CLR(2) is coupled to the second transmit line Tx_2, and so on.

[0093] An active signal on the transmit set line TX_SET(1) causes the row driver, addressed by the address on the row address line ROW_ADDR<9,0>, to drive an active signal on the first transmit line Tx_1, and similarly on the other transmit set lines. An active signal on the transmit clear line TX_CLR(1) causes the row driver, addressed by the address on the row address line ROW_ADDR<9,0>, to drive an inactive signal on the first transmit line Tx_1, and similarly on the other transmit lines.

[0094] The reset set line RST_SET, reset clear line RST_CLR, transmit set line TX_SET<4,1>, and transmit clear line TX_CLR<4,1> allow pulses to be sent to different lines during a single line time, and the length of those pulses may be longer than the time between them (pulses). When the transmit set line TX_SET1 is activated, a specific first transmit line Tx_1 line with a matching line address becomes active, and the same first transmit line Tx_1 is addressed again and remains high until the transmit clear line TX_CLR1 is activated. Lines TX_SETx and TX_CLRx, and RST_SET and RST_CLR are driven by short pulses that control the timing of the edges of longer pulses such as line Tx_1. Thus, these lines allow pulses to be sent to different lines during a single line time, and the length of those pulses may be longer than the time between them (pulses).

[0095] In this example, timing (control) uses a specific type of row driver circuit to address each row and use latches on each row signal to generate pulses that go to the pixel row control lines TXn, SEL, and RESET. There are other ways to implement this logic. Specifically, the same timing for the pixel control lines can be generated with other types of logic, and the same concepts apply.

[0096] Furthermore, these examples use a four-way shared pixel cell, where the output portion of the four-way shared pixel cell is shared among the four pixels within the Bayer group. This is particularly useful for alternate frame timing, but the examples shown here can also be applied to other pixel-sharing configurations.

[0097] The layout of the pixel array, row driver, input lines to the row driver, and output lines to the row driver of the image sensor 821A is known and therefore not described in further detail herein. A novel aspect is a sequence of signals provided on the input lines to the image sensor 821A by the frame timer 825A, which provides enhanced image sensor timing and consequently enhanced imaging capability.

[0098] Figure 8A represents an image acquisition device including an image sensor coupled to a frame timer. The image sensor includes multiple pixel rows and a visible light color filter array. The visible light color filter array includes multiple different visible light color filters, which are represented in Figure 8A by red, two green, and blue visible light color filters. Multiple pixel rows include multiple pixel cells, each of which contains multiple pixels. In the example in Figure 8A, the pixel cells are identified by positions (0,0), (0,1), (1,0), and (1,1). Each pixel of the multiple pixels in a pixel cell is covered by one of the multiple different visible light color filters. In the example in Figure 8A, each of the multiple pixels in the pixel cell at position (0,0) is covered by one of the red, two green, and blue visible light color filters. The frame timer is coupled to the image sensor and provides the image acquisition timing signal to the image sensor.

[0099] Figure 9A shows a timing diagram of pixel binning of four pixels at the row position in Figure 8A as part of a rolling shutter. In this embodiment, the frame timer 825A simultaneously transmits an active signal on each transmit set line TX_SET<4,1> and an active signal on the reset set line RST_SET. In response to these signals, as shown in Figure 9A, the addressed row driver simultaneously drives the active transmit signals on the first transmit line Tx_1, the second transmit line Tx_2, the third transmit line Tx_3, and the fourth transmit line Tx_4, respectively, and the active reset signal on line RESET.

[0100] To read the pixels, after an appropriate exposure time, the frame timer 825A simultaneously transmits an active signal on each transmit set line TX_SET<4,1> and an active signal on the row selection line ROW_SELECT. In response to these signals, as shown in Figure 9A, the addressed row driver simultaneously drives the active transmit signals on the first transmit line Tx_1, the second transmit line Tx_2, the third transmit line Tx_3, and the fourth transmit line Tx_4, respectively, and the active signal on the row selection line SELECT.

[0101] Since all four pixels at a given position are simultaneously connected to a shared column line and read, for example, at the same time, this improves the signal-to-noise level compared to integrating the four pixels in the analog stage and performing the same integration during the digital processing stage. However, this combination of pixels comes with a trade-off: instead of reducing the noise level by 50%, all color information is lost, as well as some spatial resolution.

[0102] When a single image sensor is used, as shown in Figure 7, this pixel binning can be used to improve the signal-to-noise ratio of the captured scene. When a stereoscopic image sensor is used, as shown in Figure 3, one image sensor can be used to capture a color scene at a normal frame rate, and the other sensor can be used to capture the scene at a slower frame rate with pixel binning. For example, as shown in Figure 5, the reset and selection signals for the slower frame rate sensor are generated for each row, as described with respect to Figure 9A, so that the slower frame rate and pixel binning are combined. Alternatively, when a stereoscopic image sensor is used, in one embodiment, both image sensors capture frames at the same frame rate, e.g., a normal frame rate, but one of the image sensors uses pixel binning. Thus, for each frame time interval, a color frame with full spatial resolution is captured along with a monochrome frame with a lower noise level. Both frames contain the same scene, and the spatial relationship between the two frames is known.

[0103] Multiple pixel binning using visible light color filter arrays and alternative light filter arrays In other embodiments, an interleaved array of visible light color filters and alternative light (hyperspectral or other wavelength band) filters on a CMOS image sensor is used, including a novel frame timer and four-way shared pixel cells.

[0104] In this specification, an alternative light filter refers to a filter that filters out light other than visible light. An alternative light filter comprises a group of individual alternative light filters, each configured to cover one or more image sensor pixels, typically multiple image sensor pixels. Individual alternative light filters are sometimes referred to as alternative light filter pixels. Similarly, a visible light color filter array comprises a group of different individual visible light color filters.

[0105] Visible color filter arrays using organic dyes, such as Bayer color filter arrays, are well-known and can be applied to small (<2 μm) pixels. Other filter techniques that allow selection of other wavelengths, narrow wavelength bands, or light polarization are also well-known, but the manufacturing processes required for these alternative filters prevent them from producing filter pixel sizes comparable to the pixel size of the image sensor, and therefore they are not currently applicable to the small pixel structures typically found in image sensors. Typically, the pixel size of the alternative filter is a multiple of the pixel size of the image sensor.

[0106] To overcome this problem with image sensors used in endoscopes, a single image sensor is used to capture both conventional color images and images in other wavelength bands using a filter structure as shown in Figure 9B. To compensate for the larger pixel size of the alternative light filters, a single-pixel red-green-blue (RGB) filter is interleaved with an array of individual alternative light filters, in this example, each individual alternative light filter covers a 2x2 pixel cell of the image sensor.

[0107] Through a specific structure of an image sensor with a four-way shared pixel cell, a matching configuration (arrangement) of the filter array, and the use of a specific timing sequence in the sensor's frame timer, the noise advantage of an alternative filtered signal can be obtained without sacrificing noise or frame rate of the RGB pixels of the image sensor ray. This operation utilizes a four-way shared pixel connection. As described above with respect to Figure 8A, the four-way shared pixel cell shares a single floating diffuse charge storage node among a group of four pixels.

[0108] A floating diffuse charge storage node, sometimes called a shared column driver, SHARED, can be reset by pulsing the reset line RESET and buffered and connected to a column output line by pulsing the select line SELECT. A floating diffuse charge storage node SHARED can also be connected to any or all of the four surrounding pixels by pulsing one or more corresponding transmit lines.

[0109] The four-way shared pixel cell has the flexibility to connect any of the four surrounding pixels to the floating diffuse charge storage node SHARED (and therefore to the reset and / or output), so that the connection can be made at pixels on a transmit line using pulses from the frame timer 825B, thereby ensuring that when each transmit line connected to a row of pixels is pulsed, the pixels in the row connected to the floating diffuse charge storage node SHARED are connected in a pattern. TX_1: 1 - - 1 1 - - 1 1 - - … - - 1 TX_2: - 2 2 - - 2 2 - - 2 2 … 2 2 - Here, TX_1 refers to the transmission line to one row of multiple 4-way shared pixel cells, and TX_2 refers to the transmission line to the other row of multiple 4-way shared pixel cells. Thus, as shown in Figure 8B, the filters are then arranged in a staggered pattern, thereby splitting the four colors in the Bayer array between two different pixel shared cells. The two color pixels in each row are connected to different floating diffuse charge storage nodes SHARED, so that the two color pixels can be read simultaneously using the appropriate timing sequence.

[0110] The division of the Bayer array into four colors between two different pixel-shared cells forces the division of the alternative filter pixels as well, but the charges of two pixels connected in a column can be combined in the floating diffuse charge storage node SHARED during readout without adding extra noise. The pair of columns corresponding to a single filter position can be combined as a voltage at the output of the column amplifier (before the signal is digitized). The final result is low-noise readout of the alternative filter pixels without losing the spatial or temporal resolution of other pixels in the array. Thus, in this example, the individual hyperspectral filters that make up the hyperspectral filter array are staggered (shifted) relative to the individual parts of the color array filter, so that when one row is read, the charges of two rows of hyperspectral pixels can be binned, but the color pixels cannot be read individually and are not binned.

[0111] In addition to the selective pixel binning of charge regions described above, exposure time can also be selectively extended by a similar pulse sequence of transmit line TX_x, which omits specific reset and read sequences of the transmit line TX_x toward pixels containing individual alternative light filters.

[0112] Thus, Figure 8B shows a typical portion of the Bayer color filter array and alternative optical filter array, such as a hyperspectral filter array, on a CMOS image sensor including a novel frame timer 825B and four-directional shared pixel cells. In this example, the frame timer 825B is configured to generate the pulse sequence shown in Figure 9B.

[0113] Figure 8B shows a partial example of an image acquisition unit having an image sensor 821B including a Bayer color filter array and an alternative light filter array, and a frame timer 825B. The image sensor 821B and frame timer 825B are also examples of image sensor 321L and frame timer 325L, image sensor 321R and frame timer 325R, or image sensor 321 and frame timer 325.

[0114] Each position within the image sensor 821B contains multiple pixels. In Figure 8B, only six positions (0,0), (0,1), (0,2), (1,0), (1,1), and (1,2) are shown, where each position contains four pixels connected to a shared column line, which is a four-directional shared pixel cell. Other positions of the image sensor 821B not shown are arranged in a corresponding manner.

[0115] In this example, some pixels within a four-way shared pixel cell are covered by filters from a Bayer color filter array, while other pixels within the four-way shared pixel cell are covered by filters from an alternative light filter array. As mentioned above, pixels covered by the red filter R of the Bayer color filter array are called red pixels R. Pixels covered by the first green filter Gr of the Bayer color filter array are called first green pixels Gr. Pixels covered by the second green filter Gb of the Bayer color filter array are called second green pixels Gb. Pixels covered by the blue filter B of the Bayer color filter array are called blue pixels B.

[0116] Pixels within a group of pixels covered by individual alternative light filters in an alternative light filter array are represented by the same reference code Pj, where j is an integer, and are called alternative light filtered pixels. As shown above, in this example, each alternative light filter occupies a 2x2 pixel cell of the image sensor 821B, but the pixel Pj is divided between adjacent 4-directional shared pixel cells. Thus, in the image sensor 821B, each 4-directional shared pixel cell at positions (0,0), (0,1), (0,2), (1,0), (1,1), and (1,2) contains multiple visible light color filtered pixels and multiple alternative light filtered pixels.

[0117] In particular, the four-way shared pixel cell at position (0,0) contains a red pixel R, a first green pixel Gr, and two alternative light-filtered pixels P1, P1. The four-way shared pixel cell at position (0,1) contains a blue pixel B, a second green pixel Gb, and two alternative light-filtered pixels P1, P1. Thus, as described above, the four Bayer-filtered pixels, red pixel R, first green pixel Gr, second green pixel Gb, and blue pixel B are divided between two adjacent four-way shared pixel cells. Similarly, the two pixels P1, P1 covered by a single alternative light-filter array pixel are located in each of the two adjacent four-way shared pixel cells.

[0118] Each row driver of the image sensor 821B is connected to multiple pixel rows. The first color transmission line COLOR_Tx0 connects row driver 0 to each blue pixel B and each first green pixel Gr of the first row (row 0) of the image sensor 821B. The first alternate filter transmission line HYP_Tx0 connects row driver 0 to each alternate light filtering pixel of the first row. The second color transmission line COLOR_Tx1 connects row driver 0 to each red pixel R and each second green pixel Gb of the second row (row 1) of the image sensor 821B. The second alternate filter transmission line HYP_Tx1 connects row driver 0 to each alternate light filtering pixel of the second row.

[0119] The third color transmission line, COLOR_Tx2, connects row driver 1 to each blue pixel B and each first green pixel Gr in the third row (row 2) of the image sensor 821B. The third alternate filter transmission line, HYP_Tx2, connects row driver 1 to each alternate light-filtered pixel in the third row. The fourth color transmission line, COLOR_Tx3, connects row driver 1 to each red pixel R and each second green pixel Gb in the fourth row (row 3) of the image sensor 821B. The fourth alternate filter transmission line, HYP_Tx3, connects row driver 1 to each alternate light-filtered pixel in the fourth row. The line arrangement connecting row drivers 0 and 1 to pixel rows is repeated below the column of the image sensor 821B.

[0120] Thus, the transmission line connects to the pixels of adjacent pixel rows in a pattern with a frame time of 825B, as described above, providing appropriate pulses. COLOR_Tx0 1 - - 1 1 - - 1 1 - - … - - 1 HYP_Tx0 - 2 2 - - 2 2 - - 2 2 … 2 2 - COLOR_Tx1 1 - - 1 1 - - 1 1 - - … - - 1 HYP_Tx1 - 2 2 - - 2 2 - - 2 2 … 2 2 - COLOR_Tx2 - 2 2 - - 2 2 - - 2 2 … 2 2 - HYP_Tx2 1 - - 1 1 - - 1 1 - - … - - 1 COLOR_Tx3 - 2 2 - - 2 2 - - 2 2 … 2 2 - HYP_Tx3 is 1 - - 1 1 - - 1 1 - - … - - 1.

[0121] The first reset line RESET_01 connects row driver 0 to shared column driver SHARED at each position in the first and second pixel rows. The first selection line SELECT_01 connects row driver 0 to shared column driver SHARED at each position in the first and second pixel rows. As described above, in one embodiment, each shared column driver SHARED is a single floating diffuse charge storage node.

[0122] The second reset line, RESET, connects row driver 1 to the shared column driver SHARED at each position in the third and fourth pixel rows. The second select line, SELECT, connects row driver 1 to the shared column driver SHARED at each position in the third and fourth pixel rows.

[0123] The frame timer 825B is connected to each row driver of the image sensor 821A by multiple lines. In this example, the multiple lines include 21 lines.

[0124] Ten of the 21 lines are row address lines ROW_ADDR<9,0>. Row address lines ROW_ADDR<9,0> transmit the address of the row being accessed by frame timer 825B.

[0125] Three of the 21 lines are the row selection line ROW_SELECT, the reset set line RST_SET, and the reset clear line RST_CLR. An active signal on the row selection line ROW_SELECT drives an active signal on the selection line to the row driver addressed by the address of the row address line ROW_ADDR<9,0>. An active signal on the reset set line RST_SET drives an active signal on the reset line to the row driver addressed by the address of the row address line ROW_ADDR<9,0>.

[0126] An active signal on the reset clear line RST_CLR causes the row driver, addressed by the address of the row address line ROW_ADDR<9,0>, to drive an inactive signal on the reset line.

[0127] Four of the 21 lines are transmit set lines TX_SET<4,1>, and the other four lines are transmit clear lines TX_CLR<4,1>. Each transmit set line TX_SET<4,1> is connected via a row driver to one of the first, second, third, and fourth transmit lines, which are connected to the addressed row driver.

[0128] An active signal on the transmit set line TX_SET(1) causes the row driver addressed by the address on the row address line ROW_ADDR<9,0> to drive an active signal on the first transmit line, and similarly on the other transmit set lines. An active signal on the transmit clear line TX_CLR(1) causes the row driver addressed by the address on the row address line ROW_ADDR<9,0> to drive an inactive signal on the first transmit line, and similarly on the other transmit lines.

[0129] Thus, in the image sensor 821B, as shown in Figure 8A, the usual connections to the pixels in each row of the paired transmit lines are rearranged so that pixels of the same filter type (normal visible light color filter array or alternative light filter array) are connected to separate column drivers and readout circuits for each row. This connection provides separate timing control for the normal light filter array and the alternative light filter array. Figure 9B is a timing diagram showing the operation of the image sensor 821B.

[0130] The exemplary pulse sequence shown in Figure 9B from frame timer 825B demonstrates the reset of pixels on pixel rows 0 and 1, followed by the readout of those pixels. Pixel rows 0 and 1 are the rows connected to row driver 0. When the transmit pulse coincides with the reset pulse, the active transmit pulse resets both the floating diffuse charge storage node SHARED and the photodiode connected to the floating diffuse charge storage node SHARED. When the reset pulse occurs alone, the reset pulse resets only the floating diffuse charge storage node SHARED, which is necessary for correlated double sampling (CDS) to reduce readout noise.

[0131] Exemplary pulse sequence in Figure 9B: 1. Reset the color pixels in row 0. 2. Reset the color pixels in row 1. 3. Reset the alternative filter pixels for row 0 and row 1 together. 4. Later, read the color pixels of row 0. 5. Read the color pixels in row 1. 6. Read the alternative filter pixels from row 0 and row 1 and bin them together. Other exposures can be achieved by adjusting the delay between the reset and read sequences and selectively omitting the reset / read sequences for certain pixel types.

[0132] Thus, Figures 8B and 9B illustrate an image acquisition device including an image sensor coupled to a frame timer. The image sensor includes multiple pixel rows, a visible light color filter array, and an alternate light filter array. Each of the multiple pixel rows contains multiple pixel cells. For example, in Figure 8B, there are multiple pixel cells at positions (0,0), (0,1), (0,2), (1,0), (1,1), and (1,2). Each of the multiple pixel cells contains multiple pixels (four pixels in the example in Figure 8B).

[0133] A visible light color filter array contains several different visible light color filters, which are represented in Figure 8A by red, two green, and blue visible light color filters. An alternative light filter array contains several individual alternative light filters. One of the individual alternative light filters covers both a first set of pixels in a first pixel cell of a plurality of pixel cells, and a second set of pixels in a second pixel cell of a plurality of pixel cells. The first pixel cell is adjacent to the second pixel cell. For examples of individual alternative light filters, see the pixel cells at positions (0,0) and (0,1). Each of the several different individual visible light color filters covers different pixels in the first and second sets of pixels. The pixels covered by an individual visible light color filter of the several individual visible light color filters are different from the pixels covered by an individual alternative light filter.

[0134] The frame timer is coupled to the image sensor and provides the image acquisition timing signal to the image sensor. For example, the frame timer is configured to simultaneously reset pixels in first and second pixel cells, which are covered by one of several different individual visible light color filters.

[0135] As shown above, data binning and the use of a combination of a visible light color filter array and an alternative light filter array can also be carried out in other ways using shared pixel cells. For example, Figure 8C shows a typical portion of a Bayer color filter array and an alternative light filter array, such as a hyperspectral filter array, on a CMOS image sensor including a novel frame timer 825C and a four-directional shared pixel cell. As previously stated, the Bayer color filter array is an example of a visible light color filter array, and the use of a Bayer color filter array is not intended to limit the visible light color filter array to a specific combination of color filters described. Also, Figure 8C is an example of a portion of an image acquisition unit having an image sensor 821C and a frame timer 825C, including a Bayer color filter array and an alternative light filter array. The image sensor 821C and frame timer 825C are also examples of image sensor 321L and frame timer 325L, image sensor 321R and frame timer 325R, or image sensor 321 and frame timer 325.

[0136] Each position within the image sensor 821C contains multiple pixels. In Figure 8C, only six positions (0,0), (0,1), (0,2), (1,0), (1,1), and (1,2) are shown, where each position contains four pixels connected to a shared column line, which is a four-directional shared pixel cell. Other positions of the image sensor 821C not shown are arranged in a corresponding manner.

[0137] In this example, in a pair of rows, alternating four-way shared pixel cells are covered by a portion of the visible light color filter array, and alternating four-way shared pixel cells are covered by individual alternative light filters of the alternative light filter array. As mentioned above, when the visible light color filter array is a Bayer color filter array, pixels within a four-way shared pixel cell are covered by a portion of the Bayer color filter array. Specifically, pixels covered by the red filter R of the Bayer color filter array are called red pixels R. Pixels covered by the first green filter Gr of the Bayer color filter array are called first green pixels Gr. Pixels covered by the second green filter Gb of the Bayer color filter array are called second green pixels Gb. Pixels covered by the blue filter B of the Bayer color filter array are called blue pixels B. When all pixels within a four-way shared pixel cell are covered by a portion of the visible light color filter array, the pixel is called a visible light color filtered pixel cell.

[0138] Pixels within a four-directional shared pixel cell that are covered by a portion of the individual alternative light filter cells of the alternative light filter array are represented by the same reference symbol Pj, where j is an integer, and are called alternative light-filtered pixel cells. As shown above, in this example, the individual alternative light filters cover all pixels in the four-directional shared pixel cells of the image sensor 821B. Thus, in this example, there are visible light color-filtered pixel cells at positions (0,0), (1,1), and (0,2), while there are alternative light-filtered pixel cells at positions (0,1), (1,0), and (1,2).

[0139] Each row driver of the image sensor 821C is connected to multiple pixel rows. In the previous example, each row driver had two transmit lines connected to the pixel rows. In this example, each row driver has four transmit lines connected to the pixel rows. Therefore, in this example, row drivers 0 and 1 from the previous example are combined into a single row driver 0 / 1, etc.

[0140] The first transmit line TXA_0 connects row drivers 0 / 1 to each first green pixel Gr in the first row (row 0) of the image sensor 821C, for example, to every fourth pixel in the first row starting from the first pixel. The second transmit line TXB_0 connects row drivers 0 / 1 to each blue pixel B in the first row of the image sensor 821C, for example, to every fourth pixel in the first row starting from the second pixel. The third transmit line TXC_0 connects row drivers 0 / 1 to each first alternate optical filter processing pixel Px-1 (where x is equal to 1-3 in Figure 8C) in each alternate optical filter processing pixel cell of the first row of the image sensor 821C, for example, to every fourth pixel in the first row starting from the third pixel. The fourth transmit line TXD_0 connects row drivers 0 / 1 to each second alternate optical filter pixel Px-2 of each alternate optical filter pixel cell in the first row of the image sensor 821C, for example, every fourth pixel in the first row, starting from the fourth pixel.

[0141] The fifth transmit line TXA_1 connects row drivers 0 / 1 to each red pixel R in the second row (row 1) of the image sensor 821C, for example, to every fourth pixel in the second row, starting from the first pixel. The sixth transmit line TXB_1 connects row drivers 0 / 1 to each second green pixel Gb in the second row of the image sensor 821C, for example, to every fourth pixel in the second row, starting from the second pixel. The seventh transmit line TXC_1 connects row drivers 0 / 1 to each third alternative optical filtering pixel Px-3 (where x is equal to 1-3 in Figure 8C) of each alternative optical filtering pixel cell in the second row of the image sensor 821C, for example, to every fourth pixel in the second row, starting from the third pixel. The eighth transmit line TXD_1 connects line drivers 0 / 1 to the fourth alternate light filtering pixel Px-4 of each alternate light filtering pixel cell in the second row of the image sensor 821C, for example, every fourth pixel in the second row, starting from the fourth pixel.

[0142] The first reset line RESET_01 connects row drivers 0 / 1 to the shared column driver SHARED at each position in the first and second pixel rows. The first selection line SELECT_01 connects row drivers 0 / 1 to the shared column driver SHARED at each position in the first and second pixel rows. As described above, in one embodiment, each shared column driver SHARED is a single floating diffuse charge storage node.

[0143] Regarding row drivers 2 / 3, the first transmit line TXA_2 connects row drivers 2 / 3 to each first alternate optical filter processing pixel Px-1 of the third row (row 2) of the image sensor 821C (where x is equal to 1-3 in Figure 8C), for example, to every four pixels in the third row starting from the first pixel. The second transmit line TXB_2 connects row drivers 2 / 3 to each second alternate optical filter processing pixel Px-2 of each alternate optical filter processing pixel cell in the third row of the image sensor 821C, for example, to every four pixels in the third row starting from the second pixel. The third transmit line TXC_2 connects row drivers 2 / 3 to each first green pixel Gr of each visible light color filter processing pixel cell in the third row of the image sensor 821C, for example, to every four pixels in the third row starting from the third pixel. The fourth transmit line TXD_2 connects the row drivers 2 / 3 to each blue pixel B of each visible light color filtering pixel cell in the third row of the image sensor 821C, for example, every fourth pixel in the third row, starting from the fourth pixel.

[0144] Continuing with respect to row drivers 2 / 3, the fifth transmit line TXA_3 connects row drivers 2 / 3 to each third alternative optical filter processing pixel Px-3 of the fourth row (row 3) of the image sensor 821C (where x is equal to 1-3 in Figure 8C), for example, every four pixels in the fourth row starting from the first pixel. The sixth transmit line TXB_3 connects row drivers 2 / 3 to each fourth alternative optical filter processing pixel Px-4 of each alternative optical filter processing pixel cell in the fourth row of the image sensor 821C, for example, every four pixels in the fourth row starting from the second pixel. The seventh transmit line TXC_3 connects row drivers 2 / 3 to each red pixel R of each visible light color filter processing pixel cell in the fourth row of the image sensor 821C, for example, every four pixels in the fourth row starting from the third pixel. The eighth transmit line TXD_3 connects the row drivers 2 / 3 to the second green pixel Gb of each visible light color filter processing pixel cell in the fourth row of the image sensor 821C, for example, every fourth pixel in the fourth row, starting from the fourth pixel.

[0145] The second reset line, RESET_23, connects row drivers 2 / 3 to the shared column driver SHARED at each position in the third and fourth pixel rows. The second select line, SELECT_23, connects row drivers 2 / 3 to the shared column driver at each position in the third and fourth pixel rows. Additional row drivers are not shown in Figure 8C because the configurations of row drivers 0 / 1 and 2 / 3 are repeated below the columns.

[0146] The frame timer 825C is connected to each row driver of the image sensor 821A by multiple lines. In this example, the multiple lines include 21 lines.

[0147] Ten of the 21 lines are row address lines ROW_ADDR<9,0>. Row address lines ROW_ADDR<9,0> transmit the address of the row being accessed by frame timer 825C.

[0148] Three of the 21 lines are the row selection line ROW_SELECT, the reset set line RST_SET, and the reset clear line RST_CLR. An active signal on the row selection line ROW_SELECT drives an active signal on the selection line to the row driver addressed by the address of the row address line ROW_ADDR<9,0>. An active signal on the reset set line RST_SET drives an active signal on the reset line to the row driver addressed by the address of the row address line ROW_ADDR<9,0>.

[0149] The active signal on the reset clear line RST_CLR drives the inactive signal on the reset line to the row driver addressed by the address on the row address line ROW_ADDR<9,0>.

[0150] Four of the 21 lines are transmit set lines TX_SET<4,1>, and the other four lines are transmit clear lines TX_CLR<4,1>. Each transmit set line TX_SET<4,1> is connected via a row driver to one of the first, second, third, and fourth transmit lines, which are connected to the addressed row driver.

[0151] The active signal of the transmit set line TX_SET(1) causes the row driver addressed by the address of the row address line ROW_ADDR<9,0> to drive an active signal on the first transmit line, and similarly drives the other transmit set lines. The active signal of the transmit clear line TX_CLR(1) causes the row driver addressed by the address of the row address line ROW_ADDR<9,0> to drive an inactive signal on the first transmit line, and similarly drives the other transmit lines.

[0152] In the image sensor 821C, four transmission gate steps are required to binn the alternative light-filtered pixels 2x2 in the charge region (providing a 4x signal without adding noise) and to sample all visible light color-filtered pixels to full resolution (unbinned). As shown in Figure 8C, to achieve this, it is necessary to perform overlapping row lines for each transmit gate through the pixel array. To make the overlapping row lines easier to distinguish, in Figure 8C, the row is labeled TXA_<row#> and TXB_<row#> The label is assigned. As mentioned above, stage A (label TXA_<row#> (shown by) proceeds to every four pixels in the row starting from the first pixel, and stage B (label TXB_<row#> The sequence (indicated by ) progresses to every fourth pixel in a row, starting from the second pixel, and so on. In the four-way shared pixel cell in Figure 8A, there are only two stages, and the row lines in each stage are connected to alternate pixels.

[0153] When alternative light-filtered pixel cells are interleaved diagonally within a visible light color-filtered cell, instead of being arranged as two sets of color pixels, two sets of hyperspectral pixels, two sets of color pixels, etc., as shown in Figure 8C, the pulse sequence when binned differs for odd-numbered and even-numbered pairs, as shown in Figure 8B. Thus, different timing diagrams are presented in Figures 9C to 9F, one for the case without binning (Figures 9C and 9E) and one for the case where the color information is at full resolution but the hyperspectral is binned 2x2 (Figures 9D and 9F). Figures 9C and 9D show the timing sequences for even-numbered pairs (0 / 1, 4 / 5, 8 / 9, ...), while Figures 9E and 9F show the timing sequences for odd-numbered pairs (2 / 3, 6 / 7, 10 / 11, ...). When binning is not performed (Figures 9C and 9E), the pulse timing is the same for even-numbered and odd-numbered row pairs. However, when binning is performed (Figures 9D and 9F), the pulse timing differs between even-numbered and odd-numbered row pairs.

[0154] When binning is enabled, all four alternative light-filtered pixels within a four-way shared cell are simultaneously connected to the shared column line, and the color pixels are read at full resolution based on the timing diagrams in Figures 9D and 9F. When binning is disabled, each pixel is read individually.

[0155] Figures 10 and 11 show several combinations that can be obtained using the stereoscopic image acquisition device of Figure 3, which includes a dual frame timer logic circuit, and the various timing sequences described above. As described above, the stereoscopic image acquisition device includes two image sensors, each of which captures a frame, and each frame contains a scene.

[0156] Firstly, a typical stereoscopic scene 1001 is obtained when each frame timer performs rolling shutter with the same exposure time for each row of image sensors. Alternatively, the left and right scenes 1002 may have different exposure times. In this embodiment, as shown in Figure 6, one frame timer performs rolling shutter with a first exposure time, and the other frame timer performs rolling shutter with a second, different exposure time.

[0157] In multi-pixel binning, one of the two scenes generated is a monochrome scene 1003. A single frame timer for the image sensor, which includes a Bayer color filter array, uses a rolling shutter to output a single pixel for each position in the row of the image sensor. Each position is a row containing multiple Bayer pixels. See, for example, Figures 8A and 9A.

[0158] By combining different exposure times and multiple pixel binning methods, it is possible to generate scenes with different exposure times, one of which is the monochrome scene 1104.

[0159] In the example in Figure 10, a stereoscopic image acquisition device was used. However, as shown in Figure 11, various combinations of the frame timer timing sequences described above can also be implemented using the image acquisition device in Figure 7, which has a single frame timer logic circuit and a single image sensor. Firstly, a normal scene 1101 is obtained when the frame timer performs rolling shutter with the same exposure time for each row of the image sensor.

[0160] In multi-pixel binning, the resulting scene is a monochrome scene 1102. The frame timer of the image sensor, which includes a Bayer color filter array, uses a rolling shutter to output a single pixel for each position in the row of the image sensor. Each position is a row containing multiple Bayer pixels. See, for example, Figures 8A and 9A.

[0161] In this specification, a computer program product includes a medium configured to store computer-readable code required for any one or any combination of the methods described herein, or a medium for storing computer-readable code for any one or any combination of the methods. Examples of computer program products include CD-ROM discs, DVD discs, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and signals transmitted over a network representing computer-readable program code. A tangible, non-temporary computer program product includes a medium configured to store computer-readable instructions for any one or any combination of the methods described herein, or a medium for storing computer-readable instructions for any one or any combination of the methods. Examples of tangible, non-temporary computer program products include CD-ROM discs, DVD discs, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, and other physical storage media.

[0162] In light of this disclosure, the instructions used in any one or any combination of the methods described herein can be implemented in a wide variety of computer system configurations using the operating system and computer programming language of interest to the user.

[0163] As used herein, “first,” “second,” “third,” etc., are adjectives used to distinguish different components or elements. Thus, “first,” “second,” and “third” are not intended to mean the order of components or elements, or the total number of components or elements.

[0164] The above description and accompanying drawings illustrating aspects and embodiments of the present invention should not be construed as limiting, and the claims define the invention to be protected. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and claims. In some cases, well-known circuits, structures, and techniques are not described or elaborated upon in order to avoid obscuring the present invention.

[0165] Furthermore, the terminology used in this specification is not intended to limit the invention. For example, Spatially relative terms such as "beneath," "below," "lower," "above," "upper," "proximal," and "distal" can be used to describe the relationship of one element or feature to another, as shown in the figure. These spatially relative terms are intended to encompass different positions (i.e., arrangements) and orientations (i.e., rotational arrangements) of the device in use or operation, in addition to the positions and orientations shown in the figure. For example, if the device in the figure is turned upside down, an element described as "below" or "below" another element or feature becomes "above" or "on top" of the other element or feature. Thus, the exemplary term "below" can encompass both upper and lower positions and orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein may be interpreted accordingly. Similarly, descriptions of movement along and around various axes include various special positions and orientations of the device.

[0166] The singular forms “a, an” and “the” are intended to include the plural unless the context indicates otherwise. Terms such as “comprises, comprising” and “includes” specify the presence of the described features, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups. Components described as being coupled may be coupled directly electrically or mechanically, or indirectly through one or more intermediate components.

[0167] All examples and illustrative references are non-limiting and should not be used to limit the claims to any specific embodiments and representations described herein or their equivalents. Headings are for formatting purposes only and should not be used to limit subject matter, as text under one heading may cross-reference or apply to text under one or more headings. Finally, in consideration of this disclosure, certain features described in relation to one aspect or embodiment may apply to other disclosed aspects.

[0168] The following is an example of the claims as originally filed. [Example 1] An image acquisition device, said image acquisition device, A first image sensor including a first set of multiple pixel rows, A second image sensor containing a second set of pixel rows, A first frame timer coupled to the first image sensor and providing an image acquisition timing signal to the first image sensor, The system includes a second frame timer coupled to the second image sensor and providing an image acquisition timing signal to the second image sensor, The first and second frame timers are different frame timers. Image acquisition device. [Example 2] The first frame timer is configured to provide image acquisition timing signals for sequentially acquiring N frames in the first image sensor. The image acquisition apparatus according to Embodiment 1, wherein the second frame timer is configured to provide an image acquisition timing signal for acquiring one frame with the second image sensor for every N frames sequentially acquired with the first image sensor. [Example 3] The first frame timer is configured to expose each of the first plurality of pixel rows over a first exposure time, The second frame timer is configured to expose each row of the second plurality of pixel rows over a second exposure time, The image acquisition apparatus according to Embodiment 1, wherein the first exposure time is different from the second exposure time. [Example 4] The image acquisition apparatus according to Embodiment 1, wherein the first image sensor includes a Bayer color filter array, each position of the first plurality of pixel rows of the first image sensor includes a set of Bayer pixels, and the first frame timer is configured to combine each set of Bayer pixels into a row to form a single output pixel. [Example 5] The first frame timer is configured to expose each of the first plurality of pixel rows over a first exposure time, The second frame timer is configured to expose each row of the second plurality of pixel rows over a second exposure time, The image acquisition apparatus according to Embodiment 4, wherein the first exposure time is different from the second exposure time. [Example 6] The first plurality of pixel rows include a plurality of pixel cells, each of which contains a plurality of pixels, and the first image sensor is A visible light color filter array containing multiple different individual visible light color filters, The present invention further includes an alternative light filter array comprising a plurality of individual alternative light filters, wherein one of the plurality of individual alternative light filters covers both a first set of pixels in a first pixel cell of the plurality of pixel cells and a second set of pixels in a second pixel cell of the plurality of pixel cells, the first pixel cell being adjacent to the second pixel cell. The image acquisition apparatus according to Embodiment 1, wherein each of the plurality of different individual visible light color filters covers a different pixel in the first and second sets of pixels, and the pixels covered by the individual visible light color filters of the plurality of different individual color filters are different from the pixels covered by the individual alternative light filters. [Example 7] The image acquisition apparatus according to Embodiment 6, wherein the first frame timer is configured to simultaneously reset pixels in the first and second pixel cells, which are covered by one of the different individual visible light color filters. [Example 8] The image acquisition apparatus according to Embodiment 6, wherein the first frame timer is configured to simultaneously read a first pixel of the first pixel cell covered by one of the plurality of different individual visible light color filters and a second pixel of the second pixel cell covered by one of the plurality of different individual visible light color filters. [Example 9] The image acquisition device according to Embodiment 6, wherein the first frame timer is configured to simultaneously read a first pixel in the set of first pixels in the first pixel cell among the plurality of pixel cells, and a second pixel in the set of second pixels in the second pixel cell among the plurality of pixels. [Example 10] The image acquisition device according to Embodiment 9, wherein the image acquisition device is configured to binn the first pixel and the second pixel that have been read. [Example 11] The image acquisition apparatus according to Embodiment 1, wherein the first image sensor further comprises a plurality of alternative light-filtered pixel cells and a plurality of interleaved visible light color-filtered cells. [Example 12] An image acquisition device, said image acquisition device, An image sensor comprising multiple pixel rows and a visible light color filter array, The visible light color filter array includes a plurality of different individual visible light color filters, The aforementioned plurality of pixel rows contain a plurality of pixel cells, and each of the plurality of pixel cells contains a plurality of pixels, An image sensor in which each of the plurality of pixels of a pixel cell is covered by a different color filter from the plurality of different individual visible light color filters, A frame timer coupled to the image sensor and providing an image acquisition timing signal to the image sensor, the frame timer being configured to combine the plurality of pixels of a pixel cell to form a single output pixel, Image acquisition device. [Example 13] An image acquisition device, said image acquisition device, The image sensor includes multiple pixel rows, a visible light color filter array, and an alternative light filter array. Each of the aforementioned multiple pixel rows contains multiple pixel cells, and each of these multiple pixel cells contains multiple pixels. The visible light color filter array includes a plurality of different individual visible light color filters, The alternative light filter array includes a plurality of individual alternative light filters, one of the plurality of individual alternative light filters covering both a first set of pixels in a first pixel cell of the plurality of pixel cells and a second set of pixels in a second pixel cell of the plurality of pixel cells, the first pixel cell being adjacent to the second pixel cell. Each of the plurality of distinct visible light color filters covers a different pixel in the first and second sets of pixels, and the pixels covered by each of the plurality of distinct visible light color filters are different from the pixels covered by the individual alternative light filters. Image acquisition device. [Example 14] The image acquisition apparatus according to Embodiment 13, further comprising a frame timer coupled to the image sensor and providing an image acquisition timing signal to the image sensor. [Example 15] The image acquisition apparatus according to Embodiment 14, wherein the frame timer is configured to simultaneously reset pixels in the first and second pixel cells, which are covered by one of the plurality of different individual visible light color filters. [Example 16] The image acquisition apparatus according to Embodiment 14, wherein the frame timer is configured to simultaneously read a first pixel of the first pixel cell covered by one of the plurality of different individual visible light color filters and a second pixel of the second pixel cell covered by one of the plurality of different individual visible light color filters. [Example 17] The image acquisition device according to Embodiment 14, wherein the frame timer is configured to simultaneously read a first pixel in the set of first pixels in the first pixel cell among the plurality of pixel cells, and a second pixel in the set of second pixels in the second pixel cell among the plurality of pixel cells. [Example 18] The image acquisition device according to Embodiment 17, wherein the image acquisition device is configured to binn the first pixel and the second pixel that have been read. [Example 19] A method, and said method is A step of using a signal from a first frame timer to expose each row of a first plurality of pixel rows of the first image sensor of the stereoscopic image acquisition device for a first exposure time, The process includes the step of using a signal from a second frame timer to expose each row of a second plurality of pixel rows of the second image sensor of the stereoscopic image acquisition device for a second exposure time, The first exposure time is different from the second exposure time. method. [Example 20] A method, and said method is The process includes a step of outputting a single output pixel from a position in an image sensor containing multiple pixels, wherein the output step involves a frame timer using a signal to combine the multiple pixels at the position to form the single output pixel. method.

Claims

[Claim 1] An image acquisition device, said image acquisition device, An image sensor including a first pixel cell and a second pixel cell adjacent to the first pixel cell, A plurality of separate alternative light filters configured to filter invisible light, wherein one of the plurality of separate alternative light filters covers both a first set of pixels in the first pixel cell and a second set of pixels in the second pixel cell, the second set of pixels is adjacent to the first set of pixels, and the first and second sets of pixels each contain at least two pixels. A plurality of individual visible light color filters covering the pixels of the first and second pixel cells, wherein the pixels of the first and second pixel cells covered by the individual visible light color filters are different from the set of pixels of the first and second pixels covered by the individual alternative light filters, comprising: Image acquisition device.

Images