Hybrid image sensor with multi-mode shutter
By designing a hybrid image sensor with multi-mode shutter, combined with roller shutter and global shutter functions, the problem of image sensor distortion and high cost in virtual reality, augmented reality and mixed reality applications is solved, achieving high-quality image acquisition and Accuracy of computer vision functions.
Patent Information
- Application Number
- CN202410701190.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing image sensors are difficult to provide undistorted images in virtual reality, augmented reality and mixed reality applications. The image distortion introduced by the roller shutter affects the accuracy of computer vision functions, and the global shutter costs are high and complex, increasing equipment costs and complexity. sex.
Design a hybrid image sensor with multi-mode shutter, combining roller shutter and global shutter functions, and realizes flexible resolution and shutter mode switching of image sensors through pixel array and related dual sampling components, supports roller shutter mode and global shutter mode, suitable for For different application requirements.
It realizes the provision of high-quality images in virtual reality, augmented reality and mixed reality applications, avoids image distortion, reduces equipment costs and complexity, and meets the accuracy requirements of computer vision functions.
Smart Images

Figure CN119071654A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to and the benefit of U.S. non-provisional patent application No. 18 / 328,344, filed on June 2, 2023, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present application relates generally to image sensors and, more particularly, to a hybrid image sensor with a multi-mode shutter. Background Art
[0004] A typical image sensor includes an array of pixel units. Each pixel unit may include a photodiode for sensing light by converting photons into electric charges (e.g., electrons or holes). The charges generated by the photodiode array may then be quantized into digital values by an analog-to-digital converter (ADC) to generate a digital image. The digital image may be exported from the sensor to another system (e.g., a viewing system for viewing the digital image, a processing system for interpreting the digital image, a compilation system for compiling a set of digital images, etc.). Summary of the invention
[0005] Various examples of hybrid image sensors with multi-mode shutters are described. In one example, the hybrid image sensor with multi-mode shutter includes a plurality of pixel arrays, each pixel array including a plurality of pixels, each pixel including a light sensing element configured to generate and store charge in response to incident light; each pixel array including: a charge storage device configured to receive charge from each of the plurality of light sensing elements of the pixel array; a first plurality of switches, each switch of the first plurality of switches connected between a corresponding pixel of the pixel array and the charge storage device; a second plurality of switches, the second plurality of switches including a high-resolution selection switch and a low-resolution selection switch, each of the high-resolution selection switch and the low-resolution selection switch connected in parallel to an output of the charge storage device; and a plurality of pixel output lines, each pixel output line configured to output a signal representing a pixel value corresponding to one or more pixel arrays coupled to the corresponding pixel output line.
[0006] An example method for capturing an image using a hybrid image sensor having a multi-mode shutter includes: enabling a global shutter mode in an image sensor having at least a rolling shutter mode and a global shutter mode, the image sensor having a plurality of pixel arrays, each pixel array including a plurality of light sensing elements and a charge storage device, the charge storage device being configured to receive charge from each of the plurality of light sensing elements of the pixel array, the plurality of light sensing elements being selectively connected to the charge storage device; resetting each charge storage device of the image sensor to establish a reset voltage; for each pixel array, transmitting the reset voltage to a corresponding correlated double sampling (“CDS”) component; accumulating charge in each of the light sensing elements of the plurality of pixel arrays during an integration period; for each pixel array, transmitting the accumulated charge from each light sensing element to a corresponding charge storage device for storage as a signal voltage; for each pixel array, transmitting the signal voltage from the corresponding charge storage device to a corresponding CDS component; and outputting the reset voltage and the signal voltage from the corresponding CDS component.
[0007] These illustrative examples are not mentioned to limit or define the scope of the present disclosure, but to provide examples to help understand the present disclosure. Each illustrative example is discussed in the detailed description, and the detailed description provides further description. By consulting this specification, the advantages provided by various examples can be further understood. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more specific examples and, together with the description of the examples, serve to explain principles and implementations of specific examples.
[0009] Figure 1A and Figure 1B is a schematic diagram of an embodiment of a near-eye display.
[0010] Figure 2 is an embodiment of a cross-section of a near-eye display.
[0011] Figure 3 An isometric view of an embodiment of a waveguide display having a single source assembly is shown.
[0012] Figure 4 A cross section of an embodiment of a waveguide display is shown.
[0013] Figure 5 is a block diagram of an embodiment of a system including a near-eye display.
[0014] Figure 6 An example of an imaging system including a hybrid image sensor with a multi-mode shutter is shown.
[0015] Figure 7 An example of a pixel array of a hybrid image sensor with a multi-mode shutter is shown.
[0016] Fig. 8A and Figure 8B An example pixel array and an example correlated double sampling component of a hybrid image sensor with a multi-mode shutter are shown.
[0017] Fig. 9A A portion of an example hybrid image sensor with a multi-mode shutter is shown.
[0018] Fig. 9B The layout of an example hybrid image sensor with a multi-mode shutter is shown.
[0019] Fig. 9C and Fig.9D An example timing diagram for an example hybrid image sensor with a multi-mode shutter is shown.
[0020] Fig. 10A and Fig. 10B An example pixel array and an example correlated double sampling component of a hybrid image sensor with a multi-mode shutter are shown.
[0021] Fig.11 A portion of an example hybrid image sensor with a multi-mode shutter is shown.
[0022] Fig. 12A and Fig. 12B An example pixel array and an example pixel array-level ADC of a hybrid image sensor with a multi-mode shutter are shown.
[0023] Fig.13 A portion of an example hybrid image sensor with a multi-mode shutter is shown.
[0024] Fig.14A and Fig. 14B An example pixel array and an example correlated double sampling component of a hybrid image sensor with a multi-mode shutter are shown.
[0025] Fig.15 A portion of an example hybrid image sensor with a multi-mode shutter is shown.
[0026] Figures 16 to 20 An example method for image acquisition using a hybrid image sensor with a multi-mode shutter is shown. DETAILED DESCRIPTION
[0027] Examples are described herein in the context of a hybrid image sensor with a multi-mode shutter. One of ordinary skill in the art will appreciate that the following description is illustrative only and is not intended to be limiting in any way. Reference will now be made in detail to the implementation of the examples shown in the accompanying drawings. The same reference numerals will be used throughout the drawings and the following description to refer to the same or similar items.
[0028] For the sake of clarity, not all conventional features of the examples described herein are shown and described. Of course, it should be understood that in developing any such actual implementation, many implementation-specific decisions must be made (e.g., complying with application-related and business-related constraints) in order to achieve the developer's specific goals, and these specific goals will vary from implementation to implementation and developer to developer.
[0029] To capture an image, an image sensor uses a pixel array including light-sensitive elements (e.g., photodiodes) to capture incident photons and convert these incident photons into electric charge during an integration period. The charge can be stored in the light-sensitive element itself, or it can be transferred to another charge storage device (e.g., a floating diffusion). At the end of the integration period, the accumulated charge is converted into a digital value, for example, by first converting the charge into a voltage and then comparing a ramp voltage signal to the converted voltage using an analog-to-digital converter ("ADC") (e.g., a comparator). The digital value can then be used as the pixel value of the pixel.
[0030] The process of acquiring different pixel values involves a "shutter" mechanism, which is a functional analog to the mechanical shutter in a traditional film camera. The shutter in an image sensor involves the orchestration of the accumulation of electrical charge using light sensing elements and the output of corresponding pixel values within the sensor to allow a single image to be generated from those pixel values.
[0031] Two common types of shutters in image sensors are rolling shutters and global shutters. Image sensors that use rolling shutters collect pixel values one row at a time, for example by arranging an ADC to receive pixel values for a specific column of pixels in a pixel array (called a column ADC). The pixels in a row can then be integrated and then read out by closing a switch to connect these pixels to the corresponding column readout line, which connects the pixels to the column ADC. The column ADC generates the pixel values and stores them in memory, and then the next row of pixels is integrated and read out using the same process. Therefore, the image acquisition process is carried out over a period of time required to continuously integrate and read out pixel values row by row. In contrast, an image sensor with a global shutter can integrate all pixels at the same time, which can then be processed by the ADC to generate an image.
[0032] Different types of shutters have different advantages and disadvantages, and choosing the right shutter mechanism for an image sensor requires a trade-off. For example, a rolling shutter may introduce distortion in the captured image because the pixels of each consecutive row will be captured slightly offset in time from the pixels of the previous row. Therefore, for an image sensor with a large number of rows, the cumulative delay over all rows may affect the appearance of moving objects within the image because the lower part of the object will move farther than the upper part of the object when captured, thereby distorting its appearance. And although a global shutter can be used to avoid this distortion, a global shutter tends to be larger and more expensive because more circuit components must be integrated into the image sensor. However, typically, both types of sensors may be desired in virtual reality ("VR") applications, augmented reality ("AR") applications, or mixed reality ("MR") applications.
[0033] In VR / AR / MR applications, a global shutter may be required to provide undistorted images to computer vision ("CV") functions to allow high-quality VR / AR / MR experiences, such as object recognition and tracking, simultaneous localization and mapping ("SLAM") functions, etc. The image distortion introduced by the rolling shutter may affect the ability of the CV application to provide accurate or reliable output. However, rolling shutter image sensors may be desirable for providing video to users because users are less affected by such image distortion and because such image sensors are less expensive.
[0034] However, using multiple image sensors to provide CV and user video may increase the overall cost and complexity of the VR / AR / MR device: multiple global shutter sensors may be used to capture images of different fields of view (“FOV”) for CV processing, while multiple rolling shutter image sensors may be provided to provide the user with an increased FOV or stereoscopic imaging. In addition, since the CV function will generally affect the user's perception of the VR / AR / MR environment, the use of multiple image sensors for CV and for user video means that the image sensor providing the CV image will be physically offset from the image sensor providing the video to the user. This offset may affect the appearance of any virtual objects or effects generated in the user's display based on the CV function. Alternatively, additional computational complexity may be introduced to compensate for the offset between the various image sensors.
[0035] To help address these and other issues with image sensors, an example hybrid image sensor with a multi-mode shutter is configured with both global shutter functionality and local shutter functionality. The example image sensor includes a pixel array (e.g., an M×N pixel array), wherein each pixel includes a photosensitive element, and the pixels are arranged into multiple 2×2 pixel arrays (although arrays of any size may be used). Each pixel array includes a common charge storage device connected to the input of a source follower. In addition, each array has a corresponding correlated double sampling (CDS) component that includes two charge storage devices, one for storing a reset voltage for the pixel array (or multiple pixel arrays arranged to form a pixel cluster and sharing a common CDS component), and the other for storing an integrated signal voltage for the pixel array. The two charge storage devices are configured as inputs to corresponding source followers, and the outputs of each source follower are connected to column lines corresponding to the pixel array.
[0036] The source follower of the pixel array is used to output a voltage based on the charge stored in the charge storage device of the pixel array. The output of the source follower is provided to three parallel switches. The first switch (rolling-shutter select ("RSSEL") switch) connects the source follower output to the column line corresponding to the pixel array. The second switch (global shutter reset ("GSR") switch) connects the source follower output to the reset charge storage device in the CDS component. The third switch (global shutter signal ("GSS") switch) connects the source follower output to the pixel value charge storage device in the CDS component.
[0037] In operation, for a particular image, the image sensor is configured in either rolling shutter mode or global shutter mode. In the rolling shutter configuration, each photodiode in the pixel array is connected in turn to the charge storage device of the pixel array, and the RSSEL switch is used to read out the voltage to the column ADC. Therefore, for each row in the pixel array, two photodiodes are read in turn, and the column ADC converts the voltage to a pixel value in turn. Each row of the pixel array is then integrated and read out continuously. This provides a high-resolution image in which each photodiode provides a discrete pixel value, but the output image may include rolling shutter artifacts.
[0038] In global shutter mode, all four photodiodes are connected to the charge storage device at the same time, so that each pixel array has a single voltage. In CDS operation, the reset value is first collected at the CDS component by resetting the charge storage device and making the GSR line valid (asserting). Then the four photodiodes are connected to the charge storage device and a composite voltage is generated. Then the GSS line is made valid to transfer the charge to the CDS component. Since all pixels are integrated at the same time, a global shutter is achieved. The readout of the reset value and pixel value stored in the CDS component can be performed row by row using the same column ADC as described above. This mode provides a low-resolution image in which each pixel array provides a combined pixel value for all photodiodes in the cluster, but the output image has no rolling shutter artifacts. Therefore, the same image sensor can collect images suitable for presentation to the user and images suitable for performing various CV processes that can be used in the system. And although this example connects a single pixel array to the CDS component, some examples discussed below can associate multiple pixel arrays to a single CDS component.
[0039] This illustrative example is given to introduce the reader to the general subject matter discussed herein, and the present disclosure is not limited to this example.The following sections describe various additional non-limiting examples and examples of hybrid image sensors with multi-mode shutters.
[0040] Figure 1A is a schematic diagram of an embodiment of a near-eye display 100. The near-eye display 100 presents media to a user. Examples of media presented by the near-eye display 100 include one or more images, videos, and / or audio. In some embodiments, audio is presented via an external device (e.g., a speaker and / or headphones) that receives audio information from the near-eye display 100, a console, or both, and presents audio data based on the audio information. The near-eye display 100 is typically configured to operate as a virtual reality (VR) display. In some embodiments, the near-eye display 100 is modified to operate as an augmented reality (AR) display and / or a mixed reality (MR) display.
[0041] The near-eye display 100 includes a frame 105 and a display 110. The frame 105 is coupled to one or more optical elements. The display 110 is configured for a user to view content presented by the near-eye display 100. In some embodiments, the display 110 includes a waveguide display component for guiding light from one or more images to the user's eyes.
[0042] The near-eye display 100 also includes image sensors 120a, 120b, 120c, and 120d. Each of the image sensors 120a, 120b, 120c, and 120d may include a pixel array configured to generate image data representing different fields of view along different directions. For example, sensors 120a and 120b may be configured to provide image data representing two fields of view toward a direction A along the Z axis, while sensor 120c may be configured to provide image data representing a field of view toward a direction B along the X axis, and sensor 120d may be configured to provide image data representing a field of view toward a direction C along the X axis.
[0043] In some embodiments, the sensors 120a to 120d may be configured as input devices that control or affect the display content of the near-eye display 100 to provide an interactive VR / AR / MR experience to a user wearing the near-eye display 100. For example, the sensors 120a to 120d may generate physical image data of the physical environment in which the user is located. The physical image data may be provided to a position tracking system to track the user's position and / or movement path in the physical environment. The system may then update the image data provided to the display 110, for example, based on the user's position and orientation, to provide an interactive experience. In some embodiments, the position tracking system may run a SLAM algorithm to track a group of objects in the physical environment and within the user's field of view as the user moves within the physical environment. The position tracking system may build and update a map of the physical environment based on the group of objects and track the user's position within the map. The sensors 120a to 120d may provide a more comprehensive view of the physical environment to the position tracking system by providing image data corresponding to multiple fields of view, which may include more objects in the construction and update of the map. With such an arrangement, the accuracy and robustness of tracking the user's position within the physical environment may be improved.
[0044] In some embodiments, the near-eye display 100 may also include one or more active illuminators 130 for projecting light into a physical environment. The projected light may be associated with different spectra (e.g., visible light, infrared light, ultraviolet light) and may be used for various purposes. For example, the illuminator 130 may project light in a dark environment (or in an environment with lower intensity infrared light, ultraviolet light, etc.) to assist the sensors 120a to 120d in capturing images of different objects within the dark environment, thereby, for example, enabling position tracking of a user. The illuminator 130 may project certain markers onto multiple objects within the environment to assist the position tracking system in identifying these objects for map building / updating.
[0045] In some embodiments, the illuminator 130 can also implement stereoscopic imaging. For example, one or both of the sensors 120a or 120b can include both a first pixel array for visible light sensing and a second pixel array for infrared (IR) light sensing. The first pixel array can be covered with a color filter (e.g., a Bayer filter), wherein each pixel of the first pixel array is configured to measure the intensity of light associated with a specific color (e.g., one of red, green, or blue). The second pixel array (for IR light sensing) can also be covered with a filter that only allows IR light to pass, wherein each pixel of the second pixel array is configured to measure the intensity of IR light. These pixel arrays can generate an RGB image and an IR image of an object, wherein each pixel in the IR image is mapped to each pixel in the RGB image. The illuminator 130 can project a set of IR markers onto an object, and the image of the object can be collected by the IR pixel array. The system can estimate the distance of different parts of the object from the IR pixel array based on the distribution of the IR markers of the object as shown in the image, and generate a stereoscopic image of the object based on the distance. For example, the system may determine the relative position of the object with respect to the user based on the stereoscopic image of the object, and may update the image data provided to the display 110 based on the relative position information, thereby providing an interactive experience.
[0046] As described above, the near-eye display 100 can operate in environments associated with a very wide range of light intensities. For example, the near-eye display 100 can operate in an indoor environment or an outdoor environment, and / or at different times of the day. The near-eye display 100 can also operate with the active illuminator 130 turned on or not turned on. Therefore, the image sensors 120a to 120d may need to have a wide dynamic range to be able to operate correctly (e.g., to generate output related to the intensity of the incident light) across a very wide range of light intensities associated with the different operating environments of the near-eye display 100.
[0047] Figure 1B is a schematic diagram of another embodiment of a near-eye display 100 . Figure 1B 1 shows a side of the near-eye display 100 facing one or more eyeballs 135 of a user wearing the near-eye display 100. Figure 1B As shown, the near-eye display 100 may further include a plurality of illuminators 140a, 140b, 140c, 140d, 140e, and 140f. The near-eye display 100 further includes a plurality of image sensors 150a and 150b. The illuminators 140a, 140b, and 140c may face a direction D (which is the same as the direction of the image sensor 150a). Figure 1AThe sensor 150a may include a pixel array for receiving the reflected light and generating an image of the reflected pattern. Similarly, illuminators 140d, 140e, and 140f may emit NIR light with a pattern. The NIR light may be reflected by the user's right eyeball and may be received by sensor 150b. Sensor 150b may also include a pixel array for generating an image of the reflected pattern. The system may determine the user's gaze point based on the images of the reflected patterns from sensors 150a and 150b, and update the image data provided to display 110 based on the determined gaze point, thereby providing an interactive experience to the user.
[0048] As described above, in order to avoid damaging the user's eyes, the illuminators 140a, 140b, 140c, 140d, 140e, and 140f are generally configured to output very low intensity light. Figure 1A In the case of image sensors 120a to 120d in the same sensor device, when the intensity of the incident light is very low, the image sensors 120a to 120d may need to be able to generate outputs related to the intensity of the incident light, which may further increase the dynamic range requirements of these image sensors.
[0049] In addition, image sensors 120a to 120d may need to be able to generate outputs at high speed to track the movement of the eyeball. For example, the user's eyeball can perform very fast movements (e.g., saccadic movements), in which there may be a rapid jump from one eyeball position to another eyeball position. In order to track the rapid movement of the user's eyeball, image sensors 120a to 120d need to generate images of the eyeball at high speed. For example, the rate (frame rate) at which the image sensor generates image frames needs to at least match the speed of the eyeball movement. A higher frame rate requires a short total exposure time for all pixel units involved in generating the image frame, and requires that the sensor output be converted to a digital value at high speed for image generation. In addition, as described above, the image sensor also needs to be able to operate in an environment with lower light intensity.
[0050] Figure 2 yes Figure 1A and Figure 1B 1 shows an embodiment of a cross-section 200 of a near-eye display 100 shown in FIG. The display 110 includes at least one waveguide display assembly 210. The exit pupil 230 is the location where a single eyeball 220 of a user is positioned in an eyebox region when the user is wearing the near-eye display 100. For illustration purposes, Figure 2A cross section 200 is shown associated with an eyeball 220 and a single waveguide display assembly 210, while for the user's second eye, a second waveguide display is used.
[0051] The waveguide display assembly 210 is configured to guide image light to an eyebox located at an exit pupil 230, and to an eyeball 220. The waveguide display assembly 210 may be constructed of one or more materials (e.g., plastic, glass) having one or more refractive indices. In some embodiments, the near-eye display 100 includes one or more optical elements between the waveguide display assembly 210 and the eyeball 220.
[0052] In some embodiments, the waveguide display assembly 210 includes one or more waveguide displays that are stacked, and the one or more waveguide displays that are stacked include, but are not limited to, a stacked waveguide display, a variable focus waveguide display, and the like. A stacked waveguide display is a multicolor display (e.g., a red-green-blue (RGB) display) formed by stacking a plurality of waveguide displays whose corresponding monochromatic light sources have different colors. A stacked waveguide display is also a multicolor display (e.g., a multi-plane color display) that can be projected onto multiple planes. In some configurations, a stacked waveguide display is a monochromatic display (e.g., a multi-plane monochromatic display) that can be projected onto multiple planes. A variable focus waveguide display is a display that can adjust the focus position of image light emitted from a waveguide display. In alternative embodiments, the waveguide display assembly 210 may include a stacked waveguide display and a variable focus waveguide display.
[0053] Figure 3 An isometric view of an embodiment of a waveguide display 300 is shown. In some embodiments, the waveguide display 300 is a component of the near-eye display 100 (e.g., the waveguide display assembly 210). In some embodiments, the waveguide display 300 is part of some other near-eye display or other system that directs image light to a specific location.
[0054] The waveguide display 300 includes a source component 310, an output waveguide 320, and a controller 330. For purposes of illustration, Figure 3 The waveguide display 300 is shown associated with a single eyeball 220, but in some embodiments, another waveguide display that is separate or partially separate from the waveguide display 300 provides image light to the user's other eye.
[0055] The source component 310 generates image light 355 and outputs the image light 355 to the coupling element 350 located on the first side 370-1 of the output waveguide 320. The output waveguide 320 is an optical waveguide that outputs the expanded image light 340 to the user's eyeball 220. The output waveguide 320 receives the image light 355 at one or more coupling elements 350 located on the first side 370-1, and guides the received input image light 355 to the guiding element 360. In some embodiments, the coupling element 350 couples the image light 355 from the source component 310 into the output waveguide 320. The coupling element 350 can be, for example, a diffraction grating, a holographic grating, one or more cascade reflectors, one or more prismatic surface elements, and / or a holographic reflector array.
[0056] The guiding element 360 redirects the received input image light 355 to the decoupling element 365, so that the received input image light 355 is decoupled out of the output waveguide 320 via the decoupling element 365. The guiding element 360 is part of the first side 370-1 of the output waveguide 320, or is fixed to the first side 370-1 of the output waveguide. The decoupling element 365 is part of the second side 370-2 of the output waveguide 320, or is fixed to the second side 370-2 of the output waveguide, so that the guiding element 360 is opposite to the decoupling element 365. The guiding element 360 and / or the decoupling element 365 can be, for example, a diffraction grating, a holographic grating, one or more cascade reflectors, one or more prismatic surface elements, and / or a holographic reflector array.
[0057] The second side 370-2 represents a plane along the x-dimension and the y-dimension. The output waveguide 320 can be made of one or more materials that promote total internal reflection of the image light 355. The output waveguide 320 can be made of silicon, plastic, glass and / or polymer, for example. The output waveguide 320 has a relatively small form factor. For example, the output waveguide 320 can be about 50 millimeters (mm) wide along the x-dimension, about 30 mm long along the y-dimension, and about 0.5 mm to 1 mm thick along the z-dimension.
[0058] The controller 330 controls the scanning operation of the source assembly 310. The controller 330 determines the scanning instructions for the source assembly 310. In some embodiments, the output waveguide 320 outputs the expanded image light 340 to the user's eyeball 220 with a larger field of view (FOV). For example, the expanded image light 340 is provided to the user's eyeball 220 with a diagonal FOV (along x and y) equal to and / or greater than 60 degrees, and / or equal to and / or less than 150 degrees. The output waveguide 320 is configured to provide an eyebox having a length equal to or greater than 20 mm and / or equal to or less than 50 mm, and / or a width equal to or greater than 10 mm and / or equal to or less than 50 mm.
[0059] In addition, the controller 330 also controls the image light 355 generated by the source component 310 based on the image data provided by the image sensor 370. The image sensor 370 can be located on the first side 370-1 and can include, for example, Figure 1A 370-1). The image sensors 120a to 120d can be operated to perform two-dimensional (2D) sensing and three-dimensional (3D) sensing of an object 372, for example, in front of the user (e.g., facing the first side 370-1). For 2D sensing, each pixel unit of the image sensors 120a to 120d can be operated to generate pixel data representing the intensity of light 374 generated by the light source 376 and reflected from the object 372. For 3D sensing, each pixel unit of the image sensors 120a to 120d can be operated to generate pixel data representing a time-of-flight measurement result of light 378 generated by the illuminator 325. For example, each pixel unit of the image sensors 120a to 120d can determine a first time when the illuminator 325 is able to project the light 378, and a second time when the pixel unit detects the light 378 reflected from the object 372. The difference between the first time and the second time may indicate the flight time of the light 378 between the image sensors 120a to 120d and the object 372, and the flight time information may be used to determine the distance between the image sensors 120a to 120d and the object 372. The image sensors 120a to 120d may be operated to perform 2D sensing and 3D sensing at different times and provide 2D image data and 3D image data to a remote control console 390, which may or may not be located within the waveguide display 300. The remote control console may combine the 2D image and the 3D image to, for example, generate a 3D model of the environment in which the user is located, track the position and / or orientation of the user, etc. The remote control console may determine the content of the image to be displayed to the user based on the information obtained from the 2D image and the 3D image. The remote control console may send instructions related to the determined content to the controller 330. The controller 330 may control the source component 310 to generate and output image light 355 based on the instructions to provide an interactive experience to the user.
[0060] Figure 4 An embodiment of a cross-section 400 of a waveguide display 300 is shown. The cross-section 400 includes a source component 310, an output waveguide 320, and an image sensor 370. Figure 4In the example of , the image sensor 370 may include a group of pixel cells 402 located on the first side 370-1, and the group of pixel cells is used to generate an image of the physical environment in front of the user. In some embodiments, there may be a mechanical shutter 404 and a filter array 406 inserted between the group of pixel cells 402 and the physical environment. The mechanical shutter 404 can control the exposure of the group of pixel cells 402. In some embodiments, as will be discussed below, the mechanical shutter 404 can be replaced by an electronic shutter. As will be discussed below, the filter array 406 can control the light wavelength range of the following light: the group of pixel cells 402 are exposed to this light. Each pixel cell in the plurality of pixel cells 402 can correspond to a pixel in the image. Although in Figure 4 It is not shown in the figure, but it should be understood that each pixel unit in the plurality of pixel units 402 may also be covered with a filter to control the optical wavelength range of the light to be sensed by these pixel units.
[0061] The mechanical shutter 404, after receiving an instruction from the remote control station, can be opened and expose the group of pixel units 402 during an exposure period. During the exposure period, the image sensor 370 can obtain light samples incident on the group of pixel units 402, and generate image data based on the intensity distribution of the incident light samples detected by the group of pixel units 402. The image sensor 370 can then provide the image data to the remote control station, which determines the display content and provides the display content information to the controller 330. The controller 330 can then determine the image light 355 based on the display content information.
[0062] Source assembly 310 generates image light 355 according to instructions from controller 330. Source assembly 310 includes source 410 and optical system 415. Source 410 is a light source that generates coherent light or partially coherent light. Source 410 can be, for example, a laser diode, a vertical cavity surface emitting laser, and / or a light emitting diode.
[0063] The optical system 415 includes one or more optical elements that condition the light from the source 410. Conditioning the light from the source 410 may include, for example, expanding, collimating, and / or adjusting the orientation according to instructions from the controller 330. The one or more optical elements may include one or more lenses, one or more liquid lenses, one or more mirrors, one or more apertures, and / or one or more gratings. In some embodiments, the optical system 415 includes a liquid lens with multiple electrodes that allows the light beam to be scanned at a threshold of a scanning angle to shift the light beam to an area outside the liquid lens. The light emitted from the optical system 415 (and also from the source assembly 310) is referred to as image light 355.
[0064] The output waveguide 320 receives the image light 355. The coupling element 350 couples the image light 355 from the source component 310 into the output waveguide 320. In embodiments where the coupling element 350 is a diffraction grating, the pitch of the diffraction grating is selected so that total internal reflection occurs in the output waveguide 320 and the image light 355 propagates internally in the output waveguide 320 toward the decoupling element 365 (e.g., by total internal reflection).
[0065] The guiding element 360 redirects the image light 355 toward the decoupling element 365 for decoupling from the output waveguide 320. In embodiments where the guiding element 360 is a diffraction grating, the pitch of the diffraction grating is selected so that the incident image light 355 exits the output waveguide 320 at one or more oblique angles relative to the surface of the decoupling element 365.
[0066] In some embodiments, the guiding element 360 and / or the decoupling element 365 are similar in structure. The expanded image light 340 exiting the output waveguide 320 is expanded along one or more dimensions (e.g., may be elongated along the x-dimension). In some embodiments, the waveguide display 300 includes a plurality of source components 310 and a plurality of output waveguides 320. Each source component in the plurality of source components 310 emits a monochromatic image light of a specific wavelength band corresponding to a primary color (e.g., red, green, or blue). The output waveguides in the plurality of output waveguides 320 may be stacked together at a spacing distance to output multi-color expanded image light 340.
[0067] Figure 5 is a block diagram of an embodiment of a system 500 including a near-eye display 100. The system 500 includes the near-eye display 100, an imaging device 535, an input / output interface 540, and image sensors 120a to 120d and image sensors 150a to 150b, all coupled to a control circuit 510. The system 500 can be configured as a head-mounted device, a mobile device, a wearable device, etc.
[0068] The near-eye display 100 is a display that presents media to the user. Examples of media presented by the near-eye display 100 include one or more images, videos, and / or audio. In some embodiments, the audio is presented via an external device (e.g., a speaker and / or headphones) that receives audio information from the near-eye display 100 and / or the control circuit 510 and presents audio data to the user based on the audio information. In some embodiments, the near-eye display 100 can also act as an AR glasses lens. In some embodiments, the near-eye display 100 uses computer-generated elements (e.g., images, videos, sounds) to enhance the view of the physical, real-world environment.
[0069] The near-eye display 100 includes a waveguide display assembly 210, one or more position sensors 525, and / or an inertial measurement unit (IMU) 530. The waveguide display assembly 210 includes a source assembly 310, an output waveguide 320, and a controller 330.
[0070] The IMU 530 is an electronic device that generates fast calibration data based on measurement signals received from one or more of the plurality of position sensors 525 , the fast calibration data indicating an estimated position of the near-eye display 100 relative to an initial position of the near-eye display 100 .
[0071] The imaging device 535 can generate image data for various applications. For example, the imaging device 535 can generate image data for providing slow calibration data based on the calibration parameters received from the control circuit 510. The imaging device 535 can include, for example Figure 1A The image sensors 120a to 120d in the image sensor 535 are used to generate image data of the physical environment of the user for performing position tracking of the user. The imaging device 535 may also include, for example Figure 1B The image sensors 150a and 150b in the embodiment are used to generate the following image data: the image data is used to determine the user's gaze point to identify the user's object of interest.
[0072] The input / output interface 540 is a device that allows a user to send an action request to the control circuit 510. An action request is a request to perform a specific action. For example, the action request may be to start or end an application, or to perform a specific action within an application.
[0073] The control circuit 510 provides media for presentation to the user to the near-eye display 100 based on information received from one or more of the following: the imaging device 535, the near-eye display 100, and the input / output interface 540. In some examples, the control circuit 510 can be housed within the system 500 configured as a head mounted device. In some examples, the control circuit 510 can be a stand-alone console device that is communicatively coupled to other components in the system 500. Figure 5 In the example shown, the control circuit 510 includes an application repository 545 , a tracking module 550 , and an engine 555 .
[0074] The application repository 545 stores one or more applications for execution by the control circuit 510. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. Examples of applications include: game applications, conference applications, video playback applications, or other suitable applications.
[0075] Tracking module 550 uses one or more calibration parameters to calibrate system 500 , and may adjust the one or more calibration parameters to reduce errors in determining the position of near-eye display 100 .
[0076] The tracking module 550 uses the slow calibration information from the imaging device 535 to track the movement of the near-eye display 100. The tracking module 550 also uses the position information from the fast calibration information to determine the position of a reference point of the near-eye display 100.
[0077] The engine 555 executes applications within the system 500 and receives position information, acceleration information, velocity information, and / or predicted future position of the near-eye display 100 from the tracking module 550. In some embodiments, the information received by the engine 555 can be used to generate a signal (e.g., a display instruction) for the waveguide display component 210 to determine the type of content presented to the user. For example, in order to provide an interactive experience, the engine 555 can determine the content to be presented to the user based on the following items: the user's position (e.g., provided by the tracking module 550), or the user's gaze point (e.g., based on image data provided by the imaging device 535), the distance between the object and the user (e.g., based on image data provided by the imaging device 535).
[0078] Figure 6 An example of an imaging system 600 that can perform image subsampling using a color grid array is shown. Figure 6 As shown, imaging system 600 includes image sensor 602 and main processor 604. Image sensor 602 includes controller 606 and pixel array 608. In some examples, controller 606 can be implemented as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a hardware processor that executes instructions to implement image subsampling with a color grid array. In addition, main processor 604 includes a general-purpose central processing unit (CPU) that can execute application 614.
[0079] Each pixel in the pixel array 608 receives incident light and converts it into an electric charge, which is stored as a voltage on a charge storage device. In addition, each pixel in the pixel array 608 can be individually addressed using row select lines and column select lines, which causes the corresponding row select switches and column select switches to close, thereby providing the voltage from the pixel to the ADC circuit, where the voltage is converted into a pixel value that can be read out (e.g., to the controller 606 or application 614).
[0080] In the pixel array 608, multiple pixels are grouped together to form superpixels that provide common ADC circuitry for the grouped pixels. For example, a superpixel may include four pixels arranged in a 2×2 grid. Thus, a 128×128 pixel array using this configuration will create a 64×64 superpixel array. In order to provide different color or frequency sensing, different pixels within a superpixel may be configured with different filters, such as to capture different visible color bands (e.g., red, green, blue, yellow, white), different spectral bands (e.g., near infrared (“IR”), monochrome, ultraviolet (“UV”), IR cutoff, IR bandpass), or the like. Thus, by enabling or disabling different pixels, each superpixel can provide any subset of such information. In addition, by sampling only certain superpixels, sparse image sensing can be used to capture only image information corresponding to a subset of multiple pixels in the pixel array 608.
[0081] Figure 7 An example of a pixel array 608 is shown. Figure 7 As shown, the pixel array 608 may include a column selection controller 704, a row selection controller 706, and a pixel selection controller 720. The column selection controller 704 is connected to column selection lines 708 (e.g., 708a, 708b, 708c, ..., 708n), and the row selection controller 706 is connected to row selection lines 710 (e.g., 710a, 710b, ... 710n). 00 , P 01 , P 0j ,……,P ij Each box represents a pixel. Each pixel is connected to one of the column selection lines 708, one of the row selection lines 710, and an output data bus ( Figure 7 Each pixel may be individually addressed by a column enable signal 732 provided by the column select controller 704 on the column select line 708, and a row enable signal 730 provided by the row select controller 706 on the row select line 710. The column enable signal 732 and the row enable signal 730 may be generated based on information received from the controller 606 or the host processor 604.
[0082] Fig. 8A and Figure 8B An example pixel array 800 and an example CDS component 840 suitable for use with an example hybrid image sensor having a multi-mode shutter are shown. Fig. 8A, the pixel array 800 includes four photodiodes 802a to 802d, each of which is connected to a charge storage device, such as a floating diffusion ("FD") region 804. Each of the photodiodes 802a to 802d can be selectively coupled to the FD region 804 through a corresponding transfer gate switch 810a to 810d. The FD region 804 provides an input voltage to a source follower ("SF") 806, the output of which is connected to three parallel output lines, which can be selectively activated by corresponding switches 820, 822a and 822b based on a rolling shutter select (RSSEL_x) signal, a global shutter reset (GSR_x,y) signal, or a global shutter switch (GSS_x,y) signal. "x" represents a particular row in the sensor array, and "y" represents an "odd" column or an "even" column of the pixel array in the pixel array cluster. As will be referenced Fig. 9A As discussed in more detail, multiple pixel arrays 800 may share a common CDS component 840. Thus, a "y" value designating "odd" or "even" indicates which "column" of the pixel array of a particular CDS cluster is activated, while an "x" value indicates the "odd" or "even" row to be activated with respect to a particular CDS cluster. And while "y" has only two different values in this example, in some examples, the "y" value may have more than two options.
[0083] The image sensor will typically include a plurality of pixel arrays 800 arranged in a two-dimensional grid to provide the desired image sensor resolution. By selectively activating different switches within the pixel array, the pixel array 800 can operate in a rolling shutter mode or a global shutter mode. In addition, the resolution of the pixel array 800 can be adjusted between full resolution and low resolution. Therefore, the pixel array 800 provides the image sensor with the flexibility to collect the desired resolution and use a shutter suitable for the application.
[0084] For example, to operate with a rolling shutter to obtain full resolution, each of the photodiodes 802a to 802d may be sequentially connected to the FD region 804 by asserting transfer gate (“TG”) signals TG_0,E, TG_0,O, TG_1,E, and TG_1,O (corresponding to row 0 and row 1 in the image sensor, and “even” and “odd” columns in the pixel array) in any suitable sequence of signals. When the photodiodes 802a to 802d are connected to the FD region 804, the RSSEL_0 (“0” represents the first row in the image sensor) signal may be asserted to close the corresponding switch 820 and output the rolling shutter output voltage RS_0 821. Fig. 9A As shown, the RS_0 signal is output to the corresponding column line, where it can be digitized and read out. Then, the RSSEL_0 signal is deasserted, and the FD region 804 is reset by making the RST_0 signal valid. The remaining photodiodes 802a to 802d can be read out in the same manner. Then, each successive row of the image sensor can be exposed and read out accordingly.
[0085] In addition, CDS operation can be provided in rolling shutter mode. For example, by making (for example) RSSEL_0 valid and transmitting the reset voltage to the column line, the reset voltage can be obtained after each photodiode is reset but before any transmission gate is closed. Subsequently, each transmission gate can be closed in turn to obtain the corresponding output voltage from each photodiode. The stored reset voltage can then be used to eliminate any thermal noise component of the output voltage from the photodiode.
[0086] In contrast, to operate in global shutter mode (in this example, with CDS), the FD region 804 is reset and the global shutter reset ("GSR") signal (of row 0 and even columns of the pixel array connected to the corresponding CDS component 840) is asserted to connect the output of the SF 806, thereby transmitting the reset voltage to the corresponding CDS component 840. The GSR signal is then deasserted, and after an integration period, all four photodiodes 802a to 802d can be connected to the FD region 804 by asserting all four TG signals to close the corresponding switches 810a to 810d. The global shutter signal ("GSS") signal (GSS_0,E) is then asserted to transmit the global shutter ("GS") voltage output by the SF 806 to the CDS component 840. And while this example is discussed with respect to the operation of a particular pixel array, it should be understood that these operations are performed simultaneously by all pixel arrays in the image sensor to provide a global shutter for the image sensor.
[0087] It should be understood that although Fig. 8A The example pixel array shown in FIG. 1 has a 2×2 photodiode array, but any suitable size of pixel array may be used. For example, according to different examples, a 2×4, 1×4, 1×2, etc. pixel array may be used.
[0088] refer to Figure 8B , showing the Fig. 8A800 in the example CDS component 840. In this example, the GR signal and the GS signal from the pixel array 800 are received at the corresponding reset capacitor (“CR”) and the signal capacitor (“CS”). Then, the reset voltage and the signal voltage can be read out successively by making the global shutter reset select signal (“GSSELR”) and the global shutter signal select signal (“GSSELS”) valid and invalid, and in this example, the outputs of the GSSELR and the GSSELS are connected to the corresponding column lines.
[0089] Reference now Fig. 9A and Fig. 9B , Fig. 9A A partial view of an example hybrid image sensor with a multi-mode shutter is shown. Fig. 9A The view shown in FIG. 8 includes a cluster 900 of four pixel arrays 800a to 800d. The image sensor includes Fig. 9B The two-dimensional layout 910 shown is a plurality of pixel arrays 800 arranged. The layout 910 includes a regular arrangement of clusters of pixel arrays, wherein a pixel array is identified in the upper left corner of the layout 910. Fig. 9A 9 and pixel array 800a in. As can be seen in layout 910, the clusters extend rightward and downward to any desired sensor size or resolution.
[0090] As mentioned above Fig. 8A and Figure 8B As discussed, pixel arrays 800a to 800d are coupled to column lines 920a and 920b, respectively, through RSSEL switches or via CDS components 840 to enable readout of pixel values. Fig. 9C In the illustrated rolling shutter operation, as described above, the first row of pixel arrays 800a and 800b are first integrated and read out to their corresponding column lines: each pixel in each pixel array is sequentially connected to the FD region 804, which is then connected to the corresponding column line via SF 806 and the rolling shutter switch by activating RSSEL. After reading out each pixel in the first row, each pixel in the second row in pixel arrays 800c and 800d can be integrated and read out in the same manner. The remaining rows of pixels in the pixel sensor can then be read out in the same manner to capture a full resolution image.
[0091] In contrast, Fig.9DThe global shutter operation timing is shown in . In the global shutter low-resolution mode, all four pixels in each pixel array in the cluster are connected to the corresponding FD region to "bin" their charge, which is then connected to the CDS component. In CDS mode, each pixel is first connected to the FD region 804 and reset by making the TG signal valid. Then the transmission gates are disconnected and the exposure cycle begins. During the exposure cycle, the reset voltage is transmitted to the CR 842 in the corresponding CDS component 840 by making the GSR signal valid. After the voltage is transmitted, the GSR signal is invalidated. The exposure cycle ends and the transmission gates are closed to transmit the voltage of these transmission gates to the FD region 804. Then, the GSSELS signal is enabled to transmit the signal voltage to the CS 844 in the corresponding CDS component 840. Once the reset voltage and the signal voltage are stored in the CDS component 840, the voltage can be output to the corresponding column line 920a. Column lines 920a and 920b transmit signals to corresponding ADCs 930a and 930b, respectively, to generate digital values of corresponding voltages.
[0092] Although the global shutter mode provides a global shutter, the global shutter mode also provides a lower resolution than the rolling shutter mode of the present example. In the rolling shutter mode, all sixteen photodiodes in the four pixel arrays 800a to 800d will be discretely sampled to produce sixteen discrete pixel values. However, in the global shutter mode, each pixel array 800a to 800d will output a single aggregate value for all photodiodes in the corresponding pixel array 800a to 800d, and all four pixel arrays 800a to 800d are simultaneously connected to the CDS component 840 to combine and average their voltages at the CDS component 840, thereby providing an average pixel value for the 16 independent photodiodes in the cluster 900. And although the present example associates four pixel arrays 800a to 800d with each CDS component 840, any number of pixel arrays may be associated with the CDS component 840.
[0093] Reference now Fig. 10A and Fig. 10B , Fig. 10A An example pixel array 1000 is shown that is configured to provide a rolling shutter mode, a global shutter mode with CDS, and a global shutter mode with high dynamic range (“HDR”) and CDS. Fig. 8ASimilar to the example shown, the pixel array 1000 includes four photodiodes 1002a to 1002d that can be selectively connected to the FD region 1004, but the pixel array 1000 can be designed to have any number of photodiodes depending on the application. The SF 1006 provides an output from the FD region 1004, which can be transmitted to a different destination via one of four switches. Fig. 8A Similarly, switch 1020 outputs RS1021 to the corresponding column line for operating in rolling shutter mode. In global shutter mode, GSR and GSS transmit reset value 1023a and signal value 1023b to the CDS component for storage in CR 1042 and CS 1044, respectively. In addition, as will be discussed in more detail below, a third global shutter signal (global shutter HDR select ("GSHS")) transmits a voltage from SF 1006 to capacitor CHS in CDS component 1040.
[0094] As described below, operation in HDR mode involves closing a single TG (e.g., TG 1010a) after an exposure period to obtain a “highlight” value that is transmitted as GSH 1023c by asserting GSHS to close switch 1022c and couple the output of SF 1006 to CDS component 1040. The high light value may indicate whether the corresponding photodiode (PD) 1002a-1002d is saturated during the exposure period, or whether a charge level is reached that saturates FD 1004 (e.g., the stored charge at the selected PD exceeds approximately 25% of the capacity of FD 1004) when all charge from all four PDs 1002a-1002d is transferred to FD 1004. After the high light signal is transmitted, the remaining three TGs 1010b to 1010d are closed to connect the corresponding PDs 1002b to 1002d to the FD 1004, thereby recycling the charges from all four PDs 1002a to 1002d like a normal global shutter CDS operation.
[0095] Fig. 10B Shows the Fig. 10A The CDS component 1040 corresponding to the pixel array 1000 shown in FIG. Fig. 8A Like the example CDS component 840 shown in FIG. 1 , the CDS component 1040 includes a CR 1042 capacitor and a CS 1044 capacitor. In addition, the CDS component 1040 includes a third capacitor to store the voltage of GSH 1023c. Each of the plurality of stored voltages can be output by closing a switch - GSSELR, GSSLES or GSSELHS - to connect the corresponding source follower to the column line for readout.
[0096] Fig.11 A partial view of an example hybrid image sensor with a multi-mode shutter is shown. Fig.11 The views shown include Fig. 10A A cluster 1100 of four pixel arrays 1000a to 1000d of the type shown in FIG. Fig. 9A As in the example shown in FIG. , the example image sensor includes an Fig. 9B The two-dimensional layout 910 shown is a plurality of pixel arrays 1000 arranged.
[0097] As mentioned above Fig. 10A and Fig. 10B As discussed, pixel arrays 1000a to 1000d are coupled to depicted column lines 1120a and 1120b, respectively, through RSSEL switches or via CDS component 1040 to enable readout of pixel values. In rolling shutter operation, the first row of pixel arrays 1000a and 1000b are first integrated and read out to their corresponding column lines: each pixel in each pixel array is sequentially connected to FD region 1004, which is then connected to the corresponding column line via SF 1006 and rolling shutter switch by activating RSSEL. After reading out each pixel in the first row, each pixel in the second row in pixel arrays 1000c and 1000d can be integrated and read out in the same manner. The remaining rows of pixels in the pixel sensor can then be read out in the same manner to capture a full-resolution image.
[0098] In contrast, in global shutter HDR low-resolution mode, all four pixels in each pixel array are connected to the corresponding FD region, which is then connected to the CDS component. In CDS mode, similar to the above Fig. 9A , first connect each pixel to the FD region 1004 and reset each pixel. Then disconnect each transmission gate and start the exposure period. During the exposure period, the reset voltage is transmitted to the CR 1042 in the corresponding CDS part 1040 by making the GSR signal valid. After the voltage is transmitted, the GSR signal is invalidated. At the end of the exposure period, close each transmission gate to transmit the voltage of these transmission gates to the FD region 1004. Then, make the GSSELS signal valid to transmit the signal voltage to the CS 1044 in the corresponding CDS part 1040. Once the reset voltage and the signal voltage are stored in the CDS part 1040, the voltage can be output to the corresponding column line 1120a.
[0099] As above Fig. 9A and Fig. 9BAs discussed in the example shown in FIG, all four pixel arrays 1000a to 1000d are simultaneously connected to the CDS component 1040 to combine and average their voltages at the CDS component 1040 to provide an average pixel value for the 16 individual photodiodes in the cluster 1100. And although this example associates four pixel arrays 1000a to 1000d with each CDS component 1040, any number of pixel arrays may be associated with the CDS component 1040.
[0100] In order to provide HDR operation in this example, two independent readout cycles are performed. The first ADC is completed by first reading out the reset value from CR and then reading out the high light signal value from CHS. After the high light signal value is generated, the second ADC is completed by reading out the same reset value of the pixel from CR again and then reading out the regular signal value from CS. The second regular signal value is generated after the second ADC cycle. In this example, the dynamic range can be extended by 4 times, or 12 decibels (dB). The HDR information contained in the high light signal value and the regular signal value can be sent out of the sensor for off-chip processing. Traditional HDR combination and / or tone mapping algorithms can be applied to create HDR images. An on-chip HDR processing unit can also be implemented, for example, in an image signal processing unit.
[0101] Although the image sensor offers HDR functionality, this is not required. Instead, the sensor can be operated in global shutter CDS mode without HDR - highlight samples can be taken before the full value of the pixel array is acquired. Fig. 10A As discussed, after an exposure period, a single photodiode 1000a in each pixel array 1000a to 1000d may be connected to a corresponding FD region 1004 to transfer the charge of the photodiode. The FD region may be connected to the CDS component to store the charge of the single photodiode as GSH 1023c. The remaining three photodiodes may then be connected to the FD region 1004 to recover the charge of all four photodiodes, which are then transferred to the CS capacitor 1044 in the CDS component 1040. Each of the three stored values (reset value GR, HDR or highlight value GHS, and pixel array value GS) may then be read out successively by closing the corresponding switches 1050 to 1054 in the CDS component 1040 to connect them to the corresponding column lines. Fig. 9A Like the example shown in , the global shutter mode provides a global shutter with a lower resolution than the rolling shutter mode of this example.
[0102] Reference now Fig. 12A and Fig. 12B , Fig. 12AAn example pixel array 1200 configured to provide a rolling shutter mode and a global shutter mode is shown. Fig. 8A and Fig. 10A Similar to the example shown in , the pixel array 1200 includes four photodiodes 1202a to 1202d that can be selectively connected to the FD region 1204, but the pixel array 1200 can be designed to have any number of photodiodes depending on the application. The SF 1206 provides an output from the FD region 1204, which can be transmitted to a different destination via one of two switches. Fig. 8A Similarly, switch 1220 outputs RS1221 to the corresponding column line for running the rolling shutter mode. However, in the global shutter mode, Fig. 12B As shown, GS transmits pixel array signal value 1223 to pixel array level ADC 1240. Thus, although the present example provides both rolling shutter mode and global shutter mode, the global shutter mode does not include a CDS component.
[0103] The pixel array-level ADC 1240 provides analog-to-digital conversion of the values output by the pixel array, which may represent the recovered values of a single photodiode or multiple photodiodes (e.g., all four photodiodes 1202a to 1202d in this example). Fig. 8A and Fig. 10A As with the discussed examples, the global shutter mode operates in a low-resolution mode where the pixel array outputs a single pixel value representing the recycled charge from all photodiodes 1202a to 1202d in the pixel array. During global shutter operation, all four photodiodes collect charge during an exposure period, after which all four TGs 1210a to 1210d are closed to transfer charge from all four photodiodes 1202a to 1202d to the FD region 1204. A global shutter ("GS") signal is asserted for the pixel array 1200 to close the global shutter switch 1222, thereby transferring a voltage to the corresponding pixel array-level ADC 1240, which can generate a pixel value and output it to the corresponding column line. As will be seen in reference Fig.13 As discussed, multiple pixel arrays may share a single pixel array-level ADC 1240 , but in some examples, each pixel array may have a dedicated pixel array-level ADC 1240 .
[0104] The pixel array level ADC 1240 includes a comparator 1242 that receives the GS signal 1223 from the pixel array 1200 and compares it with the ramp signal 1241. In the example using CDS, a reset voltage and a signal voltage can be provided to the pixel array level ADC 1240 in sequence. In order to provide digital values representing the reset value and the signal value, an up-down counter is used in this example. The counter value can then be output to the corresponding column line by making the appropriate GSSEL signal valid. Therefore, this example configuration enables the above reference generally Fig. 8A and Fig. 10A The described rolling shutter operation, as well as a lower resolution global shutter operation using a dedicated pixel array-level ADC.
[0105] Fig.13 A partial view of an example hybrid image sensor with a multi-mode shutter is shown. Fig.13 The views shown include Fig. 12A A cluster 1300 of four pixel arrays 1200a to 1200d of the type shown in FIG. Fig. 9A As in the example shown in FIG. , the example image sensor includes an Fig. 9B The two-dimensional layout 910 shown is a plurality of pixel arrays 1200 arranged.
[0106] In this example, the image sensor can be substantially as described above with reference to Fig.11 The image sensor operates in the rolling shutter mode described. Alternatively, the image sensor can operate in the global shutter mode. As described above, in the global shutter mode, before the exposure cycle begins, the photodiodes 1202a to 1202d and the FD region 1204 are reset, and during the exposure cycle, the photodiodes 1202a to 1202d accumulate charge. After the exposure cycle, all TGs 1212a to 1212d are closed to transfer and recover the charge from the photodiodes 1202a to 1202d at the FD region 1204. Then, the global shutter switch 1222 is closed, and the pixel array value GS1223 is transmitted to the pixel array level ADC 1240, where the pixel array value GS1223 is converted into a digital value and output to the bit lines 1330a to 1330n. In this example, the cluster 1300 includes four pixel arrays 1200a to 1200d, which operate simultaneously during the same exposure cycle. As mentioned above with reference to FIG. 9 and Fig.11 As discussed, each of the four pixel arrays 1200a-1200d is simultaneously connected to the pixel array level ADC 1240 to provide an average value to the 16 photodiodes within the cluster 1300. Each pixel array is then reset for the next image frame.
[0107] Reference now Fig.14A and Fig. 14B , Fig.14A Shows the Fig. 12A 1400 is an example pixel array 1400 similar to the pixel array 1200 shown in . Like the pixel array 1200, the pixel array 1400 includes four photodiodes 1402a to 1402d connected to a charge storage device (FD region 1404) by TGs 1410a to 1410d. In the rolling shutter configuration, similar to other examples discussed herein, the photodiodes 1402a to 1402d each accumulate charge during a corresponding exposure cycle, which can be staggered in time or can occur simultaneously. Each photodiode 1402a to 1402d is sequentially connected to the FD region 1404 before its exposure cycle is completed. In addition, the RSSEL signal is enabled to close the switch 1420 and couple the output of the SF 1406 to the corresponding column line for transmitting the output voltage to the column-level ADC for conversion to a digital value. Therefore, the charge level of each photodiode 1402a to 1402d is sequentially read out and converted before being reset.
[0108] In global shutter mode, the reset voltage of the pixel array is first sampled by closing the four TG switches 1410a to 1410d and activating the GS signal to transmit the pixel array reset value to the CDS component 1440. Fig.15 In the example to be described in , as will be described below, the CDS component is shared by four pixel arrays, and the reset value of each pixel array in each pixel array will be sequentially transmitted to the CDS component 1440 and integrated. The integrated reset value is stored in the capacitor CR 1442. Then, each photodiode accumulates charge during the exposure period, and its charge is transmitted and recycled to the FD region 1404. The GS_x,y signal is again enabled to transmit the voltage of the pixel array to the CDS component, which sequentially integrates the pixel values received from the four connected pixel arrays. The integrated pixel array value is then stored in the capacitor CS1444. The reset value and the pixel array value can then be output to the corresponding column line by enabling and disabling the GSSELR signal and the GSSELS signal in sequence.
[0109] Fig. 14B An example CDS component 1440 is shown, which includes information related to, for example, Figure 8B1442 and CS capacitors are similar to the CR capacitor and CS capacitor shown in FIG. 1444. In addition, the CDS component 1440 includes a switched capacitor integrator 1460 for integrating pixel array values received from the pixel arrays 1400a to 1400d connected to the CDS component. In global shutter mode, the pixel array is reset, the transfer gate is opened, and the integration period begins. The reset voltage is then transmitted to the CDS component, which integrates the reset voltage and stores the resulting voltage in the CR capacitor 1442. During the exposure period, the pixel array accumulates charge, and after the exposure period ends, the transfer gates are closed, the photodiodes 1402a to 1402d recover their charge at the FD region 1408, and the resulting voltage is then transmitted to the input of the switched capacitor integrator 1460. Like the reset voltage, the pixel array voltage is transmitted and integrated, where the integrated voltage is stored in the CS capacitor 1444. The reset value and the pixel value can then be read sequentially by asserting and deasserting the GSSELR and GSSELS signals.
[0110] refer to Fig.15 , Fig.15 A partial view of an example hybrid image sensor with a multi-mode shutter is shown. Fig.15 The views shown include Fig.14A A cluster 1500 of four pixel arrays 1400a to 1400d of the type shown in FIG. Fig. 9A As in the example shown in FIG. , the example image sensor includes an Fig. 9B The two-dimensional layout 910 shown is a plurality of pixel arrays 1400 arranged.
[0111] In this example, the image sensor is configured for both full resolution rolling shutter mode and global shutter pixel array averaging mode. The full resolution rolling shutter mode is similar to the above reference Fig. 8A The rolling shutter mode described above operates in the same manner. However, in global shutter mode, the outputs of all four pixel arrays 1400a to 1400d associated with the CDS component are recycled and integrated, reducing the resolution of the image sensor by a factor of 16, but increasing the sensitivity of the sensor by the same factor. Thus, the loss of resolution is traded for an increase in low light sensitivity.
[0112] In the global shutter mode, each of the pixel arrays 1400a to 1400d is reset and sequentially connected to the CDS component to transfer the voltage output by its corresponding SF 1408 to the input of the switched capacitor integrator, which integrates the reset voltage and outputs the resulting voltage to the CR capacitor 1442. In addition, each of the pixel arrays 1400a to 1400d accumulates charge during a common exposure period, and after the exposure period, recycles the resulting charge into the corresponding FD region 1408. Then, the pixel arrays 1400a to 1400d are sequentially connected to the CDS component to transfer the voltage output by its corresponding SF 1408 to the input of the switched capacitor integrator, which integrates the signal voltage and outputs the resulting voltage to the CS capacitor 1444.
[0113] The operation begins by first sampling the even-row, even-column pixel reset value (GS_E, E is on) with the sampling switch S1 and the amplifier reset switch enabled. The reset value Vrst1 of the first pixel array 1400a in the cluster 1500 is sampled on C1 (Vrst1-Voff), and the voltage across C2 with an infinite gain amplifier is zero. After sampling Vrst1, S1 and the amplifier reset switch are turned off, and the amplification switch S2 is turned on. During the amplification phase, the charge transfer from C1 to C2 exhibits a voltage of (Vrst1-Voff)×(C1 / C2) at the output of the switched capacitor integrator. After Vrst1 is integrated into the output, the even-row, odd-column pixel reset value (GS_E, O is on) is sampled with S1 turned on. The reset value Vrst2 of the corresponding pixel array 1400b is sampled on C1 (Vrst2-Voff), while the voltage across C2 remains at (Vrst1-Voff)×(C1 / C2). After sampling Vrst2, S1 and the amplifier reset switch are turned off, and the amplification switch S2 is turned on again. During the amplification phase, the charge transfer from C1 to C2 enables the switched capacitor integrator to add the second reset value (Vrst2-Voff)×(C1 / C2) to the previously established value (Vrst1-Voff)×(C1 / C2). Once the amplification is completed, the output of the switched capacitor integrator becomes (Vrst1+Vrst2-2×Voff)×(C1 / C2). This operation continues until all four reset values of the four pixel arrays 1400a to 1400d are integrated at the switched capacitor integrator output as (Vrst1 + Vrst2 + Vrst3 + Vrst4 - 4 x Voff) x (C1 / C2). This value will be sampled at the reset capacitor CR1442 with the GSR enabled.
[0114] Once the voltage recovery reset value is sampled, charge transfer occurs by enabling all TG signals and exposure ends. The signal values of all four pixel arrays 1400a to 1400d will be integrated in the same manner as the reset value through the same switched capacitor integrator, and the voltage recovery signal value will be sampled as (Vsig1+Vsig2+Vsig3+Vsig4-4×Voff)×(C1 / C2) on the signal capacitor CS1444 with GSS enabled. When the ADC reads out the reset value and signal value, the CDS operation removes the noise component of every 4 shared pixel cells, resulting in a pixel value [(Vsig1+Vsig2+Vsig3+Vsig4)-(Vrst1+Vrst2+Vrst3+Vrst4)]×(C1 / C2). When C1 is equal to C2, the pixel value is [(Vsig1+Vsig2+Vsig3+Vsig4)-(Vrst1+Vrst2+Vrst3+Vrst4)], which represents the voltage recovery value of the four pixel arrays. In addition, different C1 / C2 ratios can be used as programmable gain amplifiers during voltage recovery operations.
[0115] Reference now Fig.16 , Fig.16 An example method 1600 for a hybrid image sensor with a multi-mode shutter is shown. Fig. 8A and Figure 8B ,as well as Fig. 9A Example method 1600 is discussed with reference to the image sensor shown in ; however, any suitable image sensor consistent with the present disclosure may be employed.
[0116] At block 1602, the image sensor enables a global shutter mode of the image sensor. In this example, the global shutter mode may be enabled by a controller of the image sensor (eg, controller 606) based on a command from another device (eg, host processor 604).
[0117] At block 1610, the image sensor resets the charge storage devices 804 of the pixel arrays 800a to 800d. In this example, the image sensor closes all transfer gates by making all TG_x,y signals valid and makes the reset signal of each pixel array 800a to 800d valid to reset each photodiode and charge storage device of the pixel array 800a to 800d. Fig.11 As discussed, in this example, the charge storage device is the FD region 804 , but any suitable charge storage device may be employed.
[0118] At block 1620, the image sensor turns off the transfer gates and begins an integration period for the image frame. In some examples, the reset voltage may be sampled while integration is ongoing; however, the reset voltage may alternatively be sampled before integration begins.
[0119] At block 1630, each light sensing element 802a-802d of the pixel array accumulates charge during an integration period.
[0120] At block 1640, for each pixel array 800a to 800d, the image sensor transfers the reset voltage stored at the charge storage device 804 to the corresponding CDS component 840. In this example, the image sensor asserts the GSR signal to close the corresponding switch 822a, thereby connecting the output of SF 806 to the CR capacitor 842 of the CDS component 840 of the corresponding pixel array 800a.
[0121] At block 1650, for each pixel array, the image sensor transfers the accumulated charge from each light sensing element to the corresponding charge storage device (e.g., FD region 804). In this example, the image sensor enables all transfer gates by making all TG_x,y signals valid, which recycle the charge from all light sensing elements in a particular pixel array 800a to 800d at the corresponding charge storage device (e.g., FD region 804), where the charge is stored as a signal voltage of the pixel array 800a to 800d. After recycling the charge at the charge storage device, the image sensor makes the GSS signal valid to close the corresponding GSS switch 822b in the pixel array and transfer the stored signal voltage to the CS capacitor 844 in the CDS component.
[0122] At box 1660, the voltage stored in the CDS is output to the corresponding column line (e.g., column line 920a). In this example, the reset voltage is transmitted by making the corresponding GSSELR signal valid. The GSSELR signal is then invalidated, and the corresponding GSSELS signal is then valid to transmit the signal voltage to the column line 920a. Finally, in the example where the HDR capability has been enabled in global shutter mode, the corresponding GSSELHS signal is valid to transmit the high light signal voltage to the column line. The voltage transmitted to the corresponding column line can then be converted to a digital value by the column ADC 930a. In some examples, the reset voltage and the signal voltage can be subtracted in the analog domain before performing the ADC. However, in some examples, both voltages can be converted to digital values before the subtraction is performed.
[0123] Reference now Fig.17 , Fig.17 An example method 1700 for a hybrid image sensor with a multi-mode shutter is shown. Fig. 10A and Fig. 10B ,as well as Fig.11 Example method 1700 is discussed using the image sensor shown in ; however, any suitable image sensor according to the present disclosure may be employed.
[0124] At block 1702, the image sensor enables a global shutter mode of the image sensor, generally as described above with reference to block 1602. However, in this example, the image sensor is configured with optional HDR functionality in the global shutter mode. Thus, in this example, the image sensor enables a global shutter mode with HDR. However, an image sensor configured with optional HDR functionality may enable a global shutter mode without enabling the HDR functionality, which may then be generally described above with reference to block 1602. Fig.16 Works as described.
[0125] At block 1710 , the image sensor resets the charge storage devices 1004 of the pixel array 1000 a - 1000 d , substantially as described above with reference to block 1610 .
[0126] At block 1720 , each light sensing element 1002 a - 1002 d of the pixel array accumulates charge during an integration period, substantially as described above with reference to block 1630 .
[0127] At block 1730 , the image sensor transfers the reset voltage stored at the charge storage device 1004 to the corresponding CDS component 1040 and begins an integration period for each pixel array 1000 a - 1000 d , substantially as described above with reference to block 1640 .
[0128] At block 1740, after the integration period, the image sensor transfers the accumulated charge of one light sensing element (e.g., photodiode 1002a) to the charge storage device 1004. In this example, the image sensor asserts the even column and row TG signals to close the transfer gate 1010a of one photodiode 1002a in each pixel array and transfers the accumulated charge of the photodiode 1002a to the FD region 1004. The accumulated charge of the single photodiode 1002 is stored at the FD region 1004 as a high light signal voltage.
[0129] After transmitting the high light signal voltage to the FD region 1004, the image sensor asserts the GSHS signal to transmit the stored high light voltage signal (which is output by the SF 1006) to the CHS capacitor 1046 in the corresponding CDS component, where the high light voltage signal is stored. It should be understood that in an image sensor (e.g., an image sensor not configured with a CDS component having a CHS capacitor 1046) Fig. 9AIn the example image sensor shown in FIG. 1 , or in an image sensor not operating in a global shutter mode with HDR, block 1740 may be omitted, such as in the example image sensor shown in FIG. 1 . Fig.16 as described.
[0130] At block 1750, for each pixel array, the image sensor transfers the accumulated charge from each light sensing element to a corresponding charge storage device (e.g., FD region 1004). In this example, the image sensor enables all remaining open transfer gates by asserting all invalid TG_x,y signals, which recycle the charge from all light sensing elements in a particular pixel array 1000a to 1000d at the corresponding charge storage device (e.g., FD region 1004), where the charge is stored as a signal voltage of the pixel array 1000a to 1000d. After reclaiming the charge at the charge storage device, the image sensor asserts the GSS signal of each pixel array 1000a to 1000d to transfer and average the stored signal voltage to the CS capacitor 1044 in the CDS component.
[0131] At block 1760, the voltage stored in the CDS is output to the corresponding column line (e.g., column line 1120a). In this example, the reset voltage is transmitted by making the corresponding GSSELR signal valid. The GSSELR signal is then invalidated, and the corresponding GSSELS signal is then valid to transmit the signal voltage to the column line 1120a. Finally, the corresponding GSSELHS signal is valid to transmit the high light signal voltage to the column line. The voltage transmitted to the corresponding column line can then be converted to a digital value by the column ADC 1130a. And although the voltages are transmitted out in a specific order in this example, they can be transmitted in any suitable order.
[0132] Reference now Fig.18 , Fig.18 An example method 1800 for a hybrid image sensor with a multi-mode shutter is shown. Fig. 8A and Figure 8B ,as well as Fig. 9A The example method is described using the image sensor shown in ; however, any suitable image sensor according to the present disclosure may be employed.
[0133] At block 1802, the image sensor enables a global shutter mode of the image sensor, generally as described above with reference to block 1602. However, in this example, the image sensor is configured with an optional voltage averaging function in the global shutter mode. Thus, in this example, the image sensor enables a global shutter mode with voltage averaging.
[0134] Blocks 1810 to 1830 are generally as described above with reference to Fig.16 as described.
[0135] At block 1840, for each pixel array, the image sensor transfers the accumulated charge from each light sensing element to a corresponding charge storage device (e.g., FD region 804). In this example, the image sensor enables all transfer gates by asserting all TG_x,y signals, which recycle the charge from all light sensing elements in a particular pixel array 800a to 800d at the corresponding charge storage device (e.g., FD region 804), where the charge is stored as a signal voltage of the pixel array 800a to 800d.
[0136] At block 1850, the image sensor enables the GSS signals of all pixel arrays to transmit and average the voltages from each pixel array at the corresponding CDS components. Thus, the CS capacitor 844 stores the average voltage of the four associated pixel arrays 800a to 800d. This approach reduces the resolution of the image sensor to 1 / 16 of its full resolution, but can reduce the impact of noise on the image.
[0137] At block 1860 , the reset voltage and the signal voltage are outputted, substantially as described above with reference to block 1650 .
[0138] refer to Fig.19 , Fig.19 An example method 1900 for a hybrid image sensor with a multi-mode shutter is shown. Fig.14A and Fig. 14B ,as well as Fig.15 The example method is described using the image sensor shown in ; however, any suitable image sensor according to the present disclosure may be employed.
[0139] Block 1902 and block 1910 are performed as described above with reference to block 1602 and block 1610, respectively.
[0140] At block 1920 , the pixel array 1400 a - 1400 d accumulates charge during an exposure period, generally as discussed above with reference to block 1630 .
[0141] At block 1930, the image sensor stores a reset voltage at the CDS component 1440. In this example, the CDS component 1440 includes a switched capacitor integrator 1460 that is selectively connected to each pixel array through a switch closed by the GS_x,y signal. Fig. 9A and Fig.11 This is in contrast to the example shown in , which has separate switches to transmit reset and signal voltages to the respective CDS components 840 , 1040 .
[0142] To store the reset voltage at the CDS component 1440, the image sensor validates the GS_x,y signal of the pixel array 1400a to connect the particular pixel array 1400a to the input of the switched capacitor integrator. It also validates the GSR signal to connect the output of the switched capacitor integrator 1460 to the CR capacitor in the CDS component. To complete the integration of the reset voltage, it then sequentially validates the remaining GS_x,y signals of the other pixel arrays 1400b to 1400d to connect them to the input of the switched capacitor integrator 1460.
[0143] At block 1940, the exposure period ends, and the image sensor transfers the charge from each light sensing element to the charge storage device in the pixel array, generally as described above with reference to block 1650. However, as discussed above with reference to block 1930, to transfer the voltage stored at the charge storage device to the CDS component 1440, the image sensor asserts the GS_x,y signal of one of the pixel arrays 1400a associated with the CDS component 1440, and asserts the corresponding GSS signal to connect the output of the switched capacitor integrator 1460 to the CS capacitor 1444. To complete the integration of the signal voltage, it then sequentially asserts the remaining GS_x,y signals of the other pixel arrays 1400b through 1400d and connects them to the input of the switched capacitor integrator 1460.
[0144] At block 1950 , the image sensor outputs a voltage as described above with reference to block 1660 .
[0145] Reference now Fig. 20 , Fig. 20 An example method 2000 for a hybrid image sensor with a multi-mode shutter is shown. Fig. 8A as well as Fig. 9A The example method is described using the image sensor shown in ; however, any suitable image sensor according to the present disclosure may be employed.
[0146] At block 2002, an image sensor enables a rolling shutter mode. In this example, the rolling shutter mode may be enabled by a controller of the image sensor (eg, controller 606) based on a command from another device (eg, host processor 604).
[0147] At block 2010 , the image sensor resets the charge storage device 804 , substantially as described above with reference to block 1610 .
[0148] At block 2020, the light sensing elements accumulate charge during corresponding exposure cycles. In this example, each light sensing element staggers its exposure cycle to allow the preceding light sensing element to transfer its charge to the charge storage device and transfer the resulting voltage to the corresponding column line. However, in some examples, each of the light sensing elements may begin its exposure cycle at the same time.
[0149] At block 2030, the RSSEL_0 signal is asserted to transmit a reset voltage to the column line to support CDS operation. This step may be omitted if CDS is not employed. Since block 2030 may be revisited multiple times during a single sampling operation, the charge storage device 804 may be reset each time block 2030 is executed before a new reset voltage is read out.
[0150] At block 2040 , the pixel array connects the first light sensing element 802 a to the charge storage device 804 by asserting the corresponding TG_x,y signal to close the corresponding transfer gate 810 a .
[0151] At block 2050, the pixel array connects the output of the SF 806 to the corresponding column line 920 by activating the corresponding RSSEL_X signal to close the row select switch 820 to transmit the signal voltage RS_X 821 to the column ADC. Thereafter, the method returns to block 2030 or block 2040, depending on whether the CDS operation is used. If the CDS operation is used, the method returns to block 2030. Otherwise, the method returns to block 2040 to transfer the charge of the next light sensing element 802b to 802d in the pixel array 800 and transmit the resulting voltage to the column line. Once all light sensing elements in the pixel array 800a are read out, image sampling is completed.
[0152] Figures 16 to 20 All are described in the context of one set of pixel arrays and corresponding CDS components. However, the example image sensor will include multiple sets of pixel arrays and CDS components. Therefore, the method can be performed on all pixel arrays and CDS components within the image sensor to capture an image.
[0153] The foregoing description of some examples is provided for the purpose of illustration and description only; and the description is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. For those skilled in the art, many modifications and adaptations of the present disclosure will be apparent without departing from the spirit and scope of the present disclosure.
[0154] Reference to an example or an embodiment herein means that a specific feature, structure, operation, or other feature described in conjunction with the example may be included in at least one embodiment of the present disclosure. The present disclosure is not limited to the specific examples or embodiments described. The phrases "in one example", "in an example", "in one embodiment", or "in one embodiment" or variations of these phrases that appear in different places in this specification do not necessarily refer to the same example or embodiment. Any specific feature, structure, operation, or other feature described in this specification related to an example or an embodiment may be combined with other features, structures, operations, or other features described with respect to any other example or embodiment.
[0155] As used herein, the word "or" is intended to encompass both inclusive and exclusive OR conditions. In other words, A or B or C includes any or all of the following optional combinations suitable for a particular usage: A alone; B alone; C alone; A and B only; A and C only; B and C only; and A and B and C.
Claims
1. A sensor device, comprising: a plurality of pixel arrays, each pixel array comprising a plurality of pixels, each pixel comprising a light sensing element configured to generate and store charge in response to incident light; Each pixel array consists of: a charge storage device configured to receive charge from each of a plurality of light sensing elements of the pixel array; a first plurality of switches, each switch of the first plurality of switches connected between a corresponding pixel of the pixel array and the charge storage device; a second plurality of switches, the second plurality of switches comprising a high resolution select switch and a low resolution select switch, each of the high resolution select switch and the low resolution select switch connected in parallel to an output of the charge storage device; A plurality of pixel output lines, each pixel output line configured to output a signal representing a pixel value corresponding to one or more pixel arrays coupled to the corresponding pixel output line.
2. The sensor device according to claim 1, further comprising: A plurality of analog-to-digital converters ADC, each ADC is connected to the output of a corresponding pixel output line.
3. The sensor device according to claim 1 , further comprising a correlated double sampling (CDS) component for each pixel array, the CDS component comprising: CDS reset value storage device, and CDS pixel value storage device; and in, The second plurality of switches further includes a CDS reset value switch connected in parallel with the high resolution selection switch and the low resolution selection switch to the output of the charge storage device, The CDS reset value switch connects the charge storage device to a corresponding CDS reset value storage device, The low resolution selection switch connects the charge storage device to the CDS pixel value storage device, The output of the CDS reset value storage device is connected to the corresponding pixel output line, and The output of the CDS pixel value storage device is connected to a corresponding pixel output line; and The sensor device is configured to operate in CDS mode and is configured to, for one or more pixel arrays: closing and opening a corresponding CDS reset value switch to sample a reset voltage through the CDS reset value storage device; Closing and opening a plurality of switches in the first plurality of switches to transfer charge from the plurality of light sensing elements to corresponding charge storage devices; and closing and opening corresponding resolution pixel value switches to sample pixel voltages through the CDS pixel value storage devices.
4. The sensor device according to claim 3, wherein: Pixel array subsets of the plurality of pixel arrays share corresponding CDS components, and wherein the pixel array subsets are configured to be simultaneously connected to the corresponding CDS components to provide voltage averaging for the pixel array subsets.
5. The sensor device according to claim 3, wherein: The sensor device is further configured to selectively close and open the high-resolution selection switch in sequence when selectively closing each switch of the first plurality of switches in sequence to acquire a pixel value of each corresponding light sensing element.
6. The sensor device according to claim 3, wherein: The sensor device is further configured to: selectively closing the CDS reset value switch when the first plurality of switches of each pixel array are opened to store a reset pixel value in the CDS reset value storage device; as well as The CDS selection switch is selectively closed when the first plurality of switches of each pixel array are closed to store pixel values in the CDS pixel value storage device.
7. The sensor device of claim 3, further comprising a high-intensity pixel value storage device for each pixel array, and wherein The second plurality of switches also includes a high intensity pixel value switch, and The high intensity pixel value switch connects the charge storage device to a corresponding high intensity pixel value storage device, and wherein, The sensor device is configured to operate in a high intensity CDS mode and is configured, for one or more pixel arrays: closing and opening a corresponding CDS reset value switch to sample a reset voltage through the CDS reset value storage device; closing and opening a first switch of a corresponding first plurality of switches to transfer a first charge from a first light sensing element to a corresponding charge storage device; and closing and opening a corresponding high intensity pixel value switch to sample a high intensity voltage through the high intensity pixel value storage device; closing and opening remaining switches of the first plurality of switches to transfer charge from remaining light sensing elements to corresponding charge storage devices; And closing and opening the corresponding resolution pixel value switch to sample the pixel voltage through the CDS pixel value storage device. 8 . The sensor device according to claim 3 , further configured to perform voltage recovery on multiple groups of pixel arrays.
9. The sensor device according to claim 1 further comprises one or more correlated double sampling (CDS) components, each CDS component comprising a comparator and an up-down counter, wherein the input of the comparator is connected to one or more low-resolution selection switches of one or more corresponding pixel arrays.
10. The sensor device according to claim 1, wherein The sensor device is configured to operate in a high-resolution rolling shutter mode or a low-resolution global shutter mode, and wherein the sensor device is configured to: In the high-resolution rolling shutter mode, selectively closing each switch of the first plurality of switches of each pixel array in sequence; and In the low-resolution global shutter mode, each switch in the first plurality of switches of each pixel array is closed simultaneously.
11. The sensor device according to claim 1, wherein: Each pixel array includes four light sensing elements.
12. The sensor device according to claim 1, further comprising one or more correlated double sampling (CDS) components, each CDS component comprising a switched capacitor integrator, a CDS reset value storage device, and a CDS pixel value storage device, wherein: The CDS reset value storage device and the CDS pixel value storage device are selectively coupled to the output of the switched capacitor integrator.
13. The sensor device according to claim 12, wherein: Each subset of the plurality of pixel arrays is associated with a respective CDS component of the one or more CDS components.
14. The sensor device according to claim 1, wherein: Each pixel array includes four light sensing elements arranged in a 2×2 grid.
15. A method comprising: enabling the global shutter mode in an image sensor having at least a rolling shutter mode and a global shutter mode, the image sensor having a plurality of pixel arrays, each pixel array including a plurality of light sensing elements and a charge storage device, the charge storage device being configured to receive charge from each of the plurality of light sensing elements of the pixel array, the plurality of light sensing elements being selectively connected to the charge storage device; resetting each charge storage device of the image sensor to establish a reset voltage; For each pixel array, transmitting the reset voltage to a corresponding correlated double sampling (CDS) component; During an integration period, accumulating charge within each of the light sensing elements of the plurality of pixel arrays; For each pixel array, transferring the accumulated charge from each light sensing element to a corresponding charge storage device for storage as a signal voltage; For each pixel array, transmitting the signal voltage from the corresponding charge storage device to the corresponding CDS component; The reset voltage and the signal voltage are output from corresponding CDS parts.
16. The method according to claim 15, further comprising: For each pixel array, transferring accumulated charge from one light sensing element to a corresponding charge storage device to be stored as a high light signal voltage; For each pixel array, transmitting the high optical signal voltage to a corresponding CDS component; as well as For each pixel array, the high optical signal voltage is output from the corresponding CDS component.
17. The method according to claim 15, further comprising: For each CDS component, the signal voltages from all pixel arrays corresponding to the corresponding CDS component are simultaneously transmitted to the corresponding CDS component and averaged.
18. The method according to claim 15, wherein: Each CDS component includes a switched capacitor integrator, and the method further includes: receiving the reset voltage by each CDS component of the corresponding pixel array and integrating the reset voltage; storing, by each CDS component, the integrated reset voltage as the reset voltage; receiving the signal voltage by each CDS unit of the corresponding pixel array and integrating the signal voltage; and The integrated reset voltage is stored as the reset voltage by each CDS unit.
19. The method according to claim 15, further comprising: enabling the rolling shutter mode in the image sensor; For each pixel array, resetting the corresponding charge storage device; accumulating charge within each of the respective light sensing elements of the plurality of pixel arrays during a corresponding integration period; For each light sensing element in the corresponding pixel array, in turn: transferring the accumulated charge of the corresponding light sensing element to the corresponding charge storage device to be stored as a signal voltage; as well as For each pixel array, the signal voltage from the corresponding charge storage device is transmitted to the column output line.
20. The method according to claim 15, wherein: Each pixel array includes four light sensing elements arranged in a 2×2 grid.
