System and method for controlling an image sensor

By controlling the readout of image data from subsets of photodetectors in image sensors, the system addresses power and heat challenges in portable devices, enabling efficient capture of images at varying resolutions.

JP7954198B2Active Publication Date: 2026-11-19QUALCOMM INC
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2024515625
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-09-16
Filing Date
2022-09-09
Publication Date
2026-11-19
Estimated Expiration
2042-09-09

AI Technical Summary

Technical Problem

Portable electronic devices face challenges in managing power consumption, bandwidth usage, and heat generation due to the readout of image data from image sensors, particularly when capturing images of varying resolutions, which affects their battery life and operational efficiency.

Method used

The system controls the readout of image data by selectively reading from subsets of photodetectors in an image sensor, using sparse readouts for lower resolutions and dense readouts for higher resolutions, reducing power consumption and computational resources.

Benefits of technology

This approach reduces power consumption, bandwidth usage, and heat generation, enhancing the operational efficiency and battery life of portable devices while allowing flexible resolution capture based on image content and user needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007954198000001
    Figure 0007954198000001
  • Figure 0007954198000002
    Figure 0007954198000002
  • Figure 0007954198000003
    Figure 0007954198000003
Patent Text Reader

Abstract

Systems and techniques for imaging are described. The imaging system includes an image sensor with a plurality of photodetectors grouped into a first photodetector group and a second photodetector group. The imaging system can reset the image sensor. The imaging system exposes the image sensor to light from a scene. The plurality of photodetectors convert the light into an electric charge. The imaging system stores analog photodetector signals corresponding to the electric charge from each photodetector. The imaging system reads first digital pixel data from a first subset of the analog photodetector signals corresponding to the first photodetector group without reading second digital pixel data from a second subset of the analog photodetector signals corresponding to the second photodetector group. The imaging system uses the first digital pixel data to generate an image of the scene.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001]

[0001] The present application relates to image capture and / or image processing. More specifically, the present application relates to a system and method for controlling whether to perform a first readout of first image data from a first subset of pixels of an image sensor to generate an image of a first resolution, or to perform a second readout of second image data from at least a second subset of pixels of the image sensor to generate an image of a second resolution.

BACKGROUND ART

[0002]

[0002] An image sensor includes an array of photodetectors such as photodiodes. Each photodetector receives light from a scene and converts the light into charge based on how much light reaches the photodetector. Reading out image data from the image sensor may involve operations such as amplification of the charge generated by the photodetectors, conversion of an analog signal associated with the charge into a digital signal using an analog to digital converter (ADC), and / or additional image processing operations. The amount of power consumed, the bandwidth used, the heat generated, and / or the computational resources used for image capture may be related to the number of photodetectors that are read out.

[0003]

[0003] Portable electronic devices such as mobile handsets, cellular phones, head-mounted displays (HMDs), or wearable devices are becoming increasingly popular. Portable electronic devices are generally powered by portable power storage units, such as batteries, which are part of the portable electronic device. To maintain lightness and small size, and therefore to be portable, portable electronic devices generally contain relatively small batteries. Because portable electronic devices have a limited battery life, it is beneficial to reduce the power consumption of the operations performed by the portable electronic device whenever possible. Some portable electronic devices include cameras. Images of different resolutions may be useful for different use cases. In some use cases, images with a first resolution may be sufficient. In other use cases, images with a second resolution (e.g., higher than the first resolution) may be beneficial. [Overview of the project]

[0004]

[0004] In some examples, systems and techniques for controlling the readout of an image sensor to control the resolution of the captured image and the corresponding power consumption are described herein. The imaging system includes an imaging circuit. In some examples, the imaging circuit includes an image sensor having an array of photodetectors such as photodiodes. In some examples, the photodetectors of the image sensor's photodetector array are grouped, for example, based on a pattern, such that each photodetector is one of a set of several distinct groups. In exemplary examples, some of the photodetectors of the photodetector array are in a first group, some of the photodetectors of the photodetector array are in a second group distinct from the first group, and so on. According to some examples, the imaging system uses a global shutter. According to some examples, the imaging system uses a rolling shutter. The imaging system can reset the photodetectors of its image sensor. The imaging system exposes its image sensor to light from a scene. Each of the photodetectors of the image sensor converts light from the scene into an electric charge in response to exposure to light. In some examples, the imaging system receives and / or stores analog photodetector signals corresponding to charges. For example, in some examples, the imaging system stores the analog photodetector signals in storage units (e.g., storage diodes) of a storage array. The imaging system reads first digital pixel data from a first subset of analog photodetector signals corresponding to a first group of photodetectors without reading second digital pixel data from a second subset of analog photodetector signals corresponding to a second group of photodetectors.

[0005]

[0005] Reading pixel data from some of the photodetectors of an image sensor without reading other photodetectors of the image sensor is sometimes called a sparse readout. Sparse readouts reduce power consumption, bandwidth used, heat generated, and / or computational resources used compared to dense readouts of the image sensor's photodetectors. Reading more photodetectors of an image sensor than a sparse readout is sometimes called a dense readout. Reading all, or at least a subset (e.g., at least a threshold amount) of the photodetectors of an image sensor is sometimes called a dense readout. In some examples, different types of photodetectors of an image sensor may be used for different functions. For example, a first type of photodetector (e.g., an image photodetector) may be used for image acquisition, and a second type of photodetector (e.g., a focusing photodetector) may be used for focusing, such as in phase detection autofocus (PDAF). The readout of all, or at least a subset (e.g., at least a threshold amount) of certain types of photodetectors in an image sensor (e.g., photodetectors for image acquisition rather than focusing) is sometimes referred to as a tight readout. The readout may include analog gain, analog-to-digital conversion (ADC), digital gain, demosaicing, and / or other operations. An imaging system can, at least partially, use first digital pixel data to generate an image of a scene at a first resolution by combining the first digital pixel data with an image of a first resolution. In some examples, the first resolution may differ from the full resolution possible by the image sensor (e.g., it may be less than the full resolution). In some examples, the imaging system may reset the photodetectors of the image sensor after capturing an image at a first resolution, for example, before using the image sensor to capture a subsequent image. In some cases, the imaging system may decide that an image of a second resolution (e.g., higher than the first resolution) is desirable. The imaging system may read out second digital pixel data and combine it with the first digital pixel data to generate an image of the second resolution.

[0006]

[0006] In one example, an imaging apparatus is provided. The apparatus includes a memory and one or more processors coupled to the memory (for example, implemented in a circuit). The one or more processors are configured to cause an image sensor to expose the image sensor to light from a scene, the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the photodetectors, the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the photodetectors, and to read first digital pixel data from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group without reading a second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, and to generate an image of the scene using at least the first digital pixel data.

[0007]

[0007] In another example, a method for imaging is provided. This method includes exposing an image sensor to light from a scene, wherein the image sensor has a plurality of photodetectors, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other; causing the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors; reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors; and generating an image of a scene using at least the first digital pixel data.

[0008]

[0008] In another example, a non-temporary computer-readable medium storing instructions is provided, and when an instruction is executed by one or more processors, the one or more processors are configured to cause an image sensor to expose a plurality of photodetectors of the image sensor to light from a scene, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, to light from a scene, cause the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, cause the image sensor to read first digital pixel data from a first subset of the plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second group of photodetectors, and cause the image sensor to generate an image of a scene using at least the first digital pixel data.

[0009]

[0009] In another example, an apparatus for image processing is provided. The apparatus includes an image sensor, a plurality of photodetectors of the image sensor, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors comprises at least a first group of photodetectors and a second group of photodetectors that are different from each other, means for causing the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, means for reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors, and means for generating an image of a scene using at least the first digital pixel data.

[0010]

[0010] In some embodiments, in order to read first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group without reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, one or more processors use an imaging circuit to selectively apply modifications to the first subset of multiple analog photodetector signals without applying modifications to the second subset of multiple analog photodetector signals.

[0011]

[0011] In some embodiments, one of the multiple analog photodetector signals indicates a voltage corresponding to a charge from one of the multiple photodetectors.

[0012]

[0012] In some embodiments, the methods, apparatus, and computer-readable media described above further include an image sensor.

[0013]

[0013] In some embodiments, in order to read first digital pixel data from a first subset of a plurality of analog photodetector signals, one or more processors are configured to have one or more amplifiers amplify each of the first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors into a plurality of amplified analog photodetector signals using one or more amplifiers, and the first digital pixel data is based on the plurality of amplified analog photodetector signals. In some embodiments, the methods, apparatus, and computer-readable media described above further comprise one or more amplifiers.

[0014]

[0014] In some embodiments, in order to read first digital pixel data from a first subset of a plurality of analog photodetector signals, one or more processors are configured to have one or more analog-to-digital converters (ADCs) convert a plurality of analog signals based on a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors into digital signals using one or more ADCs, and the first digital pixel data is based on the digital signals. In some embodiments, the methods, apparatus, and computer-readable media described above further comprise one or more ADCs. In some embodiments, one or more processors are configured to generate first digital pixel data based on the digital signals by processing the digital signals at least partially using one or more image processing operations, the one or more image processing operations include at least one of digital gain, demosaicing, pixel interpolation, missing pixel correction, bad pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, and upsampling. In some embodiments, the plurality of analog signals include a plurality of amplified analog photodetector signals, and one or more amplifiers generate the plurality of amplified analog photodetector signals at least partially by amplifying a first subset of the plurality of analog photodetector signals corresponding to a first group of photodetectors.

[0015]

[0015] In some embodiments, one or more processors are configured to, after generating an image, decide whether to read a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors, based on the image.

[0016]

[0016] In some embodiments, one or more processors are configured to read first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first photodetector group, then read second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group, and to use at least the first digital pixel data and the second digital pixel data to generate a second image of the scene, wherein the second resolution of the second image is higher than the first resolution of the image, and the reading of the second digital pixel data is based on the storage of the plurality of analog photodetector signals. In some embodiments, in order to read first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first photodetector group, then read second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group, one or more processors are configured to generate an image of the scene and then read second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group. In some embodiments, one or more processors are configured to identify that an image contains areas having splendor exceeding a splendor threshold, and in response to identifying that an image contains areas having splendor exceeding a splendor threshold, they are configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. In some embodiments, one or more processors are configured to identify that an image shows motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, and in response to identifying that an image shows motion of the scene captured by the image sensor relative to a previous image and exceeds a motion threshold, they are configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors.

[0017]

[0017] In some embodiments, one or more processors are configured to determine, after generating a second image, whether to read third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third group of photodetectors based on the second image, wherein the multiple photodetectors include a third group of photodetectors different from the first and second groups of photodetectors, and the image and the second image are generated without third digital pixel data. In some embodiments, one or more processors are configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, then read third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third photodetector group, and use at least first digital pixel data, second digital pixel data, and third digital pixel data to generate a third image of a scene, wherein the third resolution of the third image is higher than the second resolution of the second image, and the reading of the third digital pixel data is based on the storage of multiple analog photodetector signals, and the multiple photodetectors include a third photodetector group different from the first and second photodetector groups, and the image and the second image are generated without third digital pixel data.

[0018]

[0018] In some embodiments, one or more processors are configured to identify that an image contains a depiction of an object, and in response to identifying that an image contains a depiction of an object, are configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. In some embodiments, the object is a glyph that optically encodes information, and the glyph includes at least one of a one-dimensional barcode and a two-dimensional code. In some embodiments, the object includes at least a part of a person. In some embodiments, the object includes at least a part of a vehicle. In some embodiments, the object includes one or more alphanumeric characters.

[0019]

[0019] In some embodiments, the multiple photodetectors of the image sensor are arranged according to a grid of blocks, each of which comprises a separate subset of the multiple photodetectors, and each block of the grid of blocks comprises at least one photodetector from a first group of photodetectors and at least one photodetector from a second group of photodetectors. In some embodiments, the first group of photodetectors is arranged across the image sensor according to a first pattern, and the second group of photodetectors is arranged across the image sensor according to a second pattern, and the first and second patterns correspond to one or more grids.

[0020]

[0020] In some embodiments, the image sensor includes a storage array having a plurality of storage units, and the image sensor is configured to store each of the plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors in one of the plurality of storage units of the storage array.

[0021]

[0021] In some embodiments, the image sensor is configured to reset the multiple photodetectors and multiple storage units of the memory array of the image sensor after capturing an image. In some embodiments, one or more processors are configured to analyze an image using an object detection algorithm, and in order to reset the multiple photodetectors and multiple storage units of the memory array of the image sensor after capturing an image, the image sensor is configured to automatically reset the multiple photodetectors and multiple storage units of the memory array of the image sensor in response to the object detection algorithm not detecting one or more objects in the image. In some embodiments, the image sensor is configured to reset each of the multiple photodetectors of the image sensor before exposing the multiple photodetectors to light from the scene.

[0022]

[0022] In some embodiments, the image sensor is configured to move a reset pointer from a first side of the image sensor to a second side of the image sensor at a predetermined speed across a plurality of reset pointer positions, and to reset each of the plurality of photodetectors in response to the reset pointer reaching one of a plurality of reset pointer positions where the first photodetector of the plurality of photodetectors is located, and to move a read pointer from a first side of the image sensor to a second side of the image sensor at a predetermined speed across a plurality of read pointer positions, and to read a third digital pixel data corresponding to the second photodetector of the first photodetector group in response to the read pointer reaching one of a plurality of reset pointer positions where the second photodetector of the first photodetector group is located, in order to read a first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first photodetector group, and a predetermined measure is associated with a rolling shutter.

[0023]

[0023] In some embodiments, one or more processors are configured to output an image. In some embodiments, the methods, apparatus and computer-readable media described above further include a display, and in order to output an image, one or more processors are configured to use the display to show an image. In some embodiments, the methods, apparatus and computer-readable media described above further include a communication transceiver, and in order to output an image, one or more processors are configured to use the communication transceiver to transmit an image to a receiving device.

[0024]

[0024] In some embodiments, the multiple photodetectors include multiple photodiodes. In some embodiments, the multiple photodetectors of the image sensor include multiple image photodiodes and multiple focusing photodiodes, wherein the multiple focusing photodiodes are configured for phase detection autofocus (PDAF), the first group of photodetectors includes a first subset of multiple image photodiodes, and the second group of photodetectors includes a second subset of multiple image photodiodes.

[0025]

[0025] In some embodiments, the apparatus is a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a vehicle computing device or component, another device, or a combination thereof, a part thereof, and / or includes them. In some embodiments, the apparatus includes one or more cameras for capturing one or more images. In some embodiments, the apparatus further includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the apparatus described above may include one or more inertial measurement units (IMUs), such as one or more sensors (e.g., one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0026]

[0026] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter. The present subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and the appropriate parts of each claim.

[0027]

[0027] The foregoing, together with other features and embodiments, will become more apparent when referring to the following specification, claims, and accompanying drawings.

Brief Description of the Drawings

[0028]

[0028] Exemplary embodiments of the present application are described in detail below with reference to the following drawings. [Figure 1]

[0029] is a block diagram illustrating an exemplary architecture of an image capture and processing system, in accordance with some examples. [Figure 2]

[0030] is a block diagram illustrating an exemplary architecture of an imaging system that implements a process for image capture, at least in part, by storing analog photodetector signals and reading out digital pixel data from the photodetector, in accordance with some examples. [Figure 3A]

[0031] is a conceptual diagram illustrating an example of an image sensor including a plurality of pixels labeled P0 to P63, in accordance with some examples. [Figure 3B]

[0032] is a conceptual diagram illustrating an example of a storage array that temporarily stores pixel data corresponding to each of the pixels of the image sensor of FIG.3A, in accordance with some examples. [Figure 4A]

[0033] is a conceptual diagram illustrating capture of image data via an image sensor of an imaging system using a rolling shutter, in accordance with some examples. [Figure 4B]

[0034] is a conceptual diagram illustrating capture of image data via an image sensor of an imaging system using a global shutter, in accordance with some examples. [Figure 5A]

[0035] This is a conceptual diagram illustrating, with several examples, the organization of an image sensor's pixel array into a grid of 24 blocks, where each block contains one pixel from each group, and the organization of each pixel in the pixel array into one of 16 groups. [Figure 5B]

[0036] This is a conceptual diagram illustrating the capture of an image at a first resolution by performing sparse readout of an image sensor based on reading only pixels within the first group of 16 groups, as shown in several examples. [Figure 5C]

[0037] This is a conceptual diagram illustrating the capture of low-resolution images by performing sparse readout of an image sensor based on reading only pixels within the second group out of 16 groups, using several examples. [Figure 6A]

[0038] This is a conceptual diagram illustrating the capture of an image of a first resolution, a first image of a second resolution, and / or a second image of a second resolution by performing sparse readout of an image sensor having a pixel array organized into a grid of 4 blocks, where each pixel of the pixel array is organized into one of 13 groups, and each block contains 16 pixels, each containing at least one pixel from each group. [Figure 6B]

[0039] This is a conceptual diagram illustrating the capture of an image of first resolution and / or a first image of second resolution by performing sparse readout of an image sensor having a pixel array organized into a grid of four blocks, where each pixel of the pixel array is organized into one of six groups, and each block contains nine pixels, each containing at least one pixel from each group. [Figure 7]

[0040] This is a conceptual diagram illustrating the capture of a first image at a first resolution, a second image at a first resolution, an image at a second resolution, and a third image at a first resolution, using several examples. [Figure 8]

[0041] This is a conceptual diagram illustrating the movement of the reset pointer and the read pointer along the image sensor in a rolling shutter imaging system, using several examples. [Figure 9]

[0042] This is a conceptual diagram illustrating a rolling shutter imaging system that performs first, second, and third image acquisition, using several examples. [Figure 10A]

[0043] This is a conceptual diagram illustrating the image capture of six image frames, each with a first resolution, via a first readout pattern, using a rolling shutter imaging system with four pixel groups, as shown in several examples. [Figure 10B]

[0044] This is a conceptual diagram illustrating the image capture of eight image frames, each with a second resolution, via a second readout pattern, using a rolling shutter imaging system with four pixel groups, as shown in several examples. [Figure 10C]

[0045] This is a conceptual diagram illustrating the transition from capturing an image frame of a second resolution via a second readout pattern to capturing an image frame of a first resolution via a first readout pattern, using a rolling shutter imaging system with four pixel groups, with several examples. [Figure 11]

[0046] This is a conceptual diagram showing rolling shutter readout patterns using a rolling shutter imaging system, with several examples. [Figure 12]

[0047] This is a flowchart illustrating the operation for imaging, using several examples. [Figure 13]

[0048] This figure shows an example of a computing system for implementing some of the embodiments described herein. [Modes for carrying out the invention]

[0029]

[0049] Specific aspects and embodiments of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some may be applied in combination. The following description includes specific details for illustrative purposes to provide a complete understanding of the embodiments of this application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

[0030]

[0050] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the Disclosure. Rather, the following description of exemplary embodiments provides a description that enables the implementation of exemplary embodiments for those skilled in the art. It should be understood that various modifications may be made to the function and arrangement of the elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0031]

[0051] A camera is a device that uses an image sensor to accept light and capture image frames, such as still images or video frames. The terms “image,” “image frame,” and “frame” are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, F / stop, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to the image sensor for capturing one or more image frames. Other camera settings, such as contrast, brightness, saturation, sharpness, levels, curves, or color changes, may constitute post-processing of one or more image frames. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by the image sensor.

[0032]

[0052] An image sensor includes an array of photodetectors, such as photodiodes. Each photodetector receives light from the scene and converts the light into an electric charge based on how much light reaches the photodetector. Photodetector data from a given photodetector can be processed (e.g., via analog gain, analog-to-digital conversion, digital gain, and / or demosaicing) to provide pixel data for pixels in an image captured by the image sensor. In the case of a color image sensor, different photodetectors can provide pixel data corresponding to different color components (e.g., red, green, or blue) for a given pixel. Reading image data from an image sensor may involve operations such as amplification of the charge generated by the photodetectors (e.g., analog gain), conversion of the analog signal associated with the charge to a digital signal using an analog-to-digital converter (ADC) (e.g., digitization), demosaicing, digital gain, missing pixel correction, faulty pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, upsampling, additional image processing operations, or combinations thereof. Performing a readout for each photodetector of an image sensor can have a significant impact on the imaging device in terms of power consumption, bandwidth usage, heat generated, and / or the use of computing resources (e.g., for ISP operation and / or image processing operation). Resetting a photodetector refers to clearing the charge of the photodetector before it accepts and detects light for capturing a new image. Resetting an image sensor may refer to resetting at least a subset of the photodetectors of the image sensor. Resetting an image sensor may refer to resetting all of the photodetectors of the image sensor.

[0033]

[0053] Portable electronic devices are generally powered by a power storage unit, such as a battery, which is part of the portable electronic device itself. To maintain lightness and small size, and therefore portability, portable electronic devices generally contain relatively small batteries. Because portable electronic devices have limited battery life, it is beneficial to reduce the power consumption of the operations performed by the portable electronic device as much as possible. Therefore, some portable electronic devices also have less processing power than devices connected to a power outlet in order to reduce power consumption and increase battery life. Some portable electronic devices include cameras. Images of different resolutions may be useful for different use cases. In some use cases, images with a first resolution may be sufficient. In other use cases, images with a second resolution (e.g., higher than the first resolution) may be beneficial. In some examples, reading image data of the first resolution may consume less battery power than reading image data of the second resolution, or vice versa.

[0034]

[0054] Electronic devices that draw power from a power source can generate heat from components that use power to perform their activities, and from those components that use power to perform their activities. Electronic devices may include heat dissipation components that help dissipate heat, such as heat sinks, fans, liquid coolers, tubes for transporting fluids (e.g., liquids and / or gases), and channels for transporting fluids (e.g., liquids and / or gases). To maintain lightness and small size, and therefore portability, portable electronic devices generally contain little space for heat dissipation components. Therefore, many portable electronic devices contain little to no heat dissipation components. Portable electronic devices that include heat dissipation components are generally limited in terms of the type and size of heat dissipation components that a portable electronic device may contain, and this can also limit the effectiveness of the heat dissipation components used in such portable electronic devices. If the heat of components in an electronic device becomes too high (e.g., exceeds a threshold temperature or thermal level), the electronic device may be modified and / or become defective. For example, at high thermal levels, certain components or connections between components may melt, deform, transform, change state, become brittle, change shape, break, or, in some cases, be modified. In addition, portable electronic devices are often held in the user's hand, worn on the user's body (e.g., as a wearable device), or stored in the user's pocket adjacent to the user's body. If the heat of components in a portable electronic device becomes too high (e.g., exceeding a threshold temperature or thermal level), the portable electronic device may burn, injure, and / or cause discomfort to the user. Since some portable electronic devices may have limited heat dissipation capabilities, it is beneficial to reduce the heat generated by the operations performed by the portable electronic device whenever possible.Some portable electronic devices have less processing power than devices connected to a power outlet in order to reduce heat generation, reduce heat dissipation required to maintain a threshold temperature or threshold thermal level, reduce the amount of heat dissipated by any heat dissipation component to maintain a threshold temperature or threshold thermal level, or a combination thereof. Some portable electronic devices include cameras. Images of different resolutions may be useful for different use cases. In some use cases, images with a first resolution may suffice. In other use cases, images with a second resolution (e.g., higher than the first resolution) may be beneficial. In some examples, reading image data of a first resolution may generate less heat (and thus reduce heat dissipation required to maintain a threshold temperature or threshold thermal level) than reading image data of a second resolution, or vice versa.

[0035]

[0055] A rolling shutter camera can capture each frame of a still image or video by rapidly scanning across a scene from one side of the image sensor to the other. Typically, a rolling shutter camera scans the scene vertically, for example, from the top of the image sensor to the bottom of the image sensor. In some cases, a rolling shutter may scan the scene horizontally instead. In some examples, a rolling shutter is used to record different parts of an image of a scene at different moments in time. In some examples, a rolling shutter is not used to record all parts of an image of a scene at exactly the same moment. Images captured using a rolling shutter can depict different parts of the scene as they appeared at slightly different times (e.g., moments) in the scene. A rolling shutter camera can reset its photodetectors line by line (e.g., row by row) from one side of the image sensor to the other. After resetting those photodetectors, the rolling shutter can read pixel data from its photodetectors line by line (e.g., row by row) from one side of the image sensor to the other.

[0036]

[0056] A global shutter camera can be used to capture each frame of a still image or video by simultaneously scanning the entire area of ​​the image. A global shutter camera may include a storage array of storage units, such as memory diodes, that store photodetector data from each photodetector of the image sensor, and therefore for each pixel of the full-resolution image that the image sensor can produce. The global shutter camera can stop exposure at the photodetector array, allowing the accumulated charge to move from the photodetector array to the storage array. The storage units of the storage array may be reset after a period of time, for example, immediately before another image is captured by the global shutter camera. In some examples, the storage array can store the photodetector data as analog photodetector data (e.g., charge at each individual photodetector before or after amplification).

[0037]

[0057] This specification describes systems and techniques for controlling whether a camera performs a first readout from a first subset of pixels of its image sensor to produce an image of a first resolution with first power consumption, or whether a camera performs a second readout from at least a second subset of pixels of its image sensor to produce an image of a second resolution with second power consumption. The imaging system may include an image sensor having an array of photodetectors such as photodiodes. The photodetectors in the photodetector array of the image sensor may be grouped such that each photodetector is one of a set of several distinct groups. For example, some of the photodetectors may be in a first group, some of the photodetectors may be in a second group distinct from the first group, and so on. The photodetectors in the photodetector array may be grouped into different groups based on a predetermined pattern. For example, the photodetectors in the photodetector array may be grouped into a grid of blocks, each having predetermined dimensions (e.g., width n photodetectors × height m photodetectors), each containing one or more photodetectors from each group, and possibly arranged in a predetermined order within the blocks. In this example, both n and m are integer values ​​greater than or equal to 1, and may be equal to or different from each other. For example, each block of photodetectors may contain a photodetector in the first group, followed by a photodetector in the second group, and so on. The imaging system may use a global shutter or a rolling shutter.

[0038]

[0058] The imaging system can reset the photodetectors in the photodetector array of its image sensor line by line (e.g., rolling shutter), area by area (e.g., global shutter), group by group, or a combination thereof. The imaging system can expose its image sensor and the photodetector array on it to light from a scene. Each photodetector is configured and capable of converting light from a scene into electric charge in response to exposure to light from a scene. The imaging system can receive and / or store analog photodetector signals corresponding to the charges from each photodetector. In some examples, the imaging system uses a global shutter and can store each analog photodetector signal in one of the storage units of a storage array. In some examples, the imaging system uses a rolling shutter and can store each analog photodetector signal in the photodetector until read out from the photodetector.

[0039]

[0059] The imaging system can read first digital pixel data from a first subset of the analog photodetector signal corresponding to a first photodetector group without reading second digital pixel data from a second subset of the analog photodetector signal corresponding to a second photodetector group (and / or without reading additional digital pixel data from additional subsets of the analog photodetector signal corresponding to additional photodetector groups such as a third or fourth photodetector group). When an imaging device reads only digital pixel data from a subset of its photodetector group, this is sometimes called sparse readout. Sparse readout is also sometimes called a low-power mode. Readout may involve amplification of the analog photodetector signal (e.g., analog gain). Readout may involve conversion of the analog photodetector signal to a digital signal (e.g., via an ADC). In some examples, the digital signal from the ADC is the first digital pixel data. In some examples, the digital signal from the ADC is further processed through one or more image processing operations (e.g., digital gain, demosaicing, pixel interpolation, missing pixel correction, faulty pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, and upsampling) to obtain first digital pixel data. The imaging system can capture a low-resolution image of the scene, at least partially, by combining the first digital pixel data with the low-resolution image. In some examples, the imaging system can reset the photodetector of the image sensor after capturing the low-resolution image, for example, before capturing a subsequent image.

[0040]

[0060] In some examples, the imaging system may determine that a high-resolution image is desirable based on, for example, a user interface input by the user, the imaging device determining that a low-resolution image contains depictions of objects such as barcodes or quick response (QR) codes which may be easier to scan and / or interpret in a high-resolution image; the low-resolution image contains depictions of objects such as human faces which may be easier to recognize using a high-resolution image; the low-resolution image contains depictions of alphanumeric characters which may be easier to analyze using a high-resolution image; the imaging device determining that a low-resolution image contains image areas which have a splendor exceeding a splendor threshold (e.g., having at least a threshold size); the imaging device determining that a low-resolution image contains movement (e.g., exceeding a movement threshold) compared to a previous image; or a combination thereof. After reading the first digital pixel data, the imaging device may read second digital pixel data from a second subset of analog photodetector signals corresponding to a second group of photodetectors from a stored analog photodetector signal. An imaging device can capture a high-resolution image of a scene, at least partially, by combining both the first and second digital pixel data into a high-resolution image. A high-resolution image has a higher resolution than a low-resolution image. When an imaging device reads digital pixel data from more of its photodetectors than in sparse readout, it is sometimes called tight readout. When an imaging device reads digital pixel data from all of its photodetectors, it is sometimes called full readout. Tight readout can include full readout. Tight readout is sometimes called high-power mode.

[0041]

[0061] In some cases, the imaging device can read a second set of digital pixel data from a second subset of analog photodetector signals corresponding to a second group of photodetectors from the stored analog photodetector signals, before the photodetector and / or the memory unit of the memory array is reset. Therefore, the imaging device can determine on the fly to switch from low-resolution image readout to high-resolution image readout, even before the acquisition of the next frame begins. This is a technical improvement over conventional cameras where setting changes are generally not applied until several frames after the change request is made. The imaging device offers technical improvements such as reduced power consumption, reduced bandwidth usage, and / or reduced computational resource usage. These improvements are achieved by using sparse readout when low-resolution images are sufficient and using dense readout (e.g., full readout) when requested (e.g., by the user or by the imaging device itself) for operations where high-resolution images are beneficial. In some cases, sparse readout can be used in cameras that are left on for extended periods to record, such as always-on (AON) cameras, and / or in continuous imaging applications to reduce power consumption. The imaging system can switch from sparse to dense readout as needed (e.g., by the user or by the imaging device itself) for operations where high-resolution images are beneficial.

[0042]

[0062] Various aspects of application examples will be described with reference to the figures. Figure 1 is a block diagram showing the architecture of the image capture and processing system 100. The image capture and processing system 100 includes various components used to capture and process images of a scene (e.g., images of scene 110). The image capture and processing system 100 can capture standalone images (or photographs) and / or capture videos containing multiple images (or video frames) in a specific order. The lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0043]

[0063] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or information from the image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components. For example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may include additional control mechanisms other than those shown, such as control mechanisms for analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0044]

[0064] The focus control mechanism 125B of the control mechanism 120 can acquire the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 by acting a motor or servo. In some cases, additional lenses, such as one or more microlenses on each photodetector of the image sensor 130 (e.g., photodetectors), may be included in the system 100, each of which bends the light received from the lens 115 toward the corresponding photodetector before the light reaches the photodetector. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or any combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as the image acquisition setting and / or image processing setting.

[0045]

[0065] The exposure control mechanism 125A of the control mechanism 120 can acquire exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on these exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or F / stop), the duration for which the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. Exposure settings are sometimes referred to as image capture settings and / or image processing settings.

[0046]

[0066] The zoom control mechanism 125C of the control mechanism 120 can acquire the zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by operating one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting is sometimes called the image acquisition setting and / or image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be lens 115) that first receives light from the scene 110, and then the light passes through an infinite focus zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, an infinite focus zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference) together with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the infinite focus zoom system, such as the negative lens and one or both of the positive lenses.

[0047]

[0067] The image sensor 130 includes an array of one or more photodetectors, such as photodiodes or other photosensitive elements. Each photodetector measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodetectors may be covered by different color filters, and thus may measure light that matches the color of the filter covering the photodetector. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, and each pixel in the image is produced based on red light data from at least one photodetector covered within the red filter, blue light data from at least one photodetector covered within the blue filter, and green light data from at least one photodetector covered within the green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of, or in addition to, the red, blue, and / or green filters. Some image sensors may have no color filters at all and instead use different photodetectors across the entire pixel array (potentially stacked vertically). Different photodetectors across the entire pixel array may have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore color depth.

[0048]

[0068] In some cases, the image sensor 130 may include, as an alternative or addition, an opaque and / or reflective mask that prevents light from reaching some photodetectors or parts of some photodetectors at a given time and / or from certain angles, which may be used for phase-detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodetectors, and / or an analog-to-digital converter (ADC) 132 for converting the analog signal output of the photodetectors (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, instead or in addition, some components or functions discussed with respect to one or more of the control mechanisms 120 may be included within the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an n-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or any other combination thereof.

[0049]

[0069] In some examples, the array of photodetectors of the image sensor 130 may include a focal photodetector and an image photodetector. Both the image photodetector and the focal photodetector can accept light from the scene being photographed. Photodiode data from the image photodetector may be used to capture an image of the scene, as described herein. Photodiode data from the focal photodetector may be used for PDAF. Both the image photodetector and the focal photodetector may be located beneath a microlens. In some examples, a microlens on a focal photodiode may be partially masked with an opacity mask or a reflective mask such that one side of the microlens allows light from the scene to pass through to the focal photodiode, while the other side of the microlens prevents light from the scene from reaching the focal photodiode. This masking may restrict the light reaching the focal photodiode to light approaching the focal photodiode from a specific angular range that may be useful for PDAF. In some examples, a microlens may be split between two or more focal photodiodes such that the light reaching each focal photodiode is restricted to light passing through a specific portion of the microlens, and therefore to light approaching the photodiode from a specific angular range that may be useful for PDAF. In some examples, image data from the focal photodiode of the image sensor 130 may be discarded by the image acquisition and processing system 100. The ISP 154, host processor 152, and / or image processor 150 may treat the pixel data from the focal photodiode as missing pixels and / or bad pixels that the ISP 154, host processor 152, and / or image processor 150 can correct using missing pixel correction, bad pixel correction, and / or interpolation. In some examples, image data from the focal photodiode of the image sensor 130 may be used by the image acquisition and processing system 100 when generating an acquired image, for example, with some corrections based on adjacent pixels (for example, using missing pixel correction, bad pixel correction, and / or interpolation). Pixels corresponding to image photodiodes are sometimes called image pixels. Pixels corresponding to focal photodiodes are sometimes called focal pixels.

[0050]

[0070] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1310 discussed in relation to the computing system 1300. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., called a system-on-a-chip or SoC) including the host processor 152 and ISP 154. In some cases, the chip may include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPU), graphics processing units (GPU), broadband modems (e.g., 3G, 4G, or LTE®, 5G, etc.), memory, connectivity components (e.g., Bluetooth®, Global Positioning System, GPS, etc.), any combination thereof, and / or other components.The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (e.g., a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one exemplary example, the host processor 152 can communicate with the image sensor 130 using the I2C port, and the ISP 154 can communicate with the image sensor 130 using the MIPI port.

[0051]

[0071] In some examples, the image acquisition and processing system 100 may include a storage array 135, which includes an array of storage units. In some examples, the storage units may be storage diodes, memory diodes, memory units, or a combination thereof. The storage array 135 can store photodetector data from the photodetector array of the image sensor 130 before the photodetector data is converted to digital by the ADC 132. In some examples, each storage unit of the storage array 135 stores analog photodiode data (e.g., charge or a signal indicating charge) from one of the photodetectors in the photodetector array of the image sensor 130, for example, as shown with respect to the image sensor 310 in Figure 3A and the corresponding storage array 320 in Figure 3B. The storage array 135 can provide the stored analog photodiode data to the ADC 132 and / or image processor 150 (e.g., ISP 154 and / or host processor 152) for readout, for example, as shown with respect to the imaging system in Figure 2.

[0052]

[0072] The image processor 150 may perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, integration of image frames to form an HDR image, image recognition, object recognition, feature recognition, input acceptance, output management, memory management, or any combination thereof. The image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 and / or 1320, read-only memory (ROM) 145 and / or 1325, a cache, a storage unit, another storage device, or any combination thereof.

[0053]

[0073] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, trackpad, touch-sensitive screen, printer, any other output device 1335, any other input device 1345, or any combination thereof. In some cases, captions may be input to the image processing device 105B through the physical keyboard or keypad of the I / O device 160, or through the virtual keyboard or keypad of the touchscreen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or send data to one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable wireless connectivity between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the previously described types of I / O devices 160, and when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 160.

[0054]

[0074] In some cases, the image acquisition and processing system 100 may be a single device. In some cases, the image acquisition and processing system 100 may be two or more separate devices, including an image acquisition device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image acquisition device 105A and the image processing device 105B may be coupled wirelessly, for example, via one or more wires, cables, or other electrical connectors and / or via one or more wireless transceivers. In some implementations, the image acquisition device 105A and the image processing device 105B may be isolated from each other.

[0055]

[0075] As shown in Figure 1, the vertical dashed line divides the image acquisition and processing system 100 in Figure 1 into two parts, representing the image acquisition device 105A and the image processing device 105B, respectively. The image acquisition device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components shown in the image acquisition device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image acquisition device 105A.

[0056]

[0076] The image capture and processing system 100 may include electronic devices such as mobile or fixed telephone handsets (e.g., smartphones, mobile phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device such as a mobile handset, desktop computer, or other computing device.

[0057]

[0077] While the image capture and processing system 100 is shown as comprising several components, those skilled in the art will understand that the image capture and processing system 100 may comprise more components than those shown in Figure 1. The components of the image capture and processing system 100 may comprise one or more combinations of software, hardware, or software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may comprise one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), electronic circuits, or other electronic hardware, and / or may be implemented using them, and / or may comprise computer software, firmware, or any combination thereof, to perform the various operations described herein, and / or may be implemented using them. The software and / or firmware may comprise one or more instructions, which are stored on a computer-readable storage medium and are executable by one or more processors of the electronic devices implementing the image capture and processing system 100.

[0058]

[0078] Figure 2 is a block diagram illustrating an exemplary architecture of an imaging system that performs a process 200 for image acquisition, at least partially, by storing an analog photodetector signal 250 and reading digital pixel data 255 from a photodetector 215. The image acquisition system includes a photodetector 215, such as a photodiode or phototransistor. The photodetector 215 may be a photodetector in a photodetector array of an image sensor, such as an image sensor 130 or an image sensor 310. The photodetector 215 can receive photons of light 205 from a scene 210. Scene 110 may be an example of scene 210. The light 205 that reaches the photodetector 215 may be called incident light. The photodetector 215 can convert the light 205 into an electric charge. The charge can form a voltage 225 and flow as an analog photodetector signal (e.g., as an electric current). In some examples, filters 220 (e.g., low-pass filters, high-pass filters, band-pass filters) are formed using a capacitor to ground, as shown in the imaging system of Figure 2, for example, and can filter out noise from the analog photodetector signal from the photodetector 215. In some examples, filters 220 may be omitted. The analog photodetector signal can flow to an amplifier 230, which can amplify the analog photodetector signal and thus provide analog gain to the analog photodetector signal. The amplified analog photodetector signal can then flow from the amplifier 230 to an analog-to-digital converter (ADC) 235. In some examples (not shown), another filter (similar to, but separate from, filter 220) can filter the analog photodetector signal on its way from the amplifier 230 to the ADC 235. The ADC 235 may be an example of the ADC 132. The ADC 235 converts the analog photodetector signal into digital pixel data. In some examples, the image processor 240 may process and / or modify the digital pixel data using one or more image processing operations, depending on the circumstances, based on other digital pixel data from the ADC 235 (but from the same image sensor) corresponding to a different photodiode than the photodiode 215.For example, the image processor 240 can process and / or modify digital pixel data at least partially by adjusting digital gain, demosaicing, pixel interpolation, performing missing pixel correction, performing defective pixel correction, adjusting brightness, adjusting contrast, adjusting saturation, adjusting histogram, performing color space conversion, performing automatic white balance (AWB), performing automatic black balance (ABB), downsampling, upsampling, or a combination thereof. The image processor 240 may be an example of an ISP 154, a host processor 152, an image processor 150, an image processing device 105B, or a combination thereof.

[0059]

[0079] The term digital pixel data 255 may refer to digital pixel data generated by the ADC 235 that has not yet been processed by the image processor 240, or digital pixel data generated by the ADC 235 that is also processed and / or modified by the image processor 240. The digital pixel data 255 may be combined with digital pixel data corresponding to other photodiodes and / or pixels of the image sensor in order to perform image acquisition 270 and capture an image.

[0060]

[0080] The analog photodetector signal 250 may refer to an analog photodetector signal output by the photodetector 215 but not yet filtered by the filter 220 or not yet amplified by the amplifier 230, an analog photodetector signal filtered by the filter 220 but not yet amplified by the amplifier 230, an analog photodetector signal amplified by the amplifier 230 and then filtered by a second filter (not shown), or a combination thereof. The analog photodetector signal 250 may be stored in a storage unit 260 of a storage array 265. The storage array 265 may be an example of the storage array 135 and / or the storage array 320. The storage unit 260 may be an example of one of the storage units S0 to S63 in Figure 3B. The storage unit 260 may be a storage diode, a memory diode, a memory unit, or a combination thereof.

[0061]

[0081] Figure 3A is a conceptual diagram showing an example of an image sensor 310 containing multiple pixels labeled P0 to P63. The image sensor 310 has a width of 9 pixels and a height of 9 pixels. The pixels are numbered sequentially from P0 to P63, starting from left to right within each row, and counting upwards from the top row to the bottom row. The image sensor 310 may be an example of the image sensor 130. Each of the individual pixels (P0 to P63) of the image sensor 310 may contain one or more photodetectors, such as the photodetector 215. In some examples, each of the elements labeled P0 to P63 may be an individual photodetector, such as the photodetector 215. In some examples, all pixels P0 to P63 in Figure 3A are image pixels. In some examples, all photodetectors P0 to P63 in Figure 3A are image photodetectors. In some examples, pixels P0 to P63 in Figure 3A include both image pixels and focal pixels. In some examples, the photodetectors P0 to P63 in Figure 3A include an image photodetector and a focal photodetector.

[0062]

[0082] Figure 3B is a conceptual diagram showing an example of a storage array 320 that temporarily stores pixel data corresponding to each pixel of the image sensor 310 shown in Figure 3A. The storage array 320 includes multiple storage units labeled S0 to S63. Like the image sensor 310, the storage array 320 has a width of 9 cells (pixels) and a height of 9 cells (pixels). The cells are numbered sequentially from S0 to S63, starting from left to right within each row, and counting upwards from the top row to the bottom row.

[0063]

[0083] Each pixel data set within each cell of the memory array 320 corresponds to a pixel in the image sensor 310. For example, the pixel data set stored in cell S0 of the memory array 320 corresponds to pixel P0 in the image sensor 310, the pixel data set stored in cell S1 of the memory array 320 corresponds to pixel P1 in the image sensor 310, and so on. The pixel data set in the memory array 320 may represent one or more analog pixel signals and / or one or more analog photodetector signals corresponding to one or more photodetectors (e.g., photodiodes) of the image sensor that represent the pixel data set for that pixel. In some examples, each pixel Pa of pixels P0 to P63 of the image sensor 310 contains a set of b photodetectors, and the corresponding cell Ma of cells S0 to S63 of the memory array 320 corresponds to the pixel data from the set of b photodetectors and contains a set of b memory diodes that store it. In this example, a can be any integer from 0 to 63 (including both extreme values), and b can be any integer greater than or equal to 1.

[0064]

[0084] Figure 4A is a conceptual diagram 400A illustrating the capture of image data via an image sensor in an imaging system using a rolling shutter. The capture of different parts of the image data by the imaging system is shown with respect to a horizontal time axis 490, where time flows from left to right, and to a vertical row axis 495, which represents different rows of the image sensor. The time axis 490 is sometimes called a timeline. Different rows of the image sensor are shown along different horizontal lines that are perpendicular to the vertical row axis 495 and intersect the vertical row axis 495 at different points in time. Examples of image sensors may include image sensor 310 and / or image sensor 130. An image sensor may include an array of photodetectors and / or pixels arranged in rows. In some examples, the rolling shutter may be part of a control mechanism 120. In some examples, the rolling shutter may be an electronic rolling shutter that electronically controls when specific photodiodes and / or pixels (and / or their rows) are reset, exposed, and / or read out. In some examples, a rolling shutter may be a mechanical rolling shutter that physically blocks the image sensor from time to time and physically reveals the image sensor one row at a time (for example, by using the rolling or rotational motion of the physical shutter).

[0065]

[0085] Each row shown includes a reset block. The length of the reset block along time axis 490 represents the duration of time during which the row is reset by the imaging system. Each row shown includes an exposure block after the reset block. The length of the exposure block along time axis 490 represents the duration of time during which the row (e.g., the photodiode and / or pixels within the row) is exposed to light from the scene by the imaging system. Each row shown includes a readout block after the exposure block. The length of the readout block along time axis 490 represents the duration of time during which image data is read out from the row (e.g., the photodiode and / or pixels within the row) by the imaging system.

[0066]

[0086] Eight rows are shown, including rows 402, 412, 422, 432, 442, 452, 462, and 472. The reset block 404 on row 402 is the first action performed along the time axis 490. The reset block 414 on row 412 begins after the reset block 404 on row 402. The reset block 424 on row 422 begins after the reset block 414 on row 412. This pattern continues, with each consecutive reset block along the row axis 495 beginning after the reset block on the row above it. This pattern applies to reset blocks 404, 414, 424, 434, 444, 454, 464, and 474. Some of the durations corresponding to different reset blocks may overlap.

[0067]

[0087] Similarly, each consecutive exposure block along row axis 495 begins after the exposure block of the row above it. This pattern applies to exposure blocks 406, 416, 426, 436, 446, 456, 466, and 476. Some of the durations corresponding to different exposure blocks may overlap. Similarly, each consecutive readout block along row axis 495 begins after the readout block of the row above it. This pattern applies to readout blocks 408, 418, 428, 438, 448, 458, 468, and 478. Some of the durations corresponding to different readout blocks may overlap. Exposure blocks may include operations related to the integration of data associated with a row. In some examples, reset, integration, and / or readout operations scan across the image sensor in raster scan order.

[0068]

[0088] Figure 4B is a conceptual diagram 400B illustrating the capture of image data via an image sensor in an imaging system using a global shutter. The capture of image data using the imaging system with a global shutter in Figure 4B is shown along the same time axis 490 and row axis 495 as in Figure 4A. The capture of image data using the imaging system with a global shutter in Figure 4B is shown using the same set of 8 rows and reset blocks, exposure blocks and read blocks as in Figure 4A, but arranged in a different way made possible by a storage array. An example of a storage array includes storage array 320.

[0069]

[0089] In Figure 4B, the reset blocks (e.g., reset block 404, reset block 414, reset block 424, reset block 434, reset block 444, reset block 454, reset block 464, and / or reset block 474) can be started approximately simultaneously. In Figure 4B, the reset blocks can be terminated approximately simultaneously. In Figure 4B, the duration along the time axis 490 to which each reset block extends can be approximately coincident, synchronous, and / or simultaneous.

[0070]

[0090] As shown in Figure 4B, reading out multiple rows of an image sensor can amount to a significant portion of the time used to capture image data using the global shutter. Therefore, skipping readouts for specific rows, pixels, and / or photodiodes can result in faster and more efficient capture of image data.

[0071]

[0091] It should be understood that, since the charge from the photodiode can be stored in the memory array for a period of time, imaging systems with a global shutter can further delay post-exposure readouts in time than shown in Figure 4B. In some examples, the readout of a particular row, pixel, and / or photodiode may occur multiple times (e.g., two or more times) after exposure while the charge from the photodiode is still stored in the memory array. For example, in the context of Figure 7, the acquisition of image 735 at a first resolution 727 and the acquisition of image 755 at a second resolution 757 may involve two readouts of a particular row, pixel, and / or photodiode after reset 732 (and / or integration 730) and before reset 742. In some imaging systems with a global shutter, a particular row (e.g., two or more adjacent rows) may be readout simultaneously and / or concurrently.

[0072]

[0092] Figure 5A is a conceptual diagram 500A showing the arrangement of the pixel array of the image sensor 501 into a grid 502 of 24 blocks 504, each containing one pixel from each group, and the arrangement of each pixel of the pixel array into one of 16 groups (505-580). The pixel array of the image sensor 501 is 24 pixels wide × 16 pixels high. Each of the blocks 504 is 4 pixels wide × 4 pixels high. Therefore, the pixel array of the image sensor 501 is 6 blocks wide × 4 blocks high.

[0073]

[0093] Blocks 504 are shown spaced apart with gaps between them to emphasize the organization of grid 502. It should be understood that blocks 504 can be adjacent to each other on the image sensor 501 itself without these gaps. The outline of an exemplary block 503 of block 504 is drawn with a dashed rounded rectangle. Each of the blocks 504 contains a total of 16 pixels. Each of the blocks 504 contains one pixel from each of the 16 groups (505-580). Specifically, each of the blocks 504 includes one pixel from the first group 505, one pixel from the second group 510, one pixel from the third group 515, one pixel from the fourth group 520, one pixel from the fifth group 525, one pixel from the sixth group 530, one pixel from the seventh group 535, one pixel from the eighth group 540, one pixel from the ninth group 545, one pixel from the tenth group 550, one pixel from the eleventh group 555, one pixel from the twelfth group 560, one pixel from the thirteenth group 565, one pixel from the fourteenth group 570, one pixel from the fifteenth group 575, and one pixel from the sixteenth group 580. Each pixel in the pixel array is indicated in Figures 5A to 5C and Figures 6A to 6B by the group to which it belongs. Within each block, the grouping of pixels progresses from the top row downwards, and from left to right within each row. Therefore, the top-left pixel in each block is in the first group 505, and the bottom-right pixel in each block is in the 16th group 580.

[0074]

[0094] Figure 5B is a conceptual diagram 500B showing the acquisition of an image 585 at a first resolution 587 by performing a sparse readout 586 of an image sensor 501 based on reading only pixels in the first group 505 of 16 groups (505-580). Selection 588 is performed to select the first group 505. Selection 588 may be performed based on user interface input received from the user interface. Selection 588 may also be performed automatically by the imaging system. In Figure 5B, all pixels belonging to the first group 505 are highlighted using thick black outlines. The imaging system acquires an image 585 at a first resolution 587 by performing a sparse readout 586 of the pixels belonging to the first group 505 and not reading any other pixels belonging to any of the other groups (510-580), and by combining the pixel data from all the pixels belonging to the first group 505 into the image 585 at a first resolution 587. Sparse readout allows the image sensor 501 to capture an image at a given resolution lower than the full resolution it is capable of, without downscaling or binning, as in the case of sparse readout 586 in Figure 5B or sparse readout 592 in Figure 5C. Pixels in the first group 505 may include image pixels, focal pixels, or a combination thereof. Pixels in the other groups (510-580) may include image pixels, focal pixels, or a combination thereof. The first resolution 587 is lower than the full resolution the image sensor 501 is capable of, assuming the full resolution at which the image sensor 501 can use pixels from all 16 groups (505-580).

[0075]

[0095] Figure 5C is a conceptual diagram 500C showing the acquisition of an image 590 at a first resolution 587 by performing a sparse readout 592 of an image sensor based on reading only pixels in the second group 510 of 16 groups. Selection 594 is performed to select the second group 510. Selection 594 may be performed based on user interface input received from the user interface. Selection 594 may also be performed automatically by the imaging system. All pixels belonging to the second group 510 are highlighted in Figure 5C using thick black outlines. The imaging system acquires an image 590 at a first resolution 587 by performing a sparse readout 592 of pixels belonging to the second group 510 without performing any other pixels belonging to any of the other groups (505 and 515-580), and by combining the pixel data from all pixels belonging to the second group 510 into the image 590 at a first resolution 587. The pixels in the second group 510 may include image pixels, focal pixels, or a combination thereof. The pixels in the other groups (505 and 515-580) may include image pixels, focal pixels, or a combination thereof. The first resolution 587 is lower than the full resolution achievable by the image sensor 501.

[0076]

[0096] In some examples, the imaging system may alternate which group is selected for sparse readout in each subsequent frame within a series of image frames (e.g., for video). For example, the imaging system may perform sparse readout on the first image frame based on the first group 505 (as in Figure 5B), on the second image frame based on the second group 510 (as in Figure 5C), on the third image frame based on the third group 515, and so on. The imaging system may cycle through the groups and eventually perform sparse readout on all groups. In this way, the imaging system can gradually collect a full-resolution view of the scene, at least for the static parts of the scene.

[0077]

[0097] In conceptual diagrams 500A to 500C, all blocks 504 within the grid 502 are uniform in shape and are all rectangular. In some examples, at least some of the blocks 504 may have a different shape, such as a rectangle or another polygon. In some examples, at least some of the blocks 504 may be polyhedra that can form a polyhedral grid. In some examples, at least some of the blocks 504 may be regular polyhedra that can form a regular polyhedral grid. In conceptual diagrams 500A to 500C, all blocks 504 within the grid 502 are uniform in size. In some examples, at least some of the blocks 504 may be of different sizes. The grid 502 is sometimes called a grid.

[0078]

[0098] Figure 6A is a conceptual diagram 600A showing the capture of an image 610 with a first resolution 612, a first image 620 with a second resolution 622, and / or a second image 630 with a second resolution 622 by performing sparse readout of an image sensor 601A, which has a pixel array organized into a grid 602A of four blocks 603A, where each pixel of the pixel array is organized into one of 13 groups, and each block contains 16 pixels, each containing at least one pixel from each group. The pixel array of the image sensor 601A is 8 pixels wide × 8 pixels high. Similar to block 504 in Figures 5A to 5C, each of the blocks 603A is 4 pixels wide × 4 pixels high. Therefore, the pixel array of the image sensor 601A is 2 blocks wide × 2 blocks high.

[0079]

[0099] Each block 603A contains a total of 16 pixels. Each block 603A contains one pixel from each of the 13 groups. The 13 groups include the 16 groups shown in Figures 5A to 5C, without the third group 515, the ninth group 545, and the eleventh group 555. Each block 603A contains four pixels from the first group 505 and one pixel from each of the remaining groups of the 13 groups. Specifically, each block 603A includes four pixels from the first group 505, one pixel from the second group 510, one pixel from the fourth group 520, one pixel from the fifth group 525, one pixel from the sixth group 530, one pixel from the seventh group 535, one pixel from the eighth group 540, one pixel from the tenth group 550, one pixel from the twelfth group 560, one pixel from the thirteenth group 565, one pixel from the fourteenth group 570, one pixel from the fifteenth group 575, and one pixel from the sixteenth group 580.

[0080]

[0100] All 16 pixels belonging to the first group 505 (four per block 603A) are highlighted in Figure 6A using a thick black outline. The imaging system captures the image 610 of first resolution 612 by performing sparse readout based on the selection 615 of the first group 505, without reading out any of the other groups (510, 520, 525, 530, 535, 540, 550, 560, 565, 570, 575, 580), and by combining all the pixels belonging to the first group 505 into the image 610 of first resolution 612.

[0081]

[0101] All four pixels belonging to the sixth group 530 (one per block 603A) are highlighted in Figure 6A using triple black outlines. The imaging system captures the first image 620 of the second resolution 622 by performing a sparse readout based on the selection 625 of the sixth group 530, without reading out any of the other groups (505, 510, 520, 525, 535, 540, 550, 560, 565, 570, 575, 580), and by combining all the pixels belonging to the sixth group 530 into the first image 620 of the second resolution 622.

[0082]

[0102] All four pixels belonging to the 16th group 580 (one for each block 603A) are highlighted in Figure 6A using black dashed outlines. The imaging system captures the second image 630 with a second resolution 622 by performing a sparse readout based on the selection 635 of the 16th group 580, without reading out any of the other groups (505, 510, 520, 525, 530, 535, 540, 550, 560, 565, 570, 575), thereby combining all the pixels belonging to the 16th group 580 into the second image 630 with a second resolution 622. In some examples, the readout of the first group 505 to capture an image 610 with a first resolution 612 may be called a tight readout in relation to the sparse readout of the sixth group 530 to generate a first image 620 with a second resolution 622, and / or the sparse readout of the sixteenth group 580 to generate a second image 630 with a second resolution 622.

[0083]

[0103] In some examples, the first resolution 612 is a higher resolution than the second resolution 622. The terms dense readout and / or sparse readout can be relative. For example, the term dense readout may refer to any readout that is denser than a specified sparse readout. The term sparse readout may refer to any readout that is less dense than a specified dense readout and / or full readout. In some examples, a readout of the first group 505 to capture image 610 at the first resolution 612 may be called a medium-density readout.

[0084]

[0104] Images of a third resolution (not shown) can be captured, for example, by selectively reading out pixel data from multiple groups. In one exemplary example, an image of a third resolution (not shown) can be captured by selectively reading out pixel data from all groups (e.g., 505, 510, 520, 525, 530, 535, 540, 550, 560, 565, 570, 575, and 580). Reading out images of a third resolution is sometimes called high-density reading. Reading out images of a third resolution is sometimes called full-density reading if the third resolution reading involves selectively reading out pixel data from all groups. In this way, an imaging system equipped with the image sensor 601A can provide a number of levels of reading density. Multiple levels of read density may include the sparsest reads (e.g., 1 pixel per block), the next read with additional pixel data (e.g., 2 pixels per block), the next read with additional pixel data (e.g., 3 pixels per block), the next read with additional pixel data (e.g., 4 pixels per block, as with group 505), the next read with additional pixel data (e.g., 5 pixels per block), and so on, up to full-resolution reads (e.g., 16 pixels per block in block 603A in Figure 6A).

[0085]

[0105] In some examples, different groups may have different numbers of pixels in each of the blocks 603A. For example, one group may have one pixel per block, a second group may have two pixels per block, a third group may have three pixels per block, a fourth group may have four pixels per block, and a fifth group may again have one pixel per block. Pixels may have any arrangement within a block. In some examples, pixels may be arranged in the same way (e.g., the same arrangement of grouping) in each of the blocks of the grid. In some examples, pixels may be arranged in different ways (e.g., different arrangements of grouping) in different blocks of the grid.

[0086]

[0106] Figure 6B is a conceptual diagram 600B illustrating the capture of an image 610 with a first resolution 612 and / or a first image 620 with a second resolution 622 by performing sparse readout of an image sensor 601B, which has a pixel array organized into a grid 602B of four blocks 603B, where each pixel of the pixel array is organized into one of six groups, and each block contains nine pixels, each containing at least one pixel from each group. The pixel array of the image sensor 601B is 6 pixels wide × 6 pixels high. Each block 603B is 3 pixels wide × 3 pixels high. Therefore, the pixel array of the image sensor 601B is 2 blocks wide × 2 blocks high.

[0087]

[0107] Despite the smaller size of block 603B compared to block 603A, the image sensor 601B in Figure 6B is still capable of capturing image 610 with a first resolution 612 and / or first image 620 with a second resolution 622, just as the image sensor 601A in Figure 6A is capable of. This is because block 603B still contains four pixels from the first group 505 and one pixel from the sixth group 530. In total, each of the blocks 603B contains four pixels from the first group 505, one pixel from the second group 510, one pixel from the fifth group 525, one pixel from the sixth group 530, one pixel from the seventh group 535, and one pixel from the tenth group 550. In some examples, the first resolution 612 is a higher resolution than the second resolution 622.

[0088]

[0108] Figures 5A-5C and 6A-6B should be understood to represent non-limiting examples of grids, blocks, and pixel grouping structures, patterns, arrays, and / or layouts. The image sensor of the imaging system may be arranged in a grid having more, fewer, or equal numbers of blocks compared to any of the grids in Figures 5A-5C and 6A-6B. The image sensor of the imaging system may be arranged in a grid having more, fewer, or equal numbers of pixels compared to any of the blocks in the grids in Figures 5A-5C and 6A-6B. The image sensor of the imaging system may be arranged in a grid having more, fewer, or equal numbers of photodiodes compared to any of the blocks in the grids in Figures 5A-5C and 6A-6B. The image sensor of the imaging system may have a pixel array having more, fewer, or equal numbers of pixels compared to any of the pixel arrays of the image sensors in Figures 5A-5C and 6A-6B. The image sensors of the imaging system may be arranged in a grid having blocks that are larger, smaller, or equal in size to any of the blocks in the grids of Figures 5A-5C and 6A-6B. The image sensors of the imaging system may be arranged in a grid having blocks that contain more pixel groups, fewer pixel groups, or an equal number of pixel groups compared to any of the blocks in the grids of Figures 5A-5C and 6A-6B. The image sensors of the imaging system may be arranged on a grid having blocks that contain more pixels, fewer pixels, or an equal number of pixels within a pixel group compared to any of the pixel groups in Figures 5A-5C and 6A-6B. The image sensors of the imaging system may be arranged in a grid having one or more square blocks, as in the blocks of Figures 5A-5C and 6A-6B. The image sensors of the imaging system may be arranged in a grid having one or more rectangular blocks. The image sensors of the imaging system may be arranged in a grid having one or more rectangular blocks, where one side of the block is longer (with respect to the number of pixels) than the other side of the block.The image sensor of an imaging system may be arranged in a grid with blocks having pixels arranged in any pattern, array, or layout of pixel grouping. In some examples, each block has its pixels arranged in the same pixel grouping pattern, as in the blocks of Figures 5A-5C and 6A-6B. In some examples, different blocks have their pixels arranged in different pixel grouping patterns. Different patterns of pixel grouping can be repeated horizontally and / or vertically across the image sensor according to one or more grids, such as grid 502, grid 602A, or grid 602B. For example, a particular pixel grouping pattern may be used for every other block in a layout. Different pixel groups may be arranged across the image sensor according to different patterns. Different patterns for different pixel groupings may be arranged according to the same grid or different grids. In some examples, the grids, blocks, and pixel grouping structures, patterns, arrays, and / or layouts of Figures 5A-5C and 6A-6B can represent examples of imaging systems with a global shutter. In some examples, the grids, blocks, and pixel grouping structures, patterns, arrays, and / or layouts in Figures 5A-5C and 6A-6B can represent examples of imaging systems with rolling shutters.

[0089]

[0109] In some examples, the pixel arrays in Figures 5A-5C and 6A-6B are arrays of image pixels. In some examples, the pixel arrays in Figures 5A-5C and 6A-6B contain only image pixels. In some examples, the pixel arrays in Figures 5A-5C and 6A-6B contain both image pixels and focal pixels. In some examples, one of the pixel groups in Figures 5A-5C and 6A-6B may be a group of image pixels. In some examples, one of the pixel groups in Figures 5A-5C and 6A-6B may contain only image pixels. In some examples, one of the pixel groups in Figures 5A-5C and 6A-6B may contain both image pixels and focal pixels. In some examples, one of the pixel groups in Figures 5A-5C and 6A-6B may contain focal pixels.

[0090]

[0110] Figure 7 is a conceptual diagram 700 showing the capture of a first image 725 at a first resolution 727, a second image 735 at a first resolution 727, a second image 755 at a second resolution 757, and a third image 745 at a first resolution 727. The captures of the first image 725 at a first resolution 727, the second image 735 at a first resolution 727, the second image 755 at a second resolution 757, and the third image 745 at a first resolution 727 are shown relative to a horizontal time axis 790 in which time flows from left to right. The time axis 790 is sometimes called a timeline.

[0091]

[0111] Three durations are identified along the time axis 790: a first time 705, a second time 710 following the first time 705, and a third time 715 following the second time 710. At the start of the first time 705, all pixels of the imaging system's image sensor are reset, as indicated by reset 722 (e.g., the “reset all” function). At the end of the first time 705, the imaging system captures a first image 725 at a first resolution 727 based on a first sparse readout 728, during which the imaging system selectively reads one or more designated pixel groups without reading any other pixel groups other than one or more designated pixel groups. In some examples, during the first sparse readout 728, the imaging system reads all image pixels of one or more designated pixel groups. In some examples, during the first sparse readout 728, the imaging system reads all image pixels and all focal pixels of one or more designated pixel groups. The time between the reset 722 and the sparse readout 728 to capture the first image 725 at the first resolution 727 is called the integral 720. Exposure of the image sensor to light from the scene over the exposure time, and the resulting accumulation of charge in the photodetector of the image sensor, may occur during the integral 720. During the integral 720, analog pixel data from all pixels of the image sensor and / or photodetectors may be stored in the memory array 135, as shown in Figures 1, 2, and 3A-3B, and discussed with respect to them.

[0092]

[0112] At the start of the second time period 710, all pixels of the image sensor of the imaging system are reset, as indicated by reset 732 (e.g., the “reset all” function). At the end of the second time period 710, the imaging system captures a second image 735 at the first resolution 727 based on a second sparse readout 738, during which the imaging system selectively reads one or more specified pixel groups without reading any other pixel groups other than one or more specified pixel groups. In some examples, the one or more specified pixel groups for the second sparse readout 738 are the same as those for the first sparse readout 728. In some examples, the one or more specified pixel groups for the second sparse readout 738 are different from those for the first sparse readout 728. In some examples, the one or more specified pixel groups for the second sparse readout 738 include at least one of the one or more specified pixel groups for the first sparse readout 728. In some examples, one or more designated pixel groups for the second sparse readout 738 exclude at least one of the one or more designated pixel groups for the first sparse readout 728. In some examples, during the second sparse readout 738, the imaging system reads all image pixels of one or more designated pixel groups. In some examples, during the second sparse readout 738, the imaging system reads all image pixels and all focal pixels of one or more designated pixel groups. The time between the reset 732 and the sparse readout 738 for capturing the second image 735 at the first resolution 727 is called the integral 730. Exposure of the image sensor to light from the scene over the exposure time, and the resulting accumulation of charge in the photodetector of the image sensor may occur during the integral 730. During the integral 730, analog pixel data from all pixels of the image sensor and / or photodetectors may be stored in the storage array 135, as shown in Figures 1, 2, and 3A-3B and discussed with respect to them.

[0093]

[0113] At the start of the third time period 715, all pixels of the imaging system's image sensor are reset, as indicated by reset 742 (e.g., the “reset all” function). At the end of the third time period 715, the imaging system captures a third image 745 at the first resolution 727 based on a third sparse readout 748, in which the imaging system selectively reads one or more specified pixel groups without reading any other pixel groups other than one or more specified pixel groups during that time. In some examples, the one or more specified pixel groups for the third sparse readout 748 are the same as those for the first sparse readout 728 and / or the second sparse readout 738. In some examples, the one or more specified pixel groups for the third sparse readout 748 are different from those for the first sparse readout 728 and / or the second sparse readout 738. In some examples, one or more designated pixel groups for the third sparse readout 748 include at least one of the one or more designated pixel groups for the first sparse readout 728 and / or the second sparse readout 738. In some examples, one or more designated pixel groups for the third sparse readout 748 exclude at least one of the one or more designated pixel groups for the first sparse readout 728 and / or the second sparse readout 738. In some examples, during the third sparse readout 748, the imaging system reads all the image pixels of one or more designated pixel groups. In some examples, during the third sparse readout 748, the imaging system reads all the image pixels of one or more designated pixel groups and all the focus pixels. The time between reset 742 and sparse readout 748 to capture the third image 745 at the first resolution 727 is called integral 740. Exposure of the image sensor to light from the scene over the exposure time, and the resulting accumulation of charge in the photodetector of the image sensor, may occur during integral 740. During integration 740, analog pixel data from all pixels of the image sensor and / or photodetectors can be stored in the storage array 135, as shown in Figures 1, 2, and 3A-3B and discussed with respect to them.

[0094]

[0114] Request 750 is received between the second time 710 and the third time 715. Request 750 requests the capture of image 755 at a second resolution 757. Since the data from all pixels and / or photodiodes of the image sensor is still stored in the storage array 135 following the integration 730 and has not yet been reset by the reset 742, the imaging system can perform a readout for the remaining pixel data from the remaining pixels and / or photodiodes that have not yet been read out as part of the sparse readout 738, and thus perform a dense readout 758 of the image sensor and capture image 755 at the second resolution 757. In some examples, the second resolution 757 is higher than the first resolution 727.

[0095]

[0115] In some examples, during a close readout 758, the imaging system selectively reads out one or more designated pixel groups without reading out any other pixel groups other than one or more designated pixel groups. In some examples, one or more designated pixel groups for the close readout 758 include one or more designated pixel groups for the second sparse readout 738. In some examples, one or more designated pixel groups for the close readout 758 include one or more other pixel groups other than the designated pixel groups for the second sparse readout 738. In some examples, one or more designated pixel groups are the same for the close readout 758 and include at least one of the one or more designated pixel groups for the first sparse readout 728 and / or for the second sparse readout 738 and / or for the third sparse readout 748. In some examples, one or more designated pixel groups are the same for dense readout 758, except for at least one of the one or more designated pixel groups for the first sparse readout 728 and / or the second sparse readout 738 and / or the third sparse readout 748. In some examples, one or more designated pixel groups for dense readout 758 are different with respect to the first sparse readout 728 and / or the second sparse readout 738 and / or the third sparse readout 748. In some examples, one or more designated pixel groups for dense readout 758 are the same as the first sparse readout 728 and / or the third sparse readout 748. In some examples, during dense readout 758, the imaging system reads all of the pixel groups of the image sensor. In some examples, during dense readout 758, the imaging system reads all of the pixels of the image sensor. In some examples, during dense readout 758, the imaging system reads all of the image pixels of one or more designated pixel groups. In some examples, during the tight readout 758, the imaging system reads all image pixels and all focal pixels of one or more specified pixel groups. In some examples, during the tight readout 758, the imaging system reads all image pixels of the image sensor. In some examples, during the tight readout 758, the imaging system reads all image pixels and all focal pixels of the image sensor. In some examples, the tight readout 758 may be a complete readout of the image sensor.

[0096]

[0116] In some examples, request 750 may be based on user interface input received from the user interface. In some examples, request 750 may be provided by the imaging system, received by the imaging system, or both. For example, request 750 may be provided and / or received by the imaging system based on the imaging system detecting characteristics of the second image 735 at the first resolution 727 (and / or the first image 725 at the first resolution 727) that suggest that an image at the second resolution 757 (or possibly at a higher resolution than the first resolution 727) may be desirable and / or beneficial for the use case. In some examples, the imaging system may provide and receive request 750 based on the imaging system determining (e.g., using an object detection algorithm) that the second image 735 at the first resolution 727 (and / or the first image 725 at the first resolution 727) includes a depiction of an object. In some examples, the object includes a glyph that optically encodes information (e.g., a one-dimensional barcode or a two-dimensional (2D) barcode) that may be easier for an imaging system (e.g., an object detection algorithm and / or glyph detection and / or scanning and / or interpretation algorithm) to scan and / or interpret at a second resolution 757 (or possibly a resolution higher than the first resolution 727). Examples of 2D barcodes include Quick Response (QR) codes, Aztec codes, data matrices, PDF417 codes, MaxiCodes, Codablock F codes, Han Xin codes, and DotCode. In some examples, these may be referred to as 2D barcodes and 2D codes. Example 754 of the detection of a glyph (QR Code®) in a second image 735 at a first resolution 727 is shown as an example of an imaging system that triggers request 750. In Example 754, the glyph is visible within the field of view of the camera of a mobile handset.A first depiction of the glyph corresponding to a first resolution 727 is shown on the mobile handset's display to indicate that the second image 735 of the first resolution 727 may include a first depiction of the glyph, based on the fact that the glyph is within the mobile handset's FOV when the second image 735 of the first resolution 727 is captured. The first depiction of the glyph may be too blurry and / or low-resolution for the imaging system to interpret the information optically encoded by the glyph from the depiction of the glyph having a confidence value above the confidence threshold (and therefore, request 750), but the first resolution 727 of the first depiction of the glyph may be sufficient to detect that the glyph is actually depicted in the second image 735 having a confidence value above the confidence threshold. The image 755 of the second resolution 757 includes a second depiction of the glyph which may be detailed, sharp, clear, and / or high-resolution enough for the imaging system to interpret the information encoded by the glyph from the second depiction of the glyph having a confidence value above the confidence threshold. In some examples, the information encoded by the glyph may include uniform resource indicators (URIs), such as uniform resource locators (URLs). In some examples, the information encoded by the glyph may include alphanumeric characters.

[0097]

[0117] In some examples, objects include glyphs, parts of people (e.g., a person's face, a person's body), parts of vehicles (e.g., license plates), parts of animals (e.g., an animal's face, an animal's body), one or more alphanumeric characters (e.g., arranged in a sequence of alphanumeric characters), or combinations thereof. Object detection algorithms may include feature detection algorithms, feature recognition algorithms, object detection algorithms, object recognition algorithms, face detection algorithms, face recognition algorithms, person detection algorithms, person recognition algorithms, optical character detection algorithms, optical character recognition (OCR) algorithms, classifiers, optical glyph detectors, optical glyph scanners, or combinations thereof. In some examples, object detection algorithms may detect objects more effectively and / or consistently using images with a second resolution of 757 (or, in some cases, a higher resolution than the first resolution of 727) than using images with a first resolution of 727.

[0098]

[0118] In some examples, the object is a set of one or more alphanumeric characters that may be easier for the object detection algorithm to analyze using an image with a second resolution of 757 (or, in some cases, a resolution higher than the first resolution of 727). The object detection algorithm may include an optical character recognition (OCR) algorithm.

[0099]

[0119] In some examples, the object is an object and / or image region having a splendor that exceeds a splendor threshold. The imaging system can determine the splendor of the object and / or image region, at least partially, by generating a splendor map based on images of a first resolution 727 (e.g., the first image 725 and / or the second image 735). In some examples, the imaging system can generate a splendor map based on images of a first resolution 727 (e.g., the first image 725 and / or the second image 735) by inputting these images at the first resolution 727 into a machine learning model that has been trained as input images. The trained machine learning model is trained to output a splendor map based on one or more input images. In some examples, the imaging system can determine the distinctiveness of an object and / or image region based at least in part on determining that a second image 735 of a first resolution 727 includes movement (e.g., exceeding a movement threshold) compared to a previous image (e.g., a first image 725 of the first resolution 727). In some examples, the imaging system can determine the distinctiveness of an object and / or image region based at least in part on identifying that the object or image region has a unique color or pattern not found elsewhere in the image of the first resolution 727. In some examples, the imaging system can determine the distinctiveness of an object and / or image region based at least in part on the distance (e.g., in color space and / or lightness space) between the color and / or lightness values ​​of pixels in the object and / or image region and the color and / or lightness values ​​of one or more other pixels in the image of the first resolution 727. In some examples, the imaging system can identify the significance of a given pixel in a second image 735 of a first resolution 727, at least partially based on the distance (e.g., in color space and / or brightness space) between the color value and / or brightness value corresponding to a given pixel in the image and the color value and / or brightness value of one or more other pixels in the image (e.g., other than the given pixel).In some examples, the color and / or brightness values ​​of one or more pixels in an image may include the average color and / or brightness values ​​of pixels in the image, so that the imaging system finds the distance between the color and / or brightness values ​​of a given pixel and the average color and / or brightness values ​​of pixels in the image. In some examples, the color and / or brightness values ​​of one or more pixels in an image may include the average color and / or brightness values ​​of pixels in the image other than the given pixel, so that the imaging system finds the distance between the color and / or brightness values ​​of a given pixel and the average color and / or brightness values ​​of pixels in the image other than the given pixel.

[0100]

[0120] Therefore, the imaging device can determine on the fly to switch from a first sparse readout 738 (e.g., sparse and / or low resolution) to a second dense readout 758 (e.g., dense and / or high resolution) before time 715, and therefore before the reset 742 and / or integration 740 begins. This is a technical improvement over conventional cameras where a change in settings may require the reconstruction of the image sensor. Conventional image sensor reconstruction for changing settings is generally not applied until a time delay has elapsed after the request for the change has been made (during which subsequent image frames may be captured). The imaging device offers technical improvements such as reduced power consumption, reduced bandwidth usage, reduced heat generation, more effective heat dissipation, and / or reduced computational resource usage. These improvements are achieved by using sparse readouts for the first sparse readout 728, second sparse readout 738, and third sparse readout 748 when low-resolution images 727 (e.g., images 725, 735, and 745) are sufficient, and by using a second readout 758 (e.g., tight readout or full readout) of the second resolution 757 when an image of second resolution 757 (e.g., image 755) is requested via request 750 (e.g., by the user or by the imaging device itself) for an operation that is beneficial or required.

[0101]

[0121] In some examples, the imaging system may select the same pixel group for each sparse readout (e.g., first sparse readout 728, second sparse readout 738, and third sparse readout 748) of a series of images with a first resolution 727 (e.g., images 725, 735, and 745). For example, in the context of Figures 5A to 5C, the imaging system may select a first group 505 for each sparse readout (e.g., first sparse readout 728, second sparse readout 738, and third sparse readout 748) of a series of images with a first resolution 727 (e.g., images 725, 735, and 745). In some examples, the imaging system can alternate which group is selected for each sparse readout (e.g., first sparse readout 728, second sparse readout 738, and third sparse readout 748) of a series of images with a first resolution 727 (e.g., images 725, 735, and 745). For example, the imaging system can perform a first sparse readout 728 on the first image 725 with a first resolution 727 based on the first group 505 (as in Figure 5B), a second readout 738 on the second image 735 with a first resolution 727 based on the second group 510 (as in Figure 5C), a third sparse readout 748 on the third image 735 with a first resolution 727 based on the third group 515, and so on. The imaging system may cycle through the groups until it has performed sparse readouts on all groups, and then routinely redo the first group. The imaging system can store the images generated in this way. By performing sparse readout on all groups during this routine, the imaging system can store images captured using all groups. The imaging system can combine these images into a higher-resolution combined image by arranging pixel data from different images based on the layout of pixel groups within a block of image sensors. This combined image may have a higher resolution than the images that were combined to form it. In this way, the imaging system can gradually collect a high-resolution (or full-resolution) view of a scene by combining multiple low-resolution captured images generated via sparse readout.This combined image can still accurately depict the scene, at least for the parts of the scene that remained still during the capture of multiple images.

[0102]

[0122] In some examples, the imaging device may attempt to perform object detection on other images of the first resolution 727, other than the second image 735 of the first resolution 727, such as the first image 725 and / or the third image 745 of the first resolution 727, using the object detection algorithm. In these attempts to perform object detection, a request for the image of the second resolution 757 (as in the case of request 750) may not occur because the attempt to perform object detection resulted in the object detection algorithm failing to detect any of the objects it attempts to detect within a threshold time, such as before the next reset (e.g., reset 732, reset 742). In some examples, the imaging system may automatically perform a reset (e.g., reset 732, reset 742) in response to the object detection algorithm failing to detect any of the objects it attempts to detect within a threshold time.

[0103]

[0123] In some examples, the second resolution 757 is higher than the first resolution 727. In some examples, the imaging system may delay the reset 742 for a certain period (e.g., in response to request 750 and / or detection 752) to give the imaging system sufficient time to perform the tight readout 758, for example, if the imaging system determines that the tight readout 758 may be interrupted by the reset 742. The delay period may be a predetermined period. The delay period may be based on the amount of time to perform the tight readout 758, on a scheduled time for the reset 742, and / or on an additional predetermined buffer period. In some examples, the second resolution 757 is the full resolution possible for the image sensor in Figure 7. In some examples, the first resolution 727 corresponds to a first set of one or more pixel groups per block, and the second resolution 757 corresponds to a second set of one or more pixel groups per block. In exemplary examples, the first resolution 727 corresponds to group 530 in Figures 6A-6B, and the second resolution 757 corresponds to group 505 in Figures 6A-6B. In some examples, a dense readout 758 at the second resolution 757 may have any of the numerous levels of readout density discussed with respect to Figures 6A-6B, exceeding the sparsest readout density level. For example, a medium-density readout, such as the readout of the first image 610 in Figures 6A-6B based on selection 615 of the first group 505, is sufficient for reading glyphs, or for face recognition, or for object recognition, or for OCR, or for any other function that the imaging device performs using the image 755.

[0104]

[0124] In some examples, requests such as request 750 may be used to trigger a second camera and / or second image sensor other than the image sensor used to capture images 725, 735, and / or 745. For example, image 755 may be captured using the second camera and / or second image sensor. In some examples, the second camera and / or second image sensor may have a different resolution than the camera and / or image sensor used to capture images 725, 735, and / or 745. In some examples, the second camera and / or second image sensor may have a higher resolution than the camera and / or image sensor used to capture images 725, 735, and / or 745. In some examples, image data (e.g., images 725, 735, and / or 745) and / or image capture settings (e.g., exposure settings, focus settings, white balance settings, zoom settings, flash settings, color balance settings, tone settings, saturation settings, gain settings, image processing settings, etc.) from the camera and / or image sensor used to capture images 725, 735, and / or 745 may be used to set image capture settings (e.g., exposure settings, focus settings, white balance settings, zoom settings, flash settings, color balance settings, tone settings, saturation settings, gain settings, image processing settings, etc.) for a second camera and / or second image sensor. This may be used, for example, to speed up the exposure convergence of the second camera and / or second image sensor.

[0105]

[0125] Please understand that in this specification, references to pixels may refer to photodiodes. Similarly, references to groups of pixels may refer to groups of photodiodes.

[0106]

[0126] Figure 8 is a conceptual diagram 800 showing the movement of the reset pointer 840 and the readout pointer 845 along the image sensor 830 in a rolling shutter imaging system. The movement of the reset pointer 840 and the readout pointer 845 along the image sensor 830 is shown with respect to a horizontal time axis 890, where time flows from left to right. The time axis 890 is sometimes called a timeline.

[0107]

[0127] Both the reset pointer 840 and the read pointer 845 move downward along the image sensor 830, from the top to the bottom. Both the reset pointer 840 and the read pointer 845 can move downward along the image sensor 830 at the same speed. Therefore, the reset pointer 840 and the read pointer 845 can maintain a time gap between them, called the integration time 835. Exposure of the image sensor 830 to light from the scene over the exposure time, and the resulting accumulation of charge in the photodetector of the image sensor 830, can occur during the integration time 835.

[0108]

[0128] Four time points are identified along the time axis 890: the first time point 805, the second time point 810 following the first time point 805, the third time point 815 following the second time point 810, and the fourth time point 820 following the third time point 815. In the first time point 805, the reset pointer 840, shown as a thick horizontal black dashed line, is located near the top of the image sensor 830 and resets the rows near the top of the image sensor 830. In the second time point 810, the reset pointer 840 moves downward toward the center of the image sensor 830 and resets the rows near the center of the image sensor 830. The read pointer 845, shown as a thick horizontal black solid line, is located near the top of the image sensor 830 and reads pixel data from the rows near the top of the image sensor 830. The distance between the reset pointer 840 and the read pointer 845 along the image sensor 830 represents the distance that can be covered during integration time 835 at the speed at which the reset pointer 840 and the read pointer 845 move. At the third time point 815, the reset pointer 840 moves downward toward the bottom of the image sensor 830, resetting the rows near the bottom of the image sensor 830. The read pointer 845 is near the center of the image sensor 830, reading pixel data from the rows near the center of the image sensor 830. The interval between the reset pointer 840 and the read pointer 845 along the image sensor 830 still corresponds to the integral time 835, as the speeds of the reset pointer 840 and the read pointer 845 are consistent. At the fourth time point 820, the reset pointer 840 moves downward past the bottom of the image sensor 830, turns back toward the top of the image sensor 830, and resets the rows near the top of the image sensor 830. The read pointer 845 is near the bottom of the image sensor 830, reading pixel data from the rows near the bottom of the image sensor 830.

[0109]

[0129] When the read pointer 845 reaches the bottom of the image sensor 830 and reads the last pixel data from the image sensor 830, the image sensor 830 may capture an image based on the read pixel data from the image sensor 830. The time axis 890 can then be reset back to time 805 and / or time 810 for depicting the reset, integration, and readout of the next image captured by the image sensor 830. In some examples, the reset pointer 840 scans across the image sensor in raster scan order. In some examples, the read pointer 845 scans across the image sensor in raster scan order.

[0110]

[0130] Figure 9 is a conceptual diagram 900 showing a rolling shutter imaging system that performs a first image acquisition 920, a second image acquisition 925, and a third image acquisition 930. The first image acquisition 920, the second image acquisition 925, and the third image acquisition 930 are shown relative to a horizontal time axis 990 in which time flows from left to right. The time axis 990 is sometimes called a timeline.

[0111]

[0131] Three durations are identified along the time axis 990: a first time 905, a second time 910 following the first time 905, and a third time 915 following the second time 910. The imaging system performs a first image acquisition 920 during the first time 905, a second image acquisition 925 during the second time 910, and a third image acquisition 925 during the third time 915. The first image acquisition 920 allows the imaging system to acquire a first image, the second image acquisition 925 allows the imaging system to acquire a second image, and the third image acquisition 930 allows the imaging system to acquire a third image. The first image, the second image, and the third image may be a sequence of video frames.

[0112]

[0132] The first image capture 920, the second image capture 925, and the third image capture 930 are each shown as parallelograms, with the upper row further to the left (earlier along the time axis 990) than the lower row, and the lower row further to the right (later along the time axis 990) than the upper row. This is because the upper row of the image sensor is reset, integrated, and / or read out earlier than the lower row of the image sensor. The left end of the parallelogram contains a thick black dashed downward arrow representing the reset of the image sensor by the reset pointer 940. The reset pointer 940 in Figure 9 moves from the top to the bottom of the image sensor, similar to the reset pointer 840 in Figure 8. The left end of the parallelogram contains a thick black downward arrow representing the reading of pixel data from the image sensor by the read pointer 945. The read pointer 945 in Figure 9 moves from the top to the bottom of the image sensor, similar to the read pointer 845 in Figure 8. Each parallelogram includes a rightward arrow representing the integral 935 between the reset at the left end of the parallelogram (e.g., corresponding to the reset pointer 940) and the read at the right end of the parallelogram (e.g., corresponding to the read pointer 945). Exposure of the image sensor to light from the scene over the exposure time, and the resulting accumulation of charge in the image sensor's photodetector, can occur during the integral 935. In some examples, the reset pointer 940 scans across the image sensor in raster scan order. In some examples, the read pointer 945 scans across the image sensor in raster scan order.

[0113]

[0133] Figure 10A is a conceptual diagram 1000A showing image capture of six image frames 1050A to 1050F, each with a first resolution 1096, via a first readout pattern 1055 using a rolling shutter imaging system having four pixel groups. The four pixel groups include a first group 1010, a second group 1020, a third group 1030, and a fourth group 1040. An example of an image sensor 1070 (or part of an image sensor 1070) is shown in Figure 10A. The image sensor 1070, or the shown part of the image sensor 1070, has dimensions of 6 pixels wide × 6 pixels high.

[0114]

[0134] The pixels of the image sensor 1070 are arranged according to the grid 1072 of block 1074, as well as the grid 502 of block 504 in Figures 5A-5C, the grid 602A of block 603A in Figure 6A, and / or the grid 602B of block 603B in Figure 6B. Each of the blocks 1074 contains one pixel from each of the four pixel groups. Thus, each of the blocks 1074 contains a pixel from the first group 1010, a pixel from the second group 1020, a pixel from the third group 1030, and a pixel from the fourth group 1040. The pixels from each of the four groups are arranged in the same order, pattern, and / or arrangement within each of the blocks 1074. The outline of an exemplary block 1073 of block 1074 is drawn as a dashed, rounded rectangle. For example, the top-left pixel in exemplary block 1073 is in the first group 1010, the top-right pixel in exemplary block 1073 is in the second group 1020, the bottom-left pixel in exemplary block 1073 is in the third group 1030, and the bottom-right pixel in exemplary block 1073 is in the fourth group 1040. These four pixel groups are arranged in each of block 1074 according to this order, pattern, and / or arrangement.

[0115]

[0135] In some examples, block 1074 may have two or more pixels in a particular group. Examples of blocks having two or more pixels in a particular group include blocks 603A and 603B in Figures 6A and 6B, each having four pixels in the first group 505. In some examples, one of the blocks 1074 may have a different order, pattern, and / or arrangement of pixel groups compared to another of the blocks 1074.

[0116]

[0136] The first readout pattern 1055 is shown with respect to a horizontal time axis 1090 in which time flows from left to right. The time axis 1090 is sometimes called a timeline. Six image frames are captured sequentially according to the first readout pattern 1055, and each image frame is captured at a first resolution 1060. The six image frames, in order from left to right along the time axis 1090, include image frame 1050A, image frame 1050B, image frame 1050C, image frame 1050D, image frame 1050E, and image frame 1050F. Each image frame is shown in Figures 10A to 10C as a series of parallelograms along the time axis 1090, including a first group 1010, a second group 1020, a third group 1030, and a fourth group 1040. Each parallelogram represents all the pixels across the image sensor 1070 belonging to the group to which the parallelogram is labeled. Specifically, the parallelogram labeled as the first group 1010 represents all the pixels on the image sensor 1070 within the first group 1010, the parallelogram labeled as the second group 1020 represents all the pixels on the image sensor 1070 within the second group 1020, the parallelogram labeled as the third group 1030 represents all the pixels on the image sensor 1070 within the third group 1030, and the parallelogram labeled as the fourth group 1040 represents all the pixels on the image sensor 1070 within the fourth group 1040. The pixel data within each group can be read from top to bottom and left to right according to the raster scanning order.

[0117]

[0137] According to the first readout pattern 1055, all pixels across the image sensor 1070 are reset, exposed, integrated, and read out for all six image frames. For all six image frames, all pixels in the first group 1010, the second group 1020, the third group 1030, and the fourth group 1040 are reset, exposed, integrated, and read out. In Figures 10A to 10C, all parallelograms along the time axis 1090 include thick black dashed lines along their left side, representing the reset of the image sensor 1070 for the pixels within the group corresponding to the group markings on the parallelogram. In Figure 10A, all parallelograms along the time axis 1090 include thick black solid lines along their right side, representing the readout of pixel data from the pixels within the group corresponding to the group markings on the parallelogram. The area of ​​the parallelogram itself represents the exposure and / or integration between reset and readout.

[0118]

[0138] Figure 10B is a conceptual diagram 1000B showing image capture of eight image frames 1052A to 1052H, each with a second resolution 1062, via a second readout pattern 1057 using a rolling shutter imaging system having four pixel groups. The eight image frames, in order from left to right along the time axis 1090, include image frame 1052A, image frame 1052B, image frame 1052C, image frame 1052D, image frame 1052E, image frame 1052F, image frame 1052G, and image frame 1052H. Similar to Figure 10A, each image frame is shown as a parallelogram row along the time axis 1090, including a first group 1010, a second group 1020, a third group 1030, and a fourth group 1040. However, in Figure 10B, some of the parallelograms along the time axis 1090 lack a thick black solid line along their right side, indicating that those pixel groups are not read out for that image frame according to the second readout pattern 1057. To help highlight the groups that are not read out according to the second readout pattern 1057, the parallelograms of those groups are also shaded with a halftone shading pattern.

[0119]

[0139] The second readout pattern 1057 reads different pixel groups within each image frame until all four pixel groups are read out, and then repeats the pattern of capturing different pixel groups for the next round of image frames. In Figure 10B, the second readout pattern 1057 captures the first group 1010 in image frame 1052A, the second group 1020 in image frame 1052B, the third group 1030 in image frame 1052C, and the fourth group 1040 in image frame 1052D. The diagonal group reading pattern established in image frames 1052A to 1052D is repeated in image frames 1052E to 1052H.

[0120]

[0140] According to the second reading pattern 1057, only the first pixel group 1010 is read from image frame 1052A and image frame 1052E. The second pixel group 1020, the third pixel group 1030, and the fourth pixel group 1040 are not read from image frame 1052A and image frame 1052E. According to the second reading pattern 1057, only the second pixel group 1020 is read from image frame 1052B and image frame 1052F. The first pixel group 1010, the third pixel group 1030, and the fourth pixel group 1040 are not read from image frame 1052B and image frame 1052F. According to the second reading pattern 1057, only the third pixel group 1030 is read from image frame 1052C and image frame 1052G. For image frames 1052C and 1052G, the first pixel group 1010, the second pixel group 1020, and the fourth pixel group 1040 are not read. According to the second reading pattern 1057, only the fourth pixel group 1040 is read for image frames 1052D and 1052H. For image frames 1052D and 1052H, the first pixel group 1010, the second pixel group 1020, and the third pixel group 1030 are not read.

[0121]

[0141] Since only one pixel group is read for each image frame, the image data captured for each individual image frame is read and captured at a second resolution of 1062. The second resolution of 1062 is lower than the first resolution of 1060 for image frames 1050A to 1050F in Figure 10A, where all four groups are read. Since the readout skips specific pixel groups for specific image frames, capturing image frames using the second readout pattern 1057 in Figure 10B uses less power, less bandwidth (e.g., to transmit image data), less computing resources (e.g., for image processing), and generates less heat than capturing image frames using the first readout pattern 1055 in Figure 10A.

[0122]

[0142] In some examples, an imaging system including an image sensor 1070 can combine pixel data from different groups read out for different image frames into a single combined image having a resolution higher than a second resolution 1062. For example, an imaging system including an image sensor 1070 can combine pixel data from a first group 1010 from image frame 1052A, pixel data from a second group 1020 from image frame 1052B, pixel data from a third group 1030 from image frame 1052C, and / or pixel data from a fourth group 1040 from image frame 1052D into a single combined image. Similarly, the imaging system can combine pixel data from a first group 1010 from image frame 1052E, pixel data from a second group 1020 from image frame 1052F, pixel data from a third group 1030 from image frame 1052G, and / or pixel data from a fourth group 1040 from image frame 1052H into a single combined image. When the imaging system combines pixel data from all four groups into a combined image, the combined image has a first resolution of 1060. In this way, the imaging system can still obtain high-quality and high-resolution images while still maintaining advantages such as using less power, using less bandwidth, using less computing resources, and generating less heat.

[0123]

[0143] This technique for generating a combined image works particularly well when the imaging system is capturing an image of a scene that includes static regions. Since the static regions of a scene do not change significantly as time progresses along the time axis 1090, these static regions appear similar while image frame 1052A is being captured, while image frame 1052B is being captured, while image frame 1052C is being captured, while image frame 1052D is being captured, while image frame 1052E is being captured, and / or while image frame 1052B is being captured. Therefore, for the static regions of a scene, the combined image may appear indistinguishable from one of the image frames 1050A-1050F in Figure 10A, where all of the pixel groups are read out.

[0124]

[0144] Some visual artifacts can arise from techniques used to generate a combined image when an imaging system is capturing images of a scene that includes dynamic regions. Visual artifacts can be caused by the combination of different sets of pixels read out at different times related to the capture of different image frames. Examples of visual artifacts associated with interlaced video, such as combing, ghosting, interline twitter, moiré, aliasing effects, or a combination thereof, may be, or may appear similar. In some examples, an imaging system including an image sensor 1070 can reduce or eliminate visual artifacts by applying one or more deinterlacing algorithms to the combined image, to other combined images adjacent to the combined image along the time axis 1090, and / or to the pixel data related to the individual sets of pixels used to generate the combined image. One or more deinterlacing algorithms may include, for example, field composite deinterlacing, field expansion deinterlacing, motion compensation deinterlacing, edge detection, blending, selective blending, weaving, inverse telecine, half-sizing, line doubling, block motion compensation, scene change detection, decimation, video filters, denoising filters, deblocking filters, or a combination thereof.

[0125]

[0145] In some examples, the imaging system may use a third readout pattern (not shown) that is similar to the second readout pattern 1057 but in a different order, pattern, and / or arrangement, allowing it to read only specific groups without reading other groups. In one exemplary example, the third readout pattern may capture the fourth group 1040 in image frame 1052A, the third group 1030 in image frame 1052B, the second group 1020 in image frame 1052C, and the fourth group 1040 in image frame 1052D. In some examples, the third readout pattern may capture only a subset of the pixel groups. In another exemplary example, the third readout pattern captures only image data and / or pixel data from one, two, or three groups (selected from, for example, the first group 1010, the second group 1020, the third group 1030, and the fourth group 1040). In some examples, the third read pattern may include several image frames that are completely skipped so that no image data or pixel data is read during those frames. In another exemplary example, the third read pattern captures only the image data and / or pixel data from every other image frame.

[0126]

[0146] Figure 10C is a conceptual diagram 1000C showing the transition from capturing an image frame of second resolution 1062 via a second readout pattern 1057 to capturing an image frame of first resolution 1060 via a first readout pattern 1055, using a rolling shutter imaging system having four pixel groups. The image frames, in order from left to right along the time axis 1090, include image frame 1054A, image frame 1054B, image frame 1054C, image frame 1054D, image frame 1054E, image frame 1054F, image frame 1054G, and image frame 1054H. Similar to Figures 10A to 10B, each image frame is shown as a parallelogram row along the time axis 1090, including a first group 1010, a second group 1020, a third group 1030, and a fourth group 1040.

[0127]

[0147] The first five image frames, including image frames 1054A, 1054B, 1054C, 1054D, and 1054E, are captured at a second resolution 1062 according to a second readout pattern 1057, similar to Figure 10B. At a time along the time axis 1090 after the capture of image frame 1054E, the imaging system, equipped with the image sensor 1070, receives a request 1085 for image data 1087 at a first resolution 1060. After receiving the request 1085, the imaging device transitions from the second readout pattern 1057 to the first readout pattern 1055. The imaging system captures image frames 1054G and 1054H according to the first readout pattern 1055, similar to Figure 10A. Since all pixel groups are reset for each image frame, even if they are not read out, the transition between the second readout pattern 1057 and the first readout pattern 1055 can be instantaneous or nearly instantaneous, as there is no need to delay until a reset can be performed. Resetting all image frames does not add a significant delay. In some examples (not shown), the image sensor 1070 of the imaging system can skip resetting a particular image frame, for example, to reset every other image frame.

[0128]

[0148] Since request 1085 is received in the middle of capturing and / or reading from different pixel groups of image frame 1054F, the capture of image frame 1054F includes reading more groups than under the second reading pattern 1057 and reading fewer groups than under the first reading pattern 1055. In an exemplary example (not shown), request 1085 may be received earlier than shown in Figure 10C, and image frame 1054F may be fully captured according to the first reading pattern 1055, and therefore, the first group 1010 may also be read. In another exemplary example (not shown), request 1085 may be received later than shown in Figure 10C, and image frame 1054F may be fully captured according to the second reading pattern 1057, and therefore, the reading of the third group 1030 and the fourth group 1040 may also be skipped.

[0129]

[0149] In some examples (not shown), an imaging system with image sensor 1070 can, in addition or alternatively, transition from a first readout pattern 1055 to a second readout pattern 1057. In some examples (not shown), an imaging system with image sensor 1070 can, in addition or alternatively, transition to and / or from one of the third readout patterns discussed with respect to Figure 10B.

[0130]

[0150] In some examples, request 1085, or a similar request for transitioning from one read pattern to another, may be triggered by a manual request received via the user interface. In some examples, request 1085 may be automatically triggered by the detection by the imaging device equipped with the image sensor 1070 of one or more objects in one or more of the image frames 1054A to 1054E captured under the second read pattern 1057 (and / or in a combined image generated based on pixel data from different sets of pixels read from different image frames). The detection may include any type of detection discussed with respect to detection 752 in Figure 7. The detection may include, for example, glyph detection, as in example 754 of detection 752 in Figure 7. Similar to Figure 7, the detection in Figure 10C may include, for example, object detection using an object detection algorithm. Objects may include, for example, glyphs, parts of people (e.g., a person's face, a person's body), parts of vehicles (e.g., license plates), parts of animals (e.g., an animal's face, an animal's body), one or more alphanumeric characters (e.g., arranged in a sequence of alphanumeric characters), or combinations thereof. Object detection algorithms may include feature detection algorithms, feature recognition algorithms, object detection algorithms, object recognition algorithms, face detection algorithms, face recognition algorithms, person detection algorithms, person recognition algorithms, optical character detection algorithms, optical character recognition (OCR) algorithms, classifiers, optical glyph detectors, optical glyph scanners, or combinations thereof.

[0131]

[0151] In some examples, the object is an object and / or image region that has a splendor exceeding a splendor threshold. The imaging system can determine the splendor of an object and / or image region by generating a splendor map based on at least one of the captured image frames 1054A to 1054E (and / or within a combined image generated based on pixel data from different sets of pixels read from different image frames) under a second readout pattern 1057. In some examples, the imaging system can generate a splendor map based on these images by inputting these images into a machine learning model that has been trained as input images. The trained machine learning model is trained to output a splendor map based on one or more input images. In some examples, the imaging system can determine the splendor of an object and / or image region by at least partly determining that the image contains motion (e.g., exceeding a motion threshold) compared to one or more previous images from the image frames 1054A to 1054E. In some examples, the imaging system can determine the splendor of an object and / or image region by at least partly identifying that the object or image region has a unique color or pattern not found elsewhere in the image. In some examples, the imaging system can determine the significance of an object and / or image region based at least partially on the distance (e.g., in color space and / or brightness space) between the color and / or brightness values ​​of pixels in the object and / or image region and the color and / or brightness values ​​of one or more other pixels in the image. In some examples, the imaging system can identify the significance of a given pixel in the image based at least partially on the distance (e.g., in color space and / or brightness space) between the color and / or brightness values ​​of a given pixel in the image and the color and / or brightness values ​​of one or more other pixels in the image (e.g., other than the given pixel). In some examples, the color and / or brightness values ​​of one or more pixels in the image may include the average color and / or brightness values ​​of pixels in the image, so that the imaging system finds the distance between the color and / or brightness value of a given pixel and the average color and / or brightness values ​​of pixels in the image.In some examples, the color and / or brightness values ​​of one or more pixels in an image may include the average color and / or brightness values ​​of pixels in the image other than the given pixel, so that the imaging system finds the distance between the color and / or brightness values ​​of a given pixel and the average color and / or brightness values ​​of pixels in the image other than the given pixel.

[0132]

[0152] Figure 11 is a conceptual diagram showing a rolling shutter readout pattern 1100 using a rolling shutter imaging system. The rolling shutter readout pattern 1100 is shown with respect to a horizontal time axis 1190 where time flows from left to right, and to a vertical row axis 1192 which represents different rows of the image sensor. The time axis 1190 is sometimes called a timeline. Different rows of the image sensor are shown along different horizontal lines that are perpendicular to the vertical row axis 1192 and intersect the vertical row axis 1192 at different points in time. The rolling shutter readout pattern 1100 includes a reset 1112 of group 1010 and a subsequent readout 1115 of group 1010, along with an integral 1110 of group 1010 between them. The rolling shutter readout pattern 1100 includes a reset 1122 of group 1020 and a subsequent readout 1125 of group 1020, along with an integral 1120 of group 1020 between them. The rolling shutter readout pattern 1100 includes a reset of group 1030 1132 and a subsequent readout of group 1030 1135, along with an integral 1130 of group 1030 between them. The rolling shutter readout pattern 1100 also includes a reset of group 1040 1142 and a subsequent readout of group 1040 1145, along with an integral 1140 of group 1040 between them. Different groups can be reset at different times. There is a temporal overlap between integrals of adjacent groups.

[0133]

[0153] Figure 12 is a flowchart showing the operation for imaging. Process 1200 may be carried out by an imaging system. In some examples, the imaging system may include, for example, an imaging system in Figure 2 that implements at least a subset of process 200, an imaging system including an image sensor 310 and / or a storage array 320, an imaging system including an image sensor 410 and a rolling shutter 420, an imaging system including an image sensor 501, an imaging system including an image sensor 601A, an imaging system with a rolling shutter in Figure 4A, an imaging system with a global shutter in Figure 4B, an imaging system with an image sensor 501 in Figures 5A-5C, an imaging system with an image sensor 601A in Figure 6A, The imaging system may include an imaging system equipped with an image sensor 601B in Figure 6B, an imaging system that captures at least a subset of the images in Figure 7 (e.g., images 725, 735, 745, and 755), an imaging system including an image sensor 830 in Figure 8, an imaging system that performs at least a subset of the image captures in Figure 9 (e.g., image captures 920, 925, and 930), an imaging system that performs readouts according to a first readout pattern 1055 in Figures 10A and 10C, an imaging system that performs readouts according to a second readout pattern 1057 in Figures 10B to 10C, an imaging system to which the rolling shutter readout pattern 1100 in Figure 11 is applied, a computing system 1300, a processor 1310, or a combination thereof.

[0134]

[0154] In operation 1205, the imaging system is configured and capable of exposing a plurality of photodetectors of the image sensor to light from the scene. Each of the plurality of photodetectors is configured to convert the light from the scene into an electric charge in response to exposure to light from the scene. The plurality of photodetectors includes at least a first group of photodetectors and a second group of photodetectors that are different from each other. In some examples, the imaging system includes an image sensor. Examples of image sensors include image sensor 130, an image sensor including photodetector 215, image sensor 310, the image sensor of the imaging system in Figure 4A, the image sensor of the imaging system in Figure 4B, image sensor 501, image sensor 601A, image sensor 601B, the image sensor of the imaging system in Figure 7, image sensor 830, the image sensor of the imaging system in Figure 9, the image sensor of any of the imaging systems in Figures 10A to 10C, the image sensor of the imaging system in Figure 11, another image sensor, or a combination thereof. Examples of photodetectors include photodetector 215, photodetectors P0 to P63 of image sensor 310, photodetectors corresponding to pixels of image sensor 501, photodetectors corresponding to pixels of image sensor 601A, photodetectors corresponding to pixels of image sensor 601B, photodetectors corresponding to pixels of image sensor 830, photodetectors corresponding to pixels of any of the image sensors shown in Figures 10A to 10C, or combinations thereof. An example of a scene is scene 110. Examples of separate photodetector groups in the context of Figures 5A-5C and Figures 6A-6B include the first group 505, the second group 510, the third group 515, pixels in the fourth group 520, the fifth group 525, the sixth group 530, the seventh group 535, pixels in the eighth group 540, the ninth group 545, the tenth group 550, pixels in the eleventh group 555, the twelfth group 560, the thirteenth group 565, the fourteenth group 570, the fifteenth group 575, and the sixteenth group 580. Examples of separate photodetector groups in the context of Figures 10A-10C include the first group 1010, the second group 1020, the third group 1030, and the fourth group 1040. Exposing the image sensor's photodetector to light from the scene may include operating one or more actuators and / or motors to open a shutter associated with the image sensor and / or open an aperture associated with the image sensor.The image sensor may, following the exposure of the image sensor's photodetector to light from the scene, terminate or end the exposure of the image sensor's photodetector to light from the scene, which may include operating one or more actuators and / or motors to close a shutter associated with the image sensor and / or close an aperture associated with the image sensor.

[0135]

[0155] In some embodiments, the multiple photodetectors include multiple photodiodes. In some embodiments, the multiple photodetectors of an image sensor include multiple image photodetectors (e.g., image photodiodes) and multiple focal photodetectors (e.g., focal photodiodes). The multiple focal photodetectors are configured for phase-detection autofocus (PDAF). The first group of photodetectors includes a first subset of multiple image photodiodes. The first group of photodetectors may also include a first subset of multiple focal photodetectors. The second group of photodetectors includes a second subset of multiple image photodiodes. The second group of photodetectors may also include a second subset of multiple focal photodetectors.

[0136]

[0156] In operation 1210, the imaging system is configured and can store a plurality of analog photodetector signals corresponding to charges from each of a plurality of photodetectors. In some embodiments, the image sensor includes a storage array comprising a plurality of storage units. To store each of the plurality of analog photodetector signals corresponding to charges from each of a plurality of photodetectors, the imaging system (e.g., the image sensor) is configured and can store each of the plurality of analog photodetector signals in one of the plurality of storage units of the storage array. Examples of storage arrays include storage array 265 and storage array 320. Examples of storage units include storage unit 260 and storage units S0 to S63 of storage array 320.

[0137]

[0157] In some embodiments, one of several analog photodetector signals represents a voltage corresponding to a charge from one of the photodetectors. An example of an analog photodetector signal is analog photodetector signal 250.

[0138]

[0158] In some embodiments, the imaging system (e.g., an image sensor) is configured and can reset each of the multiple photodetectors of the image sensor and / or the multiple storage units of the storage array before exposing the multiple photodetectors to light from the scene in operation 1205. Examples of resetting the photodetectors and / or storage units include reset block 404, reset block 414, reset block 424, reset block 434, reset block 444, reset block 454, reset block 464, reset block 474, reset 722, reset 732, reset 742, reset using reset pointer 840, reset using reset pointer 940, reset using any of the resets (indicated by dashed lines) in Figures 10A to 10C, reset 1112, reset 1122, reset 1132, reset 1142, or a combination thereof.

[0139]

[0159] In operation 1215, the imaging system is configured and capable of reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to the first photodetector group, without reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to the second photodetector group. An example of the first digital pixel data is digital pixel data 255. Reading the first digital pixel data is sometimes referred to as reading out the first digital pixel data. Examples of reading or reading out the first digital pixel data include reading from the image capture device 105A by the image processing device 105A, reading digital pixel data 255, reading from the image sensor 310, reading from the storage array 320, reading block 408, reading block 418, reading block 428, reading block 438, reading block 448, reading block 458, reading block 468, reading block 478, and sparse reading. Examples include reading 586, sparse reading 592, the three readings in Figure 6A, the two readings in Figure 6B, the first sparse reading 728, the second sparse reading 738, the third sparse reading 748, dense reading 758, reading by read pointer 845, reading by read pointer 945, reading by the first reading pattern 1055, reading by the second reading pattern 1057, reading 1115, reading 1125, reading 1135, reading 1145, or combinations thereof.

[0140]

[0160] In some examples, the imaging system may include a connector coupled to the image sensor. Reading first digital pixel data may include receiving first digital pixel data using the connector. The connector may include ports, jacks, wires, input / output (I / O) pins, conductive traces on a printed circuit board (PCB), any other type of connector discussed herein, or any combination thereof. In some examples, the imaging system may include an image sensor. In some examples, exposing the image sensor's multiple photodetectors to light from the scene, as in operation 1205, may include transmitting a signal from one or more processors of the imaging system to the image sensor via the connector. In some examples, causing the image sensor to store multiple analog photodetector signals corresponding to charges from each of the multiple photodetectors, as in operation 1210, may include transmitting a signal from one or more processors of the imaging system to the image sensor via the connector. In some examples, as in operation 1215, reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first group of photodetectors includes receiving and / or reading first digital pixel data and / or a first subset (and / or processed variant thereof) of multiple analog photodetector signals via a connector.

[0141]

[0161] In some embodiments, to read first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group without reading a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, the imaging system is configured and can be used to use an imaging circuit to selectively apply modifications to a first subset of multiple analog photodetector signals without applying modifications to a second subset of multiple analog photodetector signals. Modifications may include, for example, amplification of the multiple analog photodetector signals via one or more analog gain amplifiers, such as amplifier 230. Modifications may include filtering the multiple analog photodetector signals via one or more filters, such as a high-pass filter, a low-pass filter, a band-pass filter, or a combination thereof. An example of modifications using such filters is discussed with respect to Figure 2. Modifications may include, for example, conversion of the multiple analog photodetector signals to digital pixel data via one or more ADCs, such as analog-to-digital converters (ADCs) 235 and / or ADC 132. The corrections may include processing operations performed by ISP154, such as demosaicing, digital gain, missing pixel correction, faulty pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, upsampling, additional image processing operations, or combinations thereof.

[0142]

[0162] In some embodiments, the imaging system (e.g., an image sensor and / or an imaging circuit associated with the image sensor) includes one or more amplifiers. An example of one or more amplifiers is amplifier 230. To read first digital pixel data from a first subset of multiple analog photodetector signals, the imaging system is configured and can have one or more amplifiers to amplify each of the first subset of multiple analog photodetector signals corresponding to a first group of photodetectors into multiple amplified analog photodetector signals. An example of an amplified analog photodetector signal is analog photodetector signal 250 amplified once by amplifier 230. In some embodiments, the first digital pixel data is based on multiple amplified analog photodetector signals. An example of digital pixel data here is digital pixel data 255 based on photodetector signal 250 amplified by amplifier 230.

[0143]

[0163] In some embodiments, the imaging system (e.g., an image sensor) includes one or more analog-to-digital converters (ADCs). Examples of one or more ADCs include ADC132 and ADC235. To read first digital pixel data from a first subset of multiple analog photodetector signals, the imaging system is configured and can cause one or more ADCs to convert multiple analog signals based on a first subset of multiple analog photodetector signals corresponding to a first group of photodetectors into digital signals. The first digital pixel data is based on digital signals. Examples of digital signals include digital pixel data 255 and / or the output of ADC235 to the image processor 240.

[0144]

[0164] In some embodiments, the imaging system is configured and capable of generating, at least partially, first digital pixel data based on a digital signal by processing the digital signal using one or more image processing operations. One or more image processing operations may be performed by an image processor, such as ISP154, host processor152, image processor150, image processor240, processor1310, or a combination thereof. An example of generating first digital pixel data by processing a digital signal using one or more image processing operations performed by an image processor is generating digital pixel data 255 by processing the digital signal output by ADC235 using one or more image processing operations performed by image processor240. One or more image processing operations may include digital gain, demosaicing, pixel interpolation, missing pixel correction, faulty pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, upsampling, or a combination thereof. In some embodiments, the multiple analog signals include multiple amplified analog photodetector signals. One or more amplifiers can generate at least partially multiple amplified analog photodetector signals by amplifying a first subset of multiple analog photodetector signals corresponding to a first group of photodetectors. An example of one or more amplifiers is amplifier 230.

[0145]

[0165] In operation 1220, the imaging system is configured and capable of generating an image of the scene using at least first digital pixel data.Examples of images include images captured by the image capture and processing system 100, images captured via image capture 270 based on digital pixel data 255, images captured using image sensor 310 and / or storage array 320, images captured via imaging device with a rolling shutter as shown in Figure 4A, images captured via imaging device with a global shutter as shown in Figure 4B, and images captured using image sensor 501 (for example, based on digital pixel data from one or more of groups 505 to 580, group 505 to 580). Images (without digital pixel data from one or more different groups of 580), image 585 with a first resolution of 587, image 590 with a first resolution of 587, image 610 with a first resolution of 612, first image 620 with a second resolution of 622, second image 630 with a second resolution of 622, images captured using image sensor 601A (for example, based on digital pixel data from one or more different groups of 505-580, without digital pixel data from one or more different groups of 505-580), images captured using image sensor 601B (for example, from groups of 505-580) Images based on digital pixel data from one or more of the groups 505-580 (without digital pixel data from one or more different digital pixel data from groups 505-580), first image 725 with first resolution 727, second image 735 with first resolution 727, third image 745 with first resolution 727, image 755 with second resolution 757, image captured using image sensor 830 based on pixel data read according to read pointer 845, image captured using image capture 920 based on pixel data read according to read pointer 945, read according to read pointer 945 Examples include images captured using image capture 925 based on output pixel data, images captured using image capture 930 based on pixel data read according to read pointer 945, images captured according to a first read pattern 1055, images captured according to a second read pattern 1057, images captured using image sensor 1070, images captured according to read 1115, images captured according to read 1125, images captured according to read 1135, images captured according to read 1145, or combinations thereof.

[0146]

[0166] In some embodiments, an imaging system (e.g., an image sensor) is configured and can reset multiple photodetectors and / or multiple memory units of a memory array of the image sensor after capturing an image. Examples of resetting the photodetectors and / or memory units include reset block 404, reset block 414, reset block 424, reset block 434, reset block 444, reset block 454, reset block 464, reset block 474, reset 722, reset 732, reset 742, reset using reset pointer 840, reset using reset pointer 940, reset using any of the resets shown by dashed lines in Figures 10A to 10C, reset 1112, reset 1122, reset 1132, reset 1142, or a combination thereof.

[0147]

[0167] In some embodiments, an imaging system is configured and capable of analyzing images using an object detection algorithm. The object detection algorithm may include feature detection algorithms, feature recognition algorithms, object detection algorithms, object recognition algorithms, face detection algorithms, face recognition algorithms, person detection algorithms, person recognition algorithms, optical character detection algorithms, optical character recognition (OCR) algorithms, classifiers, optical glyph detectors, optical glyph scanners, or combinations thereof. The object detection algorithm may be executed by one or more processors of the imaging system. The object detection algorithm may include one or more trained machine learning (ML) systems capable of receiving one or more input images (e.g., images) as input, which can determine whether one or more input images depict a particular type of object, such as an optical glyph, a face, a person, at least a part of a person (e.g., a hand, a face), one or more alphanumeric characters (e.g., arranged in a string), at least a part of a vehicle, at least a part of an animal, at least a part of a plant, at least a part of a structure (e.g., a building), or combinations thereof. One or more trained ML systems may include one or more trained ML models, one or more trained neural networks (NNs), one or more trained convolutional neural networks (CNNs), one or more trained time-delay neural networks (TDNNs), one or more deep networks, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more trained support vector machines (SVMs), one or more trained random forests (RFs), or a combination thereof.

[0148]

[0168] In some examples, if the object detection algorithm detects an object, the imaging system can read a second set of digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. In some examples, if the object detection algorithm fails to detect any object of its type for at least a threshold period, the imaging system can reset multiple photodetectors and / or multiple storage units of the image sensor's storage array. In some examples, the threshold period is the period that ends at the moment when the reset of the photodetectors and / or storage array is scheduled to be performed by the imaging system. In some examples, the imaging system (e.g., the image sensor) is configured and can automatically reset multiple photodetectors and multiple storage units of the image sensor's storage array in response to the object detection algorithm failing to detect one or more objects in the image.

[0149]

[0169] In some embodiments, the imaging system is configured and can determine, after generating an image in operation 1220, whether to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. In some embodiments, the imaging system is configured and can determine, based on the image generated in operation 1220, whether to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. For example, the decision on whether to read second digital pixel data may be based on an analysis of the image generated in operation 1220 using an object detection algorithm.

[0150]

[0170] In some embodiments, the imaging system is configured and capable of reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group, and then reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group. An example of the second digital pixel data is digital pixel data 255. Reading the second digital pixel data is sometimes referred to as reading out the second digital pixel data. Examples of reading or reading out the second digital pixel data include reading from the image capture device 105A by the image processing device 105A, reading out digital pixel data 255, reading from the image sensor 310, reading from the storage array 320, reading block 408, reading block 418, reading block 428, reading block 438, reading block 448, reading block 458, reading block 468, reading block 478, and sparse reading. Examples include reading 586, sparse reading 592, the three readings in Figure 6A, the two readings in Figure 6B, the first sparse reading 728, the second sparse reading 738, the third sparse reading 748, dense reading 758, reading by read pointer 845, reading by read pointer 945, reading by the first reading pattern 1055, reading by the second reading pattern 1057, reading 1115, reading 1125, reading 1135, reading 1145, or combinations thereof.

[0151]

[0171] In some embodiments, the reading of second digital pixel data by the imaging system is based on the storage of multiple analog photodetector signals in operation 1210. An example of the reading of second digital pixel data by the imaging system based on the storage of analog photodetector signals is the reading of digital pixel data 255 based on stored analog photodetector signals 250, which are stored in the storage unit 260 of the storage array 265, and optionally amplified by the amplifier 230, converted by the ADC 235, and optionally processed by the image processor 240. In some embodiments, the imaging system is configured and capable of generating a second image of a scene using at least the first and second digital pixel data. The second image has a second resolution different from the first resolution of the image generated in operation 1220. In some embodiments, the second image has a second resolution that is higher (e.g., greater) than the first resolution of the image generated in operation 1220. In some embodiments, the imaging system is configured to read and be able to read a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors after generating an image of the scene in operation 1220.

[0152]

[0172] In some embodiments, an imaging system is configured and capable of identifying areas in an image that have a splendor exceeding a splendor threshold. For example, an imaging system can generate a splendor map based on an image. In some examples, an imaging system can generate a splendor map by using an image as input to one or more trained machine learning (ML) systems that are trained to generate splendor maps corresponding to input images that are input to one or more trained machine learning (ML) systems. In some examples, an imaging system can generate a splendor map by identifying areas that have motion compared to a previous image captured by an image sensor before the image. In some examples, an imaging system can generate a splendor map by identifying areas that have a unique color or pattern not found elsewhere in the image. In some examples, an imaging system can identify the splendor of a given pixel in an image at least in part on the distance (e.g., in color space and / or brightness space) between the color value and / or brightness value corresponding to a given pixel in the image and the color value and / or brightness value of one or more other pixels (e.g., other than the given pixel) in the image. In some examples, the color and / or brightness values ​​of one or more pixels in an image may include the average color and / or brightness values ​​of pixels in the image, so that the imaging system finds the distance between the color and / or brightness values ​​of a given pixel and the average color and / or brightness values ​​of pixels in the image. In some examples, the color and / or brightness values ​​of one or more pixels in an image may include the average color and / or brightness values ​​of pixels in the image other than the given pixel, so that the imaging system finds the distance between the color and / or brightness values ​​of a given pixel and the average color and / or brightness values ​​of pixels in the image other than the given pixel. The imaging system can use a splendor map to identify areas with splendor exceeding a splendor threshold. In some embodiments, the imaging system is configured and can read a second set of digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors in response to identifying that the image contains areas with splendor exceeding a splendor threshold.In some examples, if the imaging system generates a splendor map of the image and identifies that there are no areas in the image with splendor exceeding a splendor threshold, the imaging system may reset the photodiode and / or memory array. The identification of areas with splendor exceeding a splendor threshold, or their absence, may trigger, or be absent, some of the same behaviors as object detection using object detection algorithms.

[0153]

[0173] In some embodiments, the imaging system is configured and capable of identifying that an image indicates motion relative to a previous image of the scene captured by the image sensor, and the motion exceeds a motion threshold. In some examples, the image sensor may capture a sequence of images. For example, a sequence of images may be video frames of a video. In some examples, the previous image is an image in the sequence of images that is immediately preceding an image in the sequence of images, and there are no additional images between the previous image and the image in the sequence of images. In some examples, the previous image is an image in the sequence of images that is two or more images preceding an image in the sequence of images, and there are one or more additional images between the previous image and the image in the sequence of images. In the context of Figure 7, if the image produced in operation 1220 is a third image 745 with a first resolution 727, the previous image may be a first image 725 with a first resolution 727, a second image 735 with a first resolution 727, or an image 755 with a second resolution 757. In the context of Figure 7, if the image generated in operation 1220 is a second image 735 with a first resolution 727, then the previous image could be a first image 725 with a first resolution 727. Motion can include the movement of an object, such as the movement of a person from the previous image to the image. The motion threshold can indicate the minimum distance (e.g., in pixels) that an object must move between the previous image and that image. The imaging system is configured to read and be read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors in response to identifying that the image shows motion relative to the previous image of the scene captured by the image sensor, and the motion exceeds the motion threshold.

[0154]

[0174] In some embodiments, the imaging system is configured and can determine, after generating a second image, whether to read a third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third group of photodetectors, based on the second image. The multiple photodetectors include a third group of photodetectors distinct from the first and second groups of photodetectors. The image is generated in operation 1220 without the third digital pixel data. The second image is also generated without the third digital pixel data.

[0155]

[0175] In some embodiments, the imaging system is configured and capable of reading a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, and then reading a third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third photodetector group. Reading the third digital pixel data is based on the storage of multiple analog photodetector signals in operation 1210. An example of reading third digital pixel data by an imaging system based on the storage of analog photodetector signals is the reading of digital pixel data 255 based on stored analog photodetector signals 250, which are stored in the storage unit 260 of the storage array 265, possibly amplified by the amplifier 230, converted by the ADC 235, and possibly processed by the image processor 240. The multiple photodetectors include a third photodetector group distinct from the first and second photodetector groups. An image is generated in operation 1220 without the third digital pixel data. A second image is also generated without the third digital pixel data. The imaging system is configured and can generate a third image of the scene using at least first digital pixel data, second digital pixel data, and third digital pixel data. The third image has a third resolution that is higher (e.g., greater) than the second resolution of the second image and / or the first resolution of the first image generated in operation 1220. In some embodiments, the imaging system is configured and can read the third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third photodetector group after generating the image of the scene and / or the second image in operation 1220.

[0156]

[0176] An example of the third digital pixel data is digital pixel data 255. The reading of the third digital pixel data is sometimes called the readout of the third digital pixel data. Examples of reading or readout of the first and third pixel data include reading from the image capture device 105A by the image processing device 105A, reading of digital pixel data 255, reading from the image sensor 310, reading from the storage array 320, read block 408, read block 418, read block 428, read block 438, read block 448, read block 458, read block 468, read block 478, and sparse readout. Examples include 586, sparse read 592, the three reads in Figure 6A, the two reads in Figure 6B, the first sparse read 728, the second sparse read 738, the third sparse read 748, the dense read 758, the read using read pointer 845, the read using read pointer 945, the read using the first read pattern 1055, the read using the second read pattern 1057, read 1115, read 1125, read 1135, read 1145, or combinations thereof.

[0157]

[0177] In some embodiments, the imaging system is configured and capable of identifying, for example, using an object detection algorithm, that an image contains a depiction of an object. In response to identifying that an image contains a depiction of an object, the imaging system is configured and capable of reading a second set of digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors. In some embodiments, the object is a glyph that optically encodes information. The glyph may include at least one one-dimensional barcode, at least one two-dimensional code, or a combination thereof. In some embodiments, the object includes at least a part of a person and / or an animal. For example, the object may include a face, hand, arm, finger, fingerprint, handprint, leg, foot, footprint, eye, head, gait, pose, or a combination thereof. In some embodiments, the object includes at least a part of a vehicle. For example, the object may include the shape of the vehicle, the vehicle's license plate, a set of the vehicle's headlights, or a combination thereof. In some embodiments, the object includes one or more alphanumeric characters. For example, one or more alphanumeric characters may be arranged in a row on, for example, a piece of paper, a signature, a card, a display, a projection surface, or a combination thereof.

[0158]

[0178] In some embodiments, the multiple photodetectors of an image sensor are arranged according to a grid of blocks, each containing a distinct subset of multiple photodetectors. Each block of the grid includes at least one photodetector from a first group of photodetectors and at least one photodetector from a second group of photodetectors. Examples of grids include the grid 502 of block 504, the grid 602A of block 603A, the grid 602B of block 603B, the grid 1072 of block 1074 of pixels in Figures 10A to 10C, or combinations thereof.

[0159]

[0179] In some embodiments, a first group of photodetectors is arranged across the image sensor according to a first pattern, and a second group of photodetectors is arranged across the image sensor according to a second pattern. The first and second patterns correspond to one or more grids. The first pattern may be repeated horizontally and / or vertically according to one or more grids. The second pattern may be repeated horizontally and / or vertically according to one or more grids. Examples of one or more grids include grid 502, grid 602A, grid 602B, grid 1072 of pixel block 1074 in Figures 10A to 10C, or combinations thereof. In some embodiments, both the first and second patterns correspond to the same grid. In some embodiments, the first pattern corresponds to the first grid, and the second pattern corresponds to the second grid. The first and second grids are distinct from each other. For example, in the context of Figures 5A to 5C, the first grid could be grid 502 of block 504, while the second grid is a grid of clusters of multiple adjacent blocks 504 (for example, each cluster is a 2x2 cluster of four adjacent blocks 504).

[0160]

[0180] In some embodiments, the imaging system is configured and can move a reset pointer across a plurality of reset pointer positions from a first side of the image sensor to a second side of the image sensor at a predetermined speed. An example of moving the reset pointer across a plurality of reset pointer positions is shown in Figure 8, where the first side of the image sensor is at the top of the image sensor 830 in Figure 8 and the second side of the image sensor is at the bottom of the image sensor 830 in Figure 8. Examples of resetting using the reset pointer are also shown in at least Figures 4A, 9, 10A-10C, and 11. To reset each of the plurality of photodetectors, the imaging system is configured and can reset the first photodetector of the plurality of photodetectors in response to the reset pointer reaching one of a plurality of reset pointer positions on which the first photodetector of the plurality of photodetectors is located. A read pointer is moved across a plurality of read pointer positions from a first side of the image sensor to a second side of the image sensor at a predetermined speed. In some examples, the imaging system is configured and capable of reading out a third digital pixel data corresponding to the second photodetector of the first photodetector group in response to a readout pointer reaching one of a plurality of reset pointer positions where the second photodetector of the first photodetector group is located. A predetermined speed may be associated with a rolling shutter, such as the rolling shutter of the imaging system in Figures 4A, 8, 9, 10A to 10C, and 11.

[0161]

[0181] In some embodiments, the imaging system is configured and capable of outputting images. The imaging system may include a display. To output images, the imaging system is configured and capable of displaying images using the display. The imaging system may include a communication transceiver. To output images, one or more processors are configured and capable of transmitting images to a receiving device using the communication transceiver. Examples of displays, communication transceivers, and / or other output devices for outputting images include output device 1335, communication interface 1340, devices connected to I / O 156, devices connected to I / O 160, or combinations thereof.

[0162]

[0182] In some embodiments, the imaging system includes an image sensor, a plurality of photodetectors of the image sensor, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors comprises at least a first group of photodetectors and a second group of photodetectors that are different from each other, means for causing the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, means for reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second group of photodetectors, and means for generating an image of a scene using at least the first digital pixel data.

[0163]

[0183] Means for exposing multiple photodetectors of an image sensor include an image sensor 130, an image sensor including a photodetector 215, an image sensor 310, an image sensor of the imaging system in Figure 4A, an image sensor of the imaging system in Figure 4B, an image sensor 501, an image sensor 601A, an image sensor 601B, an image sensor of the imaging system in Figure 7, an image sensor 830, an image sensor of the imaging system in Figure 9, an image sensor of any of the imaging systems in Figures 10A to 10C, an image sensor of the imaging system in Figure 11, a control mechanism 120, an operating motor for controlling the opening and / or closing of the shutter, an operating motor for controlling the opening and / or closing of the aperture, or a combination thereof. Means for storing multiple analog photodetector signals in an image sensor include a storage unit 260, a storage array 265, a storage array 320, storage units S0 to S63, an image sensor including an image sensor 130, an image sensor including a photodetector 215, an image sensor 310, an image sensor of the imaging system in Figure 4A, an image sensor of the imaging system in Figure 4B, an image sensor 501, an image sensor 601A, an image sensor 601B, an image sensor of the imaging system in Figure 7, an image sensor 830, an image sensor of the imaging system in Figure 9, an image sensor of any of the imaging systems in Figures 10A to 10C, an image sensor of the imaging system in Figure 11, or a combination thereof. Means for reading the first digital pixel data without reading the second digital pixel data include an amplifier 230, a filter 220, an ADC 235, a memory unit 260, a memory array 265, an image processor 240, a read block 408, a read block 418, a read block 428, a read block 438, a read block 448, a read block 458, a read block 468, a read block 478, and sparse Includes read 586, sparse read 592, the three reads in Figure 6A, the two reads in Figure 6B, the first sparse read 728, the second sparse read 738, the third sparse read 748, the dense read 758, read pointer 845, read pointer 945, read by the first read pattern 1055, read by the second read pattern 1057, read 1115, read 1125, read 1135, read 1145, or combinations thereof.Means for generating an image include the imaging system in Figure 2 which implements at least a subset of the image acquisition and processing system 100, image acquisition device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, and process 200; the imaging system which includes an image processor 240, an image sensor 310 and / or a storage array 320; the imaging system which includes an image sensor 410 and a rolling shutter 420; the imaging system which includes an image sensor 501; the imaging system which includes an image sensor 601A; the imaging system which has a rolling shutter as shown in Figure 4A; the imaging system which has a global shutter as shown in Figure 4B; the imaging system which has an image sensor 501 as shown in Figures 5A to 5C; and the image sensor 60 as shown in Figure 6A. The imaging system includes an imaging system with 1A, an imaging system with an image sensor 601B of Figure 6B, an imaging system that captures at least a subset of the images of Figure 7 (e.g., images 725, 735, 745, and 755), an imaging system including an image sensor 830 of Figure 8, an imaging system that performs at least a subset of the image captures of Figure 9 (e.g., image captures 920, 925, and 930), an imaging system that performs readouts according to a first readout pattern 1055, an imaging system that performs readouts according to a second readout pattern 1057, an imaging system that applies the rolling shutter readout pattern 1100 of Figure 11, a computing system 1300, a processor 1310, or a combination thereof.

[0164]

[0184] In some examples, the processes described herein (e.g., the process in process diagram 7200, the process in Figure 7, the process in Figure 8, the process in Figure 9, the process in Figure 11, process 1200, and / or other processes described herein) may be performed by a computing device or apparatus. In some examples, process 200, the process in Figure 7, the process in Figure 8, the process in Figure 9, the process in Figure 11, and / or the process in Figure 7 may be performed by the image acquisition and processing system 100, the imaging system in Figure 2, the imaging system in Figure 4A, the imaging system in Figure 4B, the imaging system in Figure 7, or a combination thereof. In another example, process 200, the process in Figure 7, the process in Figure 8, the process in Figure 9, the process in Figure 11, and / or the process in Figure 7 may be performed by a computing device equipped with the computing system 1300 shown in Figure 13.

[0165]

[0185] Computing devices may include any suitable device such as a mobile device (e.g., a cell phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capacity to perform the processes described herein, including process 200, the process in Figure 7, the process in Figure 8, the process in Figure 9, the process in Figure 11, and / or the process in Figure 7. In some cases, a computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0166]

[0186] Components of a computing device may be implemented within a circuit. For example, a component may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), electronic circuits or other electronic hardware, and / or may be implemented using them, and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using them to perform the various operations described herein.

[0167]

[0187] Process 200, the process in Figure 7, the process in Figure 8, the process in Figure 9, the process in Figure 11, and / or the process in Figure 7 are shown as logical flow diagrams, block diagrams, or conceptual diagrams, and their operation represents a set of operations that may be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be interpreted as limiting, and any number of operations described may be combined in any order and / or in parallel to implement a process.

[0168]

[0188] In addition, the processes in Figure 7, Figure 8, Figure 9, Figure 11, process 1200, and / or other processes described herein may be carried out under the control of one or more computer systems consisting of executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is carried out collectively by hardware or a combination thereof on one or more processors. As described above, the code may be stored on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0169]

[0189] Figure 13 shows an example of a system for implementing several aspects of the present technology. More specifically, Figure 13 shows an example of a computing system 1300 which may be an internal computing system, a remote computing system, a camera, or any computing device constituting any of these components, where the components of the system communicate with each other using connection 1305. Connection 1305 may be a physical connection using a bus, or a direct connection to a processor 1310 in a chipset architecture, for example. Connection 1305 may also be a virtual connection, a network connection, or a logical connection.

[0170]

[0190] In some embodiments, the computing system 1300 is a distributed system in which the functions described herein can be distributed across one data center, multiple data centers, a peer network, and so on. In some embodiments, one or more of the system components described represent many components, each performing some or all of the functions described for the components. In some embodiments, the components can be physical or virtual devices.

[0171]

[0191] An exemplary system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that connect various system components to the processor 1310, including system memory 1315 such as read-only memory (ROM) 1320 and random access memory (RAM) 1325. The computing system 1300 may include a high-speed memory cache 1312 that is directly connected to the processor 1310, very close to the processor 1310, or integrated as part of the processor 1310.

[0172]

[0192] The processor 1310 may include an arbitrary general-purpose processor and hardware or software services such as services 1332, 1334, and 1336, which are stored in a memory device 1330 and configured to control the processor 1310, as well as dedicated processors whose software instructions are incorporated into the actual processor design. The processor 1310 may essentially be a fully self-contained computing system including multiple cores or processors, a bus, a memory controller, a cache, etc. The multicore processor may be symmetrical or asymmetrical.

[0173]

[0193] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or speech. The computing system 1300 may also include an output device 1335 that may be one or more of several output mechanisms. In some cases, a multimode system may allow a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may generally include a communication interface 1340 that can control and manage user input and system output.Communication interfaces include audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet® ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short-range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), and Worldwide Interoperability for Microwave Access. It is possible or facilitates the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing Access (WiMAX), infrared (IR) wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.The communication interface 1340 may include one or more GNSS receivers or transceivers used to determine the position of the computing system 1300 based on the reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite Systems (GNSS) systems. GNSS systems include, but are not limited to, the US Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's Beidou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no restrictions on operation with any particular hardware array, and therefore the basic features described herein may be readily replaced for improved hardware or firmware arrays as they are developed.

[0174]

[0194] The memory device 1330 may be a non-volatile and / or non-temporary and / or computer-readable memory device, such as a magnetic cassette, flash memory card, solid memory device, digital multipurpose disk, cartridge, floppy disk, flexible disk, hard disk, magnetic tape, magnetic strip / stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid memory, compact disc read-only memory (CD-ROM) optical disc, rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, Blu-ray disc (BDD) optical disc, holographic optical disc, another optical medium, secure digital (SD) card, micro secure digital (microSD) card, memory stick (registered trademark) card, smart card chip, EMV chip, subscriber identity module (SIM) card, mini / micro / nano / pico SIM card, or another integrated circuit (integrated) Circuits, ICs, chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM®), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (resistiveA hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as random-access memory (RRAM® / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), other memory chips or cartridges, and / or combinations thereof.

[0175]

[0195] The storage device 1330 may include software services, servers, services, etc., that cause the system to perform functions when code defining such software is executed by the processor 1310. In some embodiments, a hardware service that performs a particular function may include software components stored in a computer-readable medium with respect to the necessary hardware components, such as the processor 1310, connection 1305, and output device 1335, in order to perform the function.

[0176]

[0196] As used herein, the term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable medium may also include non-transient media on which data is stored and which do not contain carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital multi-purpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may have code and / or machine-executable instructions stored on the computer-readable medium that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted using any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0177]

[0197] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, as used to refer to them, non-transient computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0178]

[0198] Specific details are provided in the above description to give a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be practiced without these specific details. For clarity of description, in some cases the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments.

[0179]

[0199] Individual embodiments may be described above as processes or methods shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or main function.

[0180]

[0200] The processes and methods described above may be implemented using computer-executable instructions stored on or otherwise available from computer-readable media. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform some functions or a set of functions, or, in some cases, configure a general-purpose computer, a dedicated computer, or a processing device to perform some functions or a set of functions. The portion of computer resources used may be accessible via a network. Computer-executable instructions may be binary or intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods described above include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and network-connected storage devices.

[0181]

[0201] Devices implementing processes and methods in accordance with these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. A processor may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may be embodied in peripheral devices or add-in cards. Such functions may, as a further example, be implemented on circuit boards of different chips or on different processes running in a single device.

[0182]

[0202] Instructions, a medium for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0183]

[0203] In the above description, aspects of the present application are described with reference to their specific embodiments, but those skilled in the art will recognize that the application is not limited thereto. Therefore, while exemplary embodiments of the present application are described in detail herein, it should be understood that the concepts of the present invention can be embodied and employed in various other ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and aspects of the present application described above may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods have been described in a particular order. In alternative embodiments, it should be understood that the methods may be carried out in an order different from that described.

[0184]

[0204] Those skilled in the art will understand that the symbols or terms used herein for less than ("<") and greater than (">") may be replaced, without departing from the scope of this specification, with the symbols for less than or equal to ("≦") and greater than or equal to ("≧").

[0185]

[0205] When a component is described as "configured to perform a particular operation," such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0186]

[0206] The phrase "combined" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0187]

[0207] The wording of a claim that includes “at least one of” and / or “one or more” of a set, or any other wording, indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the wording of a claim that includes “at least one of A and B” means A, B, or A and B. In another example, the wording of a claim that includes “at least one of A, B, and C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording “at least one of” and / or “one or more” of a set does not limit the set to items enumerated within the set. For example, the wording of a claim that includes “at least one of A and B” can mean A, B, or A and B, and may also include items not enumerated in the set A and B.

[0188]

[0208] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their function. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this application.

[0189]

[0209] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the technique may be at least partially realized by a computer-readable data storage medium comprising program code that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH® memory, magnetic or optical data storage media, etc. The technique may, as an addition or alternative, be implemented at least in part by a computer-readable communication medium, such as a propagating signal or wave, which can carry or communicate program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0190]

[0210] The program code may be executed by a processor, which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to implement any of the techniques described herein. Instead of a general-purpose processor being a microprocessor, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein. In addition, in some embodiments, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC).

[0191]

[0211] Exemplary aspects of this disclosure include the following:

[0212] Embodiment 1. An imaging apparatus comprising a memory and one or more processors coupled to the memory, wherein one or more processors are configured to expose an image sensor to a plurality of photodetectors of the image sensor, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, the plurality of photodetectors including at least a first group of photodetectors and a second group of photodetectors which are different from each other, to light from a scene, to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, read first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second group of photodetectors, and generate an image of a scene using at least the first digital pixel data.

[0192]

[0213] Embodiment 2. The apparatus of Embodiment 1, wherein one or more processors use an imaging circuit to selectively apply a modification to a first subset of multiple analog photodetector signals without applying a modification to the second subset of multiple analog photodetector signals, in order to read first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group, without reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group.

[0193]

[0214] Embodiment 3. The apparatus according to either Embodiment 1 or 2, wherein one of the multiple analog photodetector signals indicates a voltage corresponding to a charge from one of the multiple photodetectors.

[0194]

[0215] Embodiment 4. An apparatus according to any one of Embodiments 1 to 3, further comprising an image sensor.

[0195]

[0216] Embodiment 5. An apparatus according to any of Embodiments 1 to 4, wherein one or more processors are configured to have one or more amplifiers amplify each of the first subsets of the multiple analog photodetector signals corresponding to the first photodetector group into multiple amplified analog photodetector signals using one or more amplifiers, in order to read first digital pixel data from a first subset of multiple analog photodetector signals, and the first digital pixel data is based on the multiple amplified analog photodetector signals.

[0196]

[0217] Embodiment 6. The apparatus of Embodiment 5, further comprising one or more amplifiers.

[0197]

[0218] Embodiment 7. An apparatus according to any of Embodiments 1 to 6, wherein one or more processors are configured to cause one or more analog-to-digital converters (ADCs) to convert multiple analog signals based on a first subset of multiple analog photodetector signals corresponding to a first group of photodetectors into digital signals using one or more ADCs, and the first digital pixel data is based on digital signals.

[0198]

[0219] Embodiment 8. The apparatus of Embodiment 7, further comprising one or more ADCs.

[0199]

[0220] Embodiment 9. An apparatus according to either Embodiment 7 or 8, wherein a plurality of analog signals include a plurality of amplified analog photodetector signals, and one or more amplifiers generate a plurality of amplified analog photodetector signals by at least partially amplified a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors.

[0200]

[0221] Embodiment 10. An apparatus according to any one of Embodiments 7 to 9, wherein one or more processors are configured to generate first digital pixel data based on a digital signal by processing the digital signal at least partially using one or more image processing operations, the one or more image processing operations include at least one of digital gain, demosaicing, pixel interpolation, missing pixel correction, bad pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, and upsampling.

[0201]

[0222] Embodiment 11. An apparatus according to any one of Embodiments 1 to 10, wherein one or more processors are configured to determine, after generating an image, whether to read a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors, based on the image.

[0202]

[0223] Embodiment 12. An apparatus according to any embodiment 1 to 11, wherein one or more processors are configured to read first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group, then read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, and use at least the first digital pixel data and the second digital pixel data to generate a second image of a scene, wherein the second resolution of the second image is higher than the first resolution of the image, and the reading of the second digital pixel data is based on the storage of multiple analog photodetector signals.

[0203]

[0224] Embodiment 13. The apparatus of Embodiment 12, wherein, after reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group, one or more processors are configured to generate an image of the scene and then read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group.

[0204]

[0225] Embodiment 14. An apparatus according to either Embodiment 12 or 13, wherein one or more processors are configured to identify that an image contains areas having a spleness exceeding a spleness threshold, and in response to identifying that an image contains areas having a spleness exceeding a spleness threshold, the processor is configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors.

[0205]

[0226] Embodiment 15. An apparatus according to any of Embodiments 12 to 14, wherein one or more processors are configured to identify that an image exhibits motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, and in response to the identification that an image exhibits motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, the processor reads second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors.

[0206]

[0227] Embodiment 16. An apparatus according to any of Embodiments 12 to 15, wherein one or more processors are configured to determine, after generating a second image, whether to read a third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third group of photodetectors based on the second image, and the multiple photodetectors include a third group of photodetectors different from the first and second groups of photodetectors, and the image and the second image are generated without third digital pixel data.

[0207]

[0228] Embodiment 17. An apparatus according to any of Embodiments 12 to 16, wherein one or more processors are configured to read second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, then read third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third photodetector group, and use at least first digital pixel data, second digital pixel data and third digital pixel data to generate a third image of a scene, wherein the third resolution of the third image is higher than the second resolution of the second image, and the reading of the third digital pixel data is based on the storage of multiple analog photodetector signals, the multiple photodetectors include a third photodetector group different from the first and second photodetector groups, and the image and the second image are generated without third digital pixel data.

[0208]

[0229] Embodiment 18. An apparatus according to any of Embodiments 12 to 17, wherein one or more processors are configured to identify that an image contains a depiction of an object, and in response to identifying that an image contains a depiction of an object, the processor is configured to read a second set of digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors.

[0209]

[0230] Embodiment 19. The apparatus of Embodiment 18, wherein the object is a glyph that optically encodes information, and the glyph includes at least one of a one-dimensional barcode and a two-dimensional code.

[0210]

[0231] Embodiment 20. An apparatus according to either Embodiment 18 or 19, wherein the object includes at least a part of a person.

[0211]

[0232] Embodiment 21. A device according to any of Embodiments 18 to 20, wherein the object includes at least a part of a vehicle.

[0212]

[0233] Embodiment 22. An apparatus according to any of Embodiments 18 to 21, wherein the object contains one or more alphanumeric characters.

[0213]

[0234] Embodiment 23. An apparatus according to any of Embodiments 1 to 22, wherein the plurality of photodetectors of the image sensor are arranged according to a grid of blocks, each of which contains an individual subset of the plurality of photodetectors, and each block of the grid of blocks contains at least one photodetector from a first group of photodetectors and at least one photodetector from a second group of photodetectors.

[0214]

[0235] Embodiment 24. An apparatus according to any of Embodiments 1 to 23, wherein a first group of photodetectors is arranged across an image sensor according to a first pattern, and a second group of photodetectors is arranged across an image sensor according to a second pattern, and the first pattern and the second pattern correspond to one or more grids.

[0215]

[0236] Embodiment 25. An apparatus according to any of Embodiments 1 to 24, wherein the image sensor includes a storage array having a plurality of storage units, and the image sensor is configured to store each of the plurality of analog photodetector signals in one of the plurality of storage units of the storage array in order to store each of the plurality of analog photodetector signals corresponding to charges from each of the plurality of photodetectors.

[0216]

[0237] Embodiment 26. The apparatus of Embodiment 25, wherein the image sensor is configured to reset multiple photodetectors of the image sensor and multiple memory units of the memory array after capturing an image.

[0217]

[0238] Embodiment 27. The apparatus of Embodiment 26, wherein one or more processors are configured to analyze an image using an object detection algorithm, and the image sensor is configured to automatically reset the multiple photodetectors of the image sensor and the multiple storage units of the storage array in response to the object detection algorithm not detecting one or more objects in the image, in order to reset the multiple photodetectors of the image sensor and the multiple storage units of the storage array after capturing an image.

[0218]

[0239] Embodiment 28. An apparatus according to any of Embodiments 1 to 27, wherein the image sensor is configured to reset each of the multiple photodetectors of the image sensor before exposing the multiple photodetectors to light from the scene.

[0219]

[0240] Embodiment 29. The apparatus of Embodiment 28, wherein the image sensor is configured to move a reset pointer from a first side of the image sensor to a second side of the image sensor at a predetermined speed across a plurality of reset pointer positions, and is configured to reset each of a plurality of photodetectors in response to the reset pointer reaching one of a plurality of reset pointer positions where the first photodetector of the plurality of photodetectors is located, and a read pointer is configured to move from a first side of the image sensor to a second side of the image sensor at a predetermined speed across a plurality of read pointer positions, and is configured to read a third digital pixel data corresponding to the second photodetector of the first photodetector group in response to the read pointer reaching one of a plurality of reset pointer positions where the second photodetector of the first photodetector group is located, in order to read a first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first photodetector group, and the predetermined speed is associated with a rolling shutter.

[0220]

[0241] Embodiment 30. An apparatus according to any of Embodiments 1 to 29, wherein one or more processors are configured to output an image.

[0221]

[0242] Embodiment 31. The apparatus of Embodiment 30, further comprising a display, wherein one or more processors are configured to display an image using the display in order to output an image.

[0222]

[0243] Embodiment 32. The apparatus of any embodiment 30 or 31, further comprising a communication transceiver, wherein one or more processors are configured to use the communication transceiver to transmit an image to a receiving device in order to output an image.

[0223]

[0244] Embodiment 33. An apparatus according to any of Embodiments 1 to 32, wherein the plurality of photodetectors of the image sensor include a plurality of image photodiodes and a plurality of focusing photodiodes, the plurality of focusing photodiodes are configured for phase detection autofocus (PDAF), the first group of photodetectors includes a first subset of the plurality of image photodiodes, and the second group of photodetectors includes a second subset of the plurality of image photodiodes.

[0224]

[0245] Embodiment 34. An apparatus according to any of Embodiments 1 to 33, wherein a plurality of photodetectors include a plurality of photodiodes.

[0225]

[0246] Embodiment 35. An apparatus according to any of Embodiments 1 to 34, wherein the apparatus includes at least one of a mobile handset, a wireless communication device, and a head-mounted display.

[0226]

[0247] Embodiment 36. A method for imaging an image sensor, comprising: exposing the image sensor to light from a scene, wherein the image sensor has a plurality of photodetectors, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other; storing a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors in the image sensor; reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors; and generating an image of a scene using at least the first digital pixel data.

[0227]

[0248] Embodiment 37. The method of Embodiment 36, wherein reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group without reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group includes selectively applying a modification to a first subset of multiple analog photodetector signals without applying a modification to a second subset of multiple analog photodetector signals.

[0228]

[0249] Embodiment 38. The method in which one analog photodetector signal among a plurality of analog photodetector signals indicates a voltage corresponding to a charge from one of the plurality of photodetectors, as in Embodiment 36 or 37.

[0229]

[0250] Embodiment 39. Any of embodiments 36 to 38, wherein the method is carried out by an imaging system including an image sensor.

[0230]

[0251] Embodiment 40. A method according to any of Embodiments 36 to 39, wherein reading first digital pixel data from a first subset of a plurality of analog photodetector signals includes having one or more amplifiers amplify each of the first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors into a plurality of amplified analog photodetector signals, and the first digital pixel data is based on the plurality of amplified analog photodetector signals.

[0231]

[0252] Embodiment 41. The method of Embodiment 40, wherein the method is carried out by an imaging system including one or more amplifiers.

[0232]

[0253] Embodiment 42. A method according to any of Embodiments 36 to 41, wherein reading first digital pixel data from a first subset of a plurality of analog photodetector signals includes causing one or more analog-to-digital converters (ADCs) to convert a plurality of analog signals based on a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors into digital signals, and the first digital pixel data is based on digital signals.

[0233]

[0254] Embodiment 43. The method of Embodiment 42, wherein the method is carried out by an imaging system including one or more ADCs.

[0234]

[0255] Embodiment 44. A method in any of Embodiment 42 or 43, wherein a plurality of analog signals include a plurality of amplified analog photodetector signals, and one or more amplifiers generate a plurality of amplified analog photodetector signals by at least partially amplified a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors.

[0235]

[0256] Embodiment 45. Any method of Embodiments 42 to 44, further comprising generating first digital pixel data based on a digital signal by processing the digital signal using one or more image processing operations, wherein one or more image processing operations include at least one of digital gain, demosaicing, pixel interpolation, missing pixel correction, bad pixel correction, brightness adjustment, contrast adjustment, saturation adjustment, histogram adjustment, color space conversion, automatic white balance adjustment, automatic black balance adjustment, downsampling, and upsampling.

[0236]

[0257] Embodiment 46. Any method of Embodiments 36 to 45, further comprising, after generating an image, determining whether to read a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second group of photodetectors based on the image.

[0237]

[0258] Aspect 47. A method according to any of aspects 36 to 46, further comprising: reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first photodetector group; then reading second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group; and using at least the first digital pixel data and the second digital pixel data to generate a second image of a scene, wherein the second resolution of the second image is higher than the first resolution of the image, wherein reading the second digital pixel data is based on the storage of a plurality of analog photodetector signals.

[0238]

[0259] Embodiment 48. The method of Embodiment 47, wherein after reading first digital pixel data from a first subset of multiple analog photodetector signals corresponding to a first photodetector group, and then reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, an image of the scene is generated, and then reading second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group.

[0239]

[0260] Aspect 49. The method of either aspect 47 or 48, further comprising identifying that an image includes an area having a splendor exceeding a splendor threshold, wherein reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group is, in response to identifying that an image includes an area having a splendor exceeding a splendor threshold, the method comprising reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group.

[0240]

[0261] Embodiment 50. A method of any embodiment 47 to 49, further comprising identifying that an image exhibits motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, and reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group in response to identifying that an image exhibits motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor.

[0241]

[0262] Embodiment 51. Any method of Embodiments 47 to 50, further comprising, after generating a second image, determining whether to read a third digital pixel data from a third subset of multiple analog photodetector signals corresponding to a third group of photodetectors based on the second image, wherein the multiple photodetectors include a third group of photodetectors different from the first and second groups of photodetectors, and the image and the second image are generated without the third digital pixel data.

[0242]

[0263] Aspect 52. A method according to any of aspects 47 to 51, further comprising: reading second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second photodetector group; then reading third digital pixel data from a third subset of a plurality of analog photodetector signals corresponding to a third photodetector group; and using at least first digital pixel data, second digital pixel data, and third digital pixel data to generate a third image of a scene, wherein the third resolution of the third image is higher than the second resolution of the second image, wherein reading the third digital pixel data is based on the storage of a plurality of analog photodetector signals, the plurality of photodetectors include a third photodetector group different from the first and second photodetector groups, and the image and the second image are generated without third digital pixel data.

[0243]

[0264] Embodiment 53. Any method of Embodiments 47 to 52, further comprising identifying that the image contains a depiction of an object, and reading a second digital pixel data from a second subset of multiple analog photodetector signals corresponding to a second photodetector group, in response to identifying that the image contains a depiction of an object.

[0244]

[0265] Embodiment 54. The method of Embodiment 53, wherein the object is a glyph that optically encodes information, and the glyph includes at least one of a one-dimensional barcode and a two-dimensional code.

[0245]

[0266] Embodiment 55. A method in either Embodiment 53 or 54, wherein the object includes at least a part of a person.

[0246]

[0267] Embodiment 56. Any method according to Embodiments 53 to 55, wherein the object includes at least a part of a vehicle.

[0247]

[0268] Embodiment 57. Any method from Embodiments 53 to 56, wherein the object contains one or more alphanumeric characters.

[0248]

[0269] Embodiment 58. The method according to any of Embodiments 36 to 57, wherein the plurality of photodetectors of the image sensor are arranged according to a grid of blocks, each of which includes a separate subset of the plurality of photodetectors, and each block of the grid of blocks includes at least one photodetector from a first group of photodetectors and at least one photodetector from a second group of photodetectors.

[0249]

[0270] Embodiment 59. Any method according to Embodiments 36 to 58, wherein a first group of photodetectors is arranged across an image sensor according to a first pattern, and a second group of photodetectors is arranged across an image sensor according to a second pattern, and the first pattern and the second pattern correspond to one or more grids.

[0250]

[0271] Embodiment 60. A method according to any of embodiments 36 to 59, wherein the image sensor includes a storage array having a plurality of storage units, and storing each of a plurality of analog photodetector signals corresponding to charges from each of a plurality of photodetectors includes storing each of the plurality of analog photodetector signals in one of the plurality of storage units of the storage array.

[0251]

[0272] Embodiment 61. The method of Embodiment 60, further comprising resetting a plurality of photodetectors of an image sensor and a plurality of memory units of a memory array after capturing an image.

[0252]

[0273] Embodiment 62. The method of Embodiment 61, further comprising analyzing an image using an object detection algorithm, and after capturing an image, resetting a plurality of photodetectors of an image sensor and a plurality of storage units of a storage array, which includes automatically resetting a plurality of photodetectors of an image sensor and a plurality of storage units of a storage array in response to the object detection algorithm not detecting one or more objects in the image.

[0253]

[0274] Embodiment 63. Any method of Embodiments 36 to 62, further comprising resetting each of the multiple photodetectors of an image sensor before exposing the multiple photodetectors to light from a scene.

[0254]

[0275] Embodiment 64. A method of Embodiment 63, further comprising: moving a reset pointer at a predetermined speed across a plurality of reset pointer positions from a first side of the image sensor to a second side of the image sensor, wherein resetting each of the plurality of photodetectors includes resetting the first photodetector of the plurality of photodetectors in response to the reset pointer reaching one of a plurality of reset pointer positions where the first photodetector of the plurality of photodetectors is located; and moving a read pointer at a predetermined speed across a plurality of read pointer positions from a first side of the image sensor to a second side of the image sensor, wherein reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors includes reading third digital pixel data corresponding to a second photodetector of the first group of photodetectors in response to the read pointer reaching one of a plurality of reset pointer positions where the second photodetector of the first group of photodetectors is located, wherein the predetermined speed is associated with a rolling shutter.

[0255]

[0276] Embodiment 65. Any method of Embodiments 36 to 64, further comprising outputting an image.

[0256]

[0277] Embodiment 66. The method of Embodiment 65, wherein outputting an image includes displaying the image using a display.

[0257]

[0278] Embodiment 67. Any method of Embodiment 65 or 66, wherein outputting an image includes transmitting the image to a receiving device using a communication transceiver.

[0258]

[0279] Embodiment 68. Any method according to Embodiments 36 to 67, wherein the plurality of photodetectors of the image sensor include a plurality of image photodiodes and a plurality of focusing photodiodes, the plurality of focusing photodiodes are configured for phase detection autofocus (PDAF), the first group of photodetectors includes a first subset of the plurality of image photodiodes, and the second group of photodetectors includes a second subset of the plurality of image photodiodes.

[0259]

[0280] Embodiment 69. Any method according to Embodiments 36 to 68, wherein the multiple photodetectors include multiple photodiodes.

[0260]

[0281] Embodiment 70. A non-temporary computer-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, the instructions cause one or more processors to expose an image sensor to light from a scene, wherein the image sensor has a plurality of photodetectors, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other; cause the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors; cause the image sensor to read first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second group of photodetectors; and cause the image sensor to generate an image of a scene using at least the first digital pixel data.

[0261]

[0282] Aspect 71. A non-temporary computer-readable medium of aspect 70, wherein the execution of an instruction by one or more processors causes one or more processors to further perform an operation according to any of aspects 2 to 35 and / or any of aspects 37 to 69.

[0262]

[0283] Embodiment 72. An imaging apparatus comprising: an image sensor, a plurality of photodetectors of the image sensor, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors comprising at least a first group of photodetectors and a second group of photodetectors that are different from each other, means for exposing the plurality of photodetectors to light from a scene; means for causing the image sensor to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors; means for reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors; and means for generating an image of a scene using at least the first digital pixel data.

[0263]

[0284] Apparatus 73. The apparatus of Apparatus 72, further comprising means for performing operations according to any of Apparatuses 2 to 35 and / or any of Apparatuses 37 to 69.

[0264]

[0285] Embodiment 74. An imaging apparatus comprising a memory and one or more processors coupled to the memory, wherein one or more processors are configured to expose an image sensor to a plurality of photodetectors of the image sensor, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, to light from a scene, to store a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, to read first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to a first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second group of photodetectors, to generate an image of a scene using at least the first digital pixel data, and after generating the image, to decide whether to read second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to a second group of photodetectors based on the image.

[0265]

[0286] Embodiment 75. The apparatus of Embodiment 74, wherein one or more processors are configured to perform one or more operations according to any of Embodiments 2 to 35.

[0266]

[0287] Embodiment 76. A method for imaging, comprising: exposing the image sensor to light from a scene, wherein the image sensor comprises a plurality of photodetectors, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors comprises at least a first group of photodetectors and a second group of photodetectors that are different from each other; storing a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors in the image sensor; reading first digital pixel data from a first subset of a plurality of analog photodetector signals corresponding to the first group of photodetectors without reading a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors; generating an image of a scene using at least the first digital pixel data; and, after generating the image, determining, based on the image, whether to read a second digital pixel data from a second subset of a plurality of analog photodetector signals corresponding to the second group of photodetectors.

[0267]

[0288] Embodiment 77. The method of Embodiment 76, further comprising one or more operations according to any of Embodiments 37 to 69.

[0268]

[0289] Embodiment 78. A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by one or more processors, the medium causes one or more processors to expose an image sensor to light from a scene, wherein the image sensor has a plurality of photodetectors, each of which is configured to convert light from a scene into charge in response to exposure to light from a scene, and the plurality of photodetectors include at least two distinct groups of first and second photodetectors, to light from a scene, the image sensor stores a plurality of analog photodetector signals corresponding to the charges from each of the plurality of photodetectors, the image sensor reads first digital pixel data from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group without reading second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the image sensor generates an image of a scene using at least the first digital pixel data, and after generating the image, the image sensor determines, based on the image, whether to read second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group.

[0269]

[0290] Aspect 79. A non-temporary computer-readable medium of Aspect 78, wherein the execution of an instruction by one or more processors causes one or more processors to further perform an operation according to any of Aspects 2 to 35 and / or any of Aspects 37 to 69.

[0270]

[0291] Aspect 80. An apparatus for imaging, comprising: means for causing an image sensor to expose a plurality of photodetectors of the image sensor to light from a scene, wherein each of the plurality of photodetectors is configured to convert light from the scene into electric charge in response to exposure to the light from the scene, and the plurality of photodetectors at least include a first photodetector group and a second photodetector group that are different from each other; means for causing the image sensor to store a plurality of analog photodetector signals corresponding to the electric charge from each of the plurality of photodetectors; means for reading first digital pixel data from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group without reading second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group; means for generating an image of the scene using at least the first digital pixel data; and means for deciding, after generating the image, whether to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group based on the image.

[0271]

[0292] The apparatus of aspect 80, further comprising means for performing the operation according to any one of aspects 2 to 35, and / or any one of aspects 33 to 69. The invention described in the original claims of this application is listed below. [C1] Memory and, One or more processors coupled to the memory, An imaging apparatus comprising, the one or more processors The image sensor includes a plurality of photodetectors, each of which is configured to convert the light from the scene into an electric charge in response to exposure to light from the scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, and the plurality of photodetectors are exposed to light from the scene. The image sensor is configured to store a plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors. Without reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the first digital pixel data is read from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group. The system is configured to generate an image of the scene using at least the first digital pixel data, Device. [C2] The apparatus according to C1, wherein one or more processors use an imaging circuit to selectively apply the modification to the first subset of the plurality of analog photodetector signals without applying the modification to the second subset of the plurality of analog photodetector signals, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, without reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group. [C3] The apparatus according to C1, wherein one of the plurality of analog photodetector signals indicates a voltage corresponding to the charge from one of the plurality of photodetectors. [C4] The image sensor The apparatus described in C1, further comprising the above. [C5] The apparatus according to C1, wherein, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals, one or more processors are configured to have one or more amplifiers amplify each of the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group into a plurality of amplified analog photodetector signals using the one or more amplifiers, and the first digital pixel data is based on the plurality of amplified analog photodetector signals. [C6] The one or more amplifiers The apparatus described in C5, further comprising the above. [C7] The apparatus according to C1, wherein, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals, one or more processors are configured to cause one or more analog-to-digital converters (ADCs) to convert a plurality of analog signals based on the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group into digital signals using the one or more ADCs, and the first digital pixel data is based on the digital signals. [C8] The one or more ADCs The apparatus described in C7, further comprising the above. [C9] The apparatus according to C7, wherein the plurality of analog signals include a plurality of amplified analog photodetector signals, and one or more amplifiers generate the plurality of amplified analog photodetector signals by at least partially amplifying a first subset of the plurality of analog photodetector signals corresponding to the first group of photodetectors. [C10] The one or more processors After generating the image, it is determined whether to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group based on the image. The apparatus described in C1, configured as follows. [C11] The one or more processors After reading the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, the second digital pixel data is read from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group. Using at least the first digital pixel data and the second digital pixel data, a second image of the scene is generated, wherein the second resolution of the second image is higher than the first resolution of the image. It is configured in such a way, Reading the second digital pixel data is based on the storage of the plurality of analog photodetector signals. The apparatus described in C1. [C12] The apparatus according to C11, wherein, after reading the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, one or more processors are configured to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group after generating the image of the scene. [C13] The one or more processors The apparatus according to C11, configured to identify that the image includes an area having a splendor exceeding a splendor threshold, and to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group in response to identifying that the image includes an area having a splendor exceeding the splendor threshold. [C14] The one or more processors The apparatus according to C11, wherein the image is configured to identify that the image shows motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, and in response to the image being configured to identify that the image shows motion exceeding the motion threshold relative to the previous image of the scene captured by the image sensor, the apparatus is configured to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group. [C15] The one or more processors The apparatus according to C11, wherein, after generating the second image, it is configured to determine whether to read third digital pixel data from a third subset of the plurality of analog photodetector signals corresponding to the third photodetector group based on the second image, wherein the plurality of photodetectors include a third photodetector group different from the first and second photodetector groups, and the image and the second image are generated without the third digital pixel data. [C16] The one or more processors After reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the third digital pixel data is read from the third subset of the plurality of analog photodetector signals corresponding to the third photodetector group. Using at least the first digital pixel data, the second digital pixel data, and the third digital pixel data, a third image of the scene is generated, wherein the third resolution of the third image is higher than the second resolution of the second image. It is configured in such a way, The apparatus according to C11, wherein reading the third digital pixel data is based on the storage of the plurality of analog photodetector signals, the plurality of photodetectors include a third photodetector group different from the first photodetector group and the second photodetector group, and the image and the second image are generated without the third digital pixel data. [C17] The one or more processors The apparatus according to C11, configured to identify that the image includes a depiction of an object, and to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group in response to identifying that the image includes the depiction of the object. [C18] The apparatus according to C17, wherein the object is a glyph that optically encodes information, and the glyph includes at least one of a one-dimensional barcode and a two-dimensional code. [C19] The apparatus according to C17, wherein the object includes at least one of a person, a vehicle, and an alphanumeric character. [C20] The apparatus according to C1, wherein the plurality of photodetectors of the image sensor are arranged according to a grid of blocks, each of which includes a separate subset of the plurality of photodetectors, and each block of the grid of blocks includes at least one photodetector from the first group of photodetectors and at least one photodetector from the second group of photodetectors. [C21] The apparatus according to C1, wherein the first group of photodetectors is arranged across the image sensor according to a first pattern, and the second group of photodetectors is arranged across the image sensor according to a second pattern, and the first pattern and the second pattern correspond to one or more grids. [C22] The apparatus according to C1, wherein the image sensor includes a storage array having a plurality of storage units, and the image sensor is configured to store each of the plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors in one of the plurality of storage units of the storage array. [C23] The image sensor, The apparatus according to C22, configured to reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array after capturing the aforementioned image. [C24] The one or more processors described above The apparatus according to C23, configured to analyze the image using an object detection algorithm, and configured to automatically reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array in response to the object detection algorithm not detecting one or more objects in the image, in order to reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array after capturing the image. [C25] The image sensor, The apparatus according to C1, wherein each of the plurality of photodetectors of the image sensor is configured to be reset before the plurality of photodetectors are exposed to the light from the scene. [C26] The image sensor, The reset pointer is configured to move at a predetermined speed from the first side of the image sensor to the second side of the image sensor across a plurality of reset pointer positions, and in order to reset each of the plurality of photodetectors, the first photodetector of the plurality of photodetectors is reset in response to the reset pointer reaching one of the plurality of reset pointer positions where the first photodetector of the plurality of photodetectors is located. The read pointer is configured to move at a predetermined speed from the first side of the image sensor to the second side of the image sensor across a plurality of read pointer positions, and in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, the read pointer is configured to read the third digital pixel data corresponding to the second photodetector of the first photodetector group in response to the read pointer reaching one of the plurality of reset pointer positions where the second photodetector of the first photodetector group is located. The apparatus according to C25, wherein the predetermined speed is associated with a rolling shutter. [C27] The one or more processors described above The apparatus described in C1, configured to output the aforementioned image. [C28] Display The system further comprises, and in order to output the image, one or more processors are configured to display the image using the display, The device described in C27. [C29] The apparatus according to C1, wherein the plurality of photodetectors of the image sensor include a plurality of image photodiodes and a plurality of focusing photodiodes, the plurality of focusing photodiodes are configured for phase detection autofocus (PDAF), the first group of photodetectors includes a first subset of the plurality of image photodiodes, and the second group of photodetectors includes a second subset of the plurality of image photodiodes. [C30] The image sensor includes a plurality of photodetectors, each of which is configured to convert the light from the scene into an electric charge in response to exposure to light from the scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, and the plurality of photodetectors are exposed to light from the scene. The image sensor is configured to store a plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors, Without reading a second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the first digital pixel data is read from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group. At least the first digital pixel data is used to generate an image of the scene, An imaging method, including the imaging method.

Claims

1. Memory and One or more processors coupled to the memory, An imaging apparatus comprising, the one or more processors The image sensor includes a plurality of photodetectors, each of which is configured to convert the light from the scene into an electric charge in response to exposure to light from the scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, and the plurality of photodetectors are exposed to light from the scene. The image sensor is configured to store a plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors. Without reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the first digital pixel data is read from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group. A first image of the scene is generated using at least the first digital pixel data, Identifying that the first image includes an area having a splendor that exceeds a splendor threshold, or identifying that the first image includes a depiction of an object, In response to identifying that the first image includes an area having a splendor exceeding a splendor threshold, or in response to identifying that the image includes a depiction of an object, the first digital pixel data is read from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, and then the second digital pixel data is read from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, wherein the reading of the second digital pixel data is based on the storage of the plurality of analog photodetector signals. Using at least the first digital pixel data and the second digital pixel data, a second image of the scene is generated, wherein the second resolution of the second image is higher than the first resolution of the first image. It is structured in such a way. Device.

2. The apparatus according to claim 1, wherein one or more processors use an imaging circuit to selectively apply a modification to the first subset of the plurality of analog photodetector signals without applying the modification to the second subset of the plurality of analog photodetector signals, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, without reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the modification comprising one or more processing operations by an image processor or filtering by one or more filters.

3. The apparatus according to claim 1, wherein one of the plurality of analog photodetector signals indicates a voltage corresponding to the charge from one of the plurality of photodetectors.

4. The aforementioned image sensor The apparatus according to claim 1, further comprising the following:

5. The apparatus according to claim 1, wherein, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals, one or more processors are configured to have one or more amplifiers amplify each of the first subsets of the plurality of analog photodetector signals corresponding to the first photodetector group into a plurality of amplified analog photodetector signals using the one or more amplifiers, and the first digital pixel data is based on the plurality of amplified analog photodetector signals.

6. The one or more amplifiers The apparatus according to claim 5, further comprising the following:

7. The apparatus according to claim 1, wherein, in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals, one or more processors are configured to cause one or more analog-to-digital converters (ADCs) to convert a plurality of analog signals based on the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group into digital signals using the one or more ADCs, and the first digital pixel data is based on the digital signals, and the apparatus further comprises one or more ADCs.

8. The apparatus according to claim 7, wherein the plurality of analog signals include a plurality of amplified analog photodetector signals, and one or more amplifiers generate the plurality of amplified analog photodetector signals by at least partially amplifying a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group.

9. The one or more processors described above The apparatus according to claim 1, further configured to identify that the first image shows motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor, and configured to read the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group in response to the identification that the first image shows motion exceeding a motion threshold relative to a previous image of the scene captured by the image sensor.

10. The one or more processors described above The system is configured to determine, after generating the second image, whether to read a third digital pixel data from a third subset of the plurality of analog photodetector signals corresponding to the third photodetector group, based on the second image, wherein the plurality of photodetectors include a third photodetector group different from the first and second photodetector groups, and the first and second images are generated without the third digital pixel data, or The one or more processors described above After reading the second digital pixel data from the second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the third digital pixel data is read from the third subset of the plurality of analog photodetector signals corresponding to the third photodetector group. Using at least the first digital pixel data, the second digital pixel data, and the third digital pixel data, a third image of the scene is generated, wherein the third resolution of the third image is higher than the second resolution of the second image. It is configured in such a way, The apparatus according to claim 1, wherein reading the third digital pixel data, based on the storage of the plurality of analog photodetector signals, the plurality of photodetectors include a third photodetector group different from the first photodetector group and the second photodetector group, and the first image and the second image are generated without the third digital pixel data.

11. The apparatus according to claim 1, wherein the first image includes a depiction of an object, the object is a glyph that optically encodes information, the glyph includes at least one of a one-dimensional barcode and a two-dimensional code, or the object includes at least one of a part of a person, a part of a vehicle, and alphanumeric characters.

12. The plurality of photodetectors of the image sensor are arranged according to a grid of blocks, each containing a separate subset of the plurality of photodetectors, and each block of the grid contains at least one photodetector from the first group of photodetectors and at least one photodetector from the second group of photodetectors, or The apparatus according to claim 1, wherein the first group of photodetectors is arranged across the image sensor according to a first pattern, and the second group of photodetectors is arranged across the image sensor according to a second pattern, and the first pattern and the second pattern correspond to one or more grids.

13. The image sensor includes a storage array comprising a plurality of storage units, and is configured to store each of the plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors in one of the plurality of storage units of the storage array, and is configured to reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array after capturing the first image, and one or more processors, The apparatus according to claim 1, further configured to analyze the first image using an object detection algorithm, wherein the image sensor is configured to automatically reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array in response to the object detection algorithm not detecting one or more objects in the first image, in order to reset the plurality of photodetectors of the image sensor and the plurality of storage units of the storage array after capturing the first image.

14. The aforementioned image sensor The image sensor is configured to reset each of the multiple photodetectors before exposing them to the light from the scene. The reset pointer is configured to move at a predetermined speed from the first side of the image sensor to the second side of the image sensor across a plurality of reset pointer positions, and in order to reset each of the plurality of photodetectors, the image sensor is configured to reset the first photodetector of the plurality of photodetectors in response to the reset pointer reaching one of the plurality of reset pointer positions where the first photodetector of the plurality of photodetectors is located. The image sensor is configured to move a read pointer across a plurality of read pointer positions from the first side of the image sensor to the second side of the image sensor at a predetermined speed, and in order to read the first digital pixel data from the first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, the image sensor is configured to read the third digital pixel data corresponding to the second photodetector of the first photodetector group in response to the read pointer reaching one of the plurality of reset pointer positions where the second photodetector of the first photodetector group is located. The apparatus according to claim 1, wherein the predetermined speed is associated with a rolling shutter.

15. The one or more processors described above The device is configured to output the first image, and the device further comprises a display, and in order to output the first image, one or more processors are configured to use the display to display the first image. The apparatus according to claim 1.

16. The apparatus according to claim 1, wherein the plurality of photodetectors of the image sensor include a plurality of image photodiodes and a plurality of focusing photodiodes, the plurality of focusing photodiodes are configured for phase detection autofocus (PDAF), the first group of photodetectors includes a first subset of the plurality of image photodiodes, and the second group of photodetectors includes a second subset of the plurality of image photodiodes.

17. The image sensor includes a plurality of photodetectors, each of which is configured to convert the light from the scene into an electric charge in response to exposure to light from the scene, and the plurality of photodetectors include at least a first group of photodetectors and a second group of photodetectors that are different from each other, and the plurality of photodetectors are exposed to light from the scene. The image sensor is configured to store a plurality of analog photodetector signals corresponding to the charge from each of the plurality of photodetectors, Without reading a second digital pixel data from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, the first digital pixel data is read from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, At least the first digital pixel data is used to generate a first image of the scene, Identifying that the first image includes an area having a splendor exceeding a splendor threshold, or identifying that the first image includes a depiction of an object, In response to identifying that the first image includes an area having a splendor exceeding a splendor threshold, or in response to identifying that the image includes a depiction of an object, the first digital pixel data is read from a first subset of the plurality of analog photodetector signals corresponding to the first photodetector group, and then the second digital pixel data is read from a second subset of the plurality of analog photodetector signals corresponding to the second photodetector group, wherein the reading of the second digital pixel data is based on the storage of the plurality of analog photodetector signals. Using at least the first digital pixel data and the second digital pixel data, a second image of the scene is generated, wherein the second resolution of the second image is higher than the first resolution of the first image. An imaging method, including the imaging method.

Citation Information

Patent Citations

  • Image processing apparatus and image processing method

    JP2013009050A

  • Control device, control method, program, and electronic apparatus system

    JP2018160799A

  • Imaging apparatus and control method thereof

    JP2020014147A

  • Solid-state imaging device and electronic equipment

    JP2020048018A

  • CMOS image sensors with feature extraction

    US20140118592A1