Image sensor with multiple image readout

Abstract

Systems and techniques for imaging are described. For example, a process may include obtaining a first image captured using a first portion of an image sensor. The process may also include obtaining a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determining an image capture setting based on the second image; and applying the image capture setting to the first image.

Description

Technical Field

[0001] The present application generally relates to capturing and processing images. For example, aspects of the present application relate to image sensors having multiple image readouts. Background Art

[0002] Many devices and systems allow a scene to be captured by generating an image (or frame) and / or video data (including multiple frames). For example, a camera or a device including a camera (or multiple cameras) can capture a sequence of frames of a scene (e.g., a video of the scene) based on the light entering the camera. To improve the quality of the frames captured by the camera, the camera can include a lens to focus the light entering the camera. The sequence of frames captured by the camera can be output for display, can be output for processing and / or consumption by other devices, and for other purposes.

[0003] In recent years, the resolution of image sensors used to capture light for cameras has been increasing. This increased resolution allows certain features that utilize the increased resolution to be implemented. As an example, the increased resolution allows interpolation across multiple pixels to help improve image quality and / or sensitivity. As a more specific example, an image sensor with a native resolution of 48 megapixels can interpolate four pixels of the image sensor into a single pixel of a 12 megapixel image produced by the image sensor. In some cases, the increased resolution allows for high-quality in-sensor digital zoom to be implemented. In some cases, this in-sensor zoom can be performed using sensor cropping, where a portion of the sensor can send a cropped area of the sensor at a higher resolution and / or frame rate than is typically transmitted by the image sensor to help improve digital zoom. For example, in-sensor digital zoom can use the central portion of a 48 megapixel sensor at full resolution (e.g., without interpolation) for the zoomed image. Summary of the Invention

[0004] In some examples, systems and techniques for improved imaging processing (e.g., for capturing images) are described. For example, an imaging system can include an image sensor having multiple image readouts. In one illustrative example, an imaging device is provided. The imaging device includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first image captured using a first portion of the image sensor; obtain a second image captured using a second portion of the image sensor, where the second portion is different from the first portion and where the second image has a lower resolution than the first image; determine a white balance setting based on the second image; and apply the white balance setting.

[0005] In another example, a method for imaging. The method for imaging includes obtaining a first image captured using a first portion of an image sensor; obtaining a second image captured using a second portion of the image sensor, where the second portion is different from the first portion; determining an image capture setting based on the second image; and applying the image capture setting to the first image.

[0006] As another example, a non-transitory computer-readable medium storing instructions is provided, which when executed by at least one processor cause the at least one processor to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, where the second portion is different from the first portion; determine an image capture setting based on the second image; and apply the image capture setting to the first image.

[0007] In another example, an imaging device is provided. The imaging device includes: means for obtaining a first image captured using a first portion of an image sensor; means for obtaining a second image captured using a second portion of the image sensor, where the second portion is different from the first portion; means for determining an image capture setting based on the second image; and means for applying the image capture setting to the first image.

[0008] In some aspects, one or more of the devices described herein are part of or include one or more of the following: a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other devices. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the location and / or orientation of the device, the state of the device, and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood in reference to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent when reference is made to the following specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1A is a block diagram illustrating the architecture of an image capture and processing device according to some examples;

[0013] Figure 1B is a block diagram illustrating an additional architecture of an image capture and processing device according to some examples;

[0014] Figure 2A and Figure 2B are diagrams illustrating exemplary red, green, and blue (RGB) color filter arrays according to some examples;

[0015] Figure 3A and Figure 3B are examples of how zoom can change the field of view according to aspects of the present disclosure;

[0016] Figures 4A to 4C illustrates exemplary images generated by an image sensor having multiple image readouts according to aspects of the present disclosure;

[0017] Figure 5 is a block diagram of an exemplary image sensor having multiple image readouts according to aspects of the present disclosure;

[0018] Figure 6A and Figure 6B illustrate additional enhancements for multiple image readouts according to aspects of the present disclosure;

[0019] Figure 7 is a flowchart illustrating techniques for applying an image capture setting to an image sensor having multiple image readouts according to aspects of the present disclosure;

[0020] Figure 8 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0021] Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that the various embodiments may be practiced without these specific details. The drawings and the description are not intended to be restrictive.

[0022] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0023] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image", "image frame", and "frame" may be used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, aperture value, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to the image sensor used to capture one or more image frames. Other camera settings can configure the post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, levels, curves, or color. For example, settings or parameters can be applied to a processor (e.g., an image signal processor (ISP)) used to process one or more image frames captured by the image sensor.

[0024] An example of an image setting includes white balance. For example, in some cases, the color temperature of the light source for a scene can make the image of the scene appear to have a certain color cast. White balance can help correct the color cast and make the image look more natural. Some imaging devices can use automatic white balance (AWB) to help adjust the white balance of an image based on the color temperature of the light source illuminating the scene. Another example of an image setting includes automatic exposure control (AEC) for controlling the exposure of the image sensor when capturing an image based on a previously captured image.

[0025] In some cases, image settings (e.g., AWB, AEC, etc.) can operate based on a reference region in an image. In certain cases, digital zoom of an imaging device can narrow the field of view of an image sensor (e.g., perform in-sensor zoom through sensor cropping), making it more difficult to locate a reference region in the image for applying image settings (e.g., AWB, AEC, etc.). In cases where multiple images or light sensors are available, other images or light sensors can be used for image settings (e.g., AWB, AEC, etc.). However, the latest trend in image sensors can use a single extremely high-resolution image sensor to perform digital zoom by imaging a portion (e.g., the central portion of the image sensor) of the image sensor at the full resolution available to the image sensor. Regions outside this central portion of the image sensor can be cropped, and this cropping results in a narrower field of view for the central portion. In some cases, it may be more difficult to apply image settings (e.g., perform AWB, AEC, etc.) on this narrow field of view. Techniques for improving image settings (e.g., AWB, AEC, etc.) for sensors that support in-sensor zoom may be helpful.

[0026] Systems, apparatuses, methods (also referred to as processes or techniques), and computer-readable media for providing an image sensor with multiple image readouts are described herein (collectively referred to as "systems and techniques" herein). For example, these systems and techniques can improve image settings (e.g., AWB, AEC, etc.) via an image sensor with multiple image readouts. AWB will be used herein as an illustrative example of an image setting that can be improved using the systems and techniques described herein. However, these systems and techniques can be used to improve other types of image settings, such as AEC.

[0027] According to some aspects, these systems and techniques can include receiving a first high-resolution image of a magnified portion of a scene and a second lower-resolution image of the scene from an image sensor. In some cases, the second image can be a full-field-of-view image of the scene. In other cases, the second image can be the portion of the scene outside the magnified portion of the scene. In cases where AWB cannot be accurately determined for the first image, AWB can be performed on the second image to determine a white balance setting, and this white balance setting can be used for the first image. In some cases, AWB can be performed concurrently on the first image and the second image. In some cases, the second image can be subsampled. For example, every other row or column can be subsampled by the image sensor for generating the second image.

[0028] The systems and techniques described herein provide advantages over existing solutions. For example, by sending a lower resolution second image in addition to a higher resolution first image of the magnified portion of the scene, the amount of bandwidth used to send the images is reduced compared to sending a higher resolution image of the full field of view. Additionally, when the white balance setting cannot be determined from the magnified narrow field of view image, the second image allows the white balance setting to be determined based on the full field of view as needed. Further, determining image settings (e.g., white balance setting, exposure setting, etc.) based on a full field of view image captured concurrently by a single sensor as an image having less than the full field of view (e.g., the magnified portion) allows for more potential references for applying the image settings (e.g., performing AWB, performing AEC, etc.) without using additional sensors or processing based on images captured at different points in time.

[0029] Various aspects of the techniques described herein will be discussed below with respect to the respective figures. Figure 1A FIG. 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photos), and / or can capture video including multiple images (or video frames) in a particular sequence. The lens 115 of the image capture and processing system 100 faces the scene 110 and receives light from the scene 110. In some cases, the lens 115 and the image sensor 130 can be associated with an optical axis. In one illustrative example, the photosensitive area of the image sensor 130 (e.g., a photodiode) and the lens 115 can both be centered on the optical axis. The lens 115 bends the incident light from the scene 110 towards the image sensor 130. The light received by the lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some cases, the aperture can have a fixed size.

[0030] One or more control mechanisms 120 can control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. One or more control mechanisms 120 can include multiple mechanisms and components; for example, the control mechanism 120 can include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 can also include additional control mechanisms other than those illustrated, such as control mechanisms for controlling analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0031] The exposure control mechanism 125A of the control mechanism 120 can obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control the size of the aperture (e.g., aperture size or aperture value), the duration for which the aperture is open (e.g., exposure time or shutter speed), the duration for which the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting can be referred to as an image capture setting and / or an image processing setting.

[0032] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 130. In some cases, different photodiodes can be covered by different filters. In some cases, different photodiodes can be covered in different color filters and can thus measure light that matches the color of the filter covering the photodiode. Various color filter arrays can be used, including a Bayer filter array, a four-color filter array (also known as a four-color Bayer filter array or QCFA), and / or any other color filter array. For example, a Bayer filter includes a red filter, a blue filter, and a green filter, where each pixel of the image is generated based on red light data from at least one photodiode covered by the red filter, blue light data from at least one photodiode covered by the blue filter, and green light data from at least one photodiode covered by the green filter.

[0033] Figure 2A An example of a Bayer filter array 200 is shown. As Figure 2A illustrated, the Bayer filter array 200 includes a repeating pattern of R filters, B filters, and G filters. Figure 2B An example quaternary color filter array (QCFA) 220 is illustrated. As shown, the QCFA 220 includes a 2×2 (or “quaternary”) pattern of filters, including a 2×2 pattern of red (R) filters, a pair of 2×2 patterns of green (G) filters, and a 2×2 pattern of blue (B) filters. Figure 2A The pattern of the Bayer filter array 200 shown, Figure 2B the pattern of the QCFA 220 shown, any other color filter, or any combination thereof can be repeated for the entire array of photodiodes of a given image sensor 130.

[0034] Returning to Figure 1A, other types of color filters can use yellow, magenta, and / or cyan (also known as "emerald") filters to replace or supplement red, blue, and / or green filters. In some cases, some photodiodes can be configured to measure infrared (IR) light. In some specific implementations, the photodiodes that measure IR light may not be covered by any filter, thus allowing the IR photodiodes to measure both visible light (e.g., RGB or other colors) and IR light. In some examples, the IR photodiodes may be covered by an IR filter, thus allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, colors). Some image sensors (e.g., image sensor 130) may lack filters entirely (e.g., colors, IR, or any other part of the spectrum), and instead may use different photodiodes (vertically stacked in some cases) throughout the pixel array.

[0035] Return to Figure 1A , in some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, the opaque and / or reflective mask can be used for phase detection autofocus (PDAF). In some cases, the opaque and / or reflective mask can be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cut-off filter, UV cut-off filter, band-pass filter, low-pass filter, high-pass filter, etc.). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output by the photodiodes (and / or the analog signal amplified by the analog gain amplifier) into a digital signal. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 can be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), complementary metal-oxide-semiconductor (CMOS), N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0036] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or with respect to Figure 8One or more processors in any other type of processor 810 discussed in computing system 800. The host processor 152 can be a digital signal processor (DSP) and / or other types of processors. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., BluetoothTM, Global Positioning System (GPS), etc.), any combination thereof and / or other components. The I / O port 156 can include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 Physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof and / or other input / output ports. In an illustrative example, the host processor 152 can communicate with the image sensor 130 using an I2C port, and the ISP 154 can communicate with the image sensor 130 using a MIPI port.

[0037] The image processor 150 can perform multiple tasks, such as demosaicking, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving inputs, managing outputs, managing memory, or some combination thereof. The image processor 150 can store the image frames and / or the processed images in a random access memory (RAM) 140 / 825, a read-only memory (ROM) 145 / 820, a cache, a memory cell, another storage device, or some combination thereof.

[0038] A variety of input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 835, any other input device 845, or some combination thereof. In some cases, captions may be input into the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via the virtual keyboard or keypad of the touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that implement a wired connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. The I / O devices 160 may include one or more wireless transceivers that implement a wireless connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. The peripheral devices may include any type of I / O device 160 discussed previously, and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 160.

[0039] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some specific implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some specific implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0040] As Figure 1A shown, the vertical dashed line divides Figure 1A the image capture and processing system 100 into two parts, representing the image capture device 105A and the image processing device 105B, respectively. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O device 160. In some cases, certain components illustrated in the image capture device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image capture device 105A.

[0041] Figure 1B Illustrates another example of the image capture and processing system 190 that includes Figure 1B the image processing device 105B and the image capture device 105C. In the illustrated example, the image capture device 105C includes the image sensor 130 and the control mechanism 120 described above with respect to Figure 1A as well as a second image sensor 180 and a control mechanism 170. The control mechanism 170 can be similar to Figure 1A the control mechanism 120 of Figure 1A and perform similar functions thereto. The exposure control mechanism 175A can be similar to Figure 1A the exposure control mechanism 125A of Figure 1A and perform similar functions thereto. The focus control mechanism 175B can be similar to Figure 1A the focus control mechanism 125B of

[0042] and perform similar functions thereto. The zoom control mechanism 175C can be similar to the zoom control mechanism 125C and perform similar functions thereto.

[0042] In some cases, the first image sensor 130 and the second image sensor 180 of the image capture device 105C, as well as the corresponding lenses 115, 165, can have at least partially overlapping fields of view. As described above, the image sensor 130 and the lens 115 can be associated with an optical axis, which is also referred to herein as the first optical axis. In some examples, the lens 165 and the second image sensor can be associated with a second optical axis. In one illustrative example, both the photosensitive area (e.g., photodiodes) of the second image sensor 180 and the lens 165 can be centered on the second optical axis. Other alignments between the image sensor 130 and the lens 115 and between the second image sensor 180 and the lens 165 can be used without departing from the scope of the present disclosure. In some cases, the first optical axis and the second optical axis can be physically close together (e.g., multiple cameras of an electronic device). In the illustrated example, the lens 165 can be a different type of lens than the lens 115. For example, as Figure 1B illustrated, the lens 115 can have a wide field of view 111 (e.g., from a wide-angle lens). In some cases, compared to the lens 115, the lens 165 can have a narrower field of view 161 (e.g., from a telephoto lens) and a higher magnification or zoom factor (as illustrated by the relatively large size of the person 163). In some cases, one of the image sensors 130 and the second image sensor 180 can be an RGB image sensor, and the other of the image sensors 130 and the second image sensor 180 can be an RGBIR sensor.

[0043] The image capture and processing system 100 and / or 190 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 and / or 190 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some embodiments, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A and / or the image capture device 105C may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing device.

[0044] Although the image capture and processing system 100 and the image capture and processing system 190 are shown as including certain components, those of ordinary skill in the art will understand that the image capture and processing system 100 and / or the image capture and processing system 190 may include Figure 1A and Figure 1B more components than those shown. The components of the image capture and processing system 100 and / or the image capture and processing system 190 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, the components of the image capture and processing system 100 and / or the image capture and processing system 190 may include an electronic circuit or other electronic hardware, and / or may be implemented using an electronic circuit or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a GPU, a DSP, a CPU, and / or other suitable electronic circuits); and / or these components may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image capture and processing system 100 and / or the image capture and processing system 190.

[0045] Automatic white balance (AWB) can be performed by an imaging device. Generally, different light sources may have different color temperatures, and these different color temperatures can cause a photograph to have a color cast. This color cast can make the photograph look more yellow or more blue compared to how the scene corresponding to the photograph may appear to the human eye. In some cases, AWB can be used to correct the color temperature to produce a photograph that looks more natural. In some cases, AWB can be performed (e.g.) by an ISP in conjunction with an image sensor. For example, AWB can compensate for different color temperatures by (e.g.) having components of the ISP analyze an image captured by the image sensor to identify gray areas. In some cases, AWB can apply one or more heuristics to the image data. For example, statistics can be collected at a low resolution for regions of the image to indicate the RGB levels in those regions. As a more specific example, the full image can be divided into a grid of 64x48 regions, and for each cell of the grid, the ratio between red and green and the ratio between blue and green can be calculated. Regions that are generally less saturated can be assumed to be gray, for example, based on statistical heuristics (e.g., using a set of look-up tables that include the ratio between blue and green and the ratio between red and green, which indicate the probability that these ratios correspond to possible light sources). Once the gray areas are identified, the color temperature can be estimated based on the gray areas and the possible light source of the image.

[0046] In some cases, if a reference region cannot be found in the image, or if there is no suitable gray area for reference, then AWB may not be successful. As a more specific example, an image of a clear sky may not have a good gray reference. In this case, AWB may not be able to distinguish whether the sky is blue at daylight color temperature or whether the sky is a uniform gray and cloudy at a shady color temperature. In some cases, when AWB cannot determine a gray reference, AWB can fallback to a default illuminance estimate.

[0047] Figure 3A and Figure 3B are examples of how zoom can change the field of view in accordance with aspects of the present disclosure. Figure 3A Illustrates an example full field of view image 300 of a scene. In image 300, a gray reference can be identified by AWB, for example, in the shadow region 302. In some cases, zoom can pose a challenge to AWB because zoom can be used to get closer to an object in the image and thus narrow the field of view. This narrowed field of view may not have a gray reference. For example, Figure 3B Illustrates an example image 350 of a magnified view of a scene. As shown in image 350, when magnified, the field of view narrows, and it may be difficult to detect a gray reference in image 350.

[0048] In some cases, some devices can perform digital zoom, where the zoom is performed using an image sensor rather than using optics. For example, certain image sensors may be able to perform in-sensor zoom, where a high-resolution sensor such as a 48-megapixel sensor (or higher) can sense and / or transmit a cropped region (e.g., a magnified region) at a higher resolution or frame rate. For example, when configured to provide a full field of view image (e.g., without applying zoom), the sensor can produce a 12-megapixel image, where each pixel of the image can be generated based on data from multiple photodiodes of the image sensor. However, when configured to sense a magnified region (e.g., a magnified region beyond a specific magnification level, such as magnification levels of 2x, 2.5x, 3x, etc.), the sensor can sense the magnified region at a higher resolution (e.g., at the full resolution of the image sensor), such that each pixel of the magnified image can be generated by a single photodiode of the image sensor (e.g., corresponding to a 48-megapixel full field of view image). This increased resolution or frame rate for the magnified image can be used to generate a higher quality image, which can be magnified for digital zoom. In some cases, when sensing a magnified region at a higher resolution, the sensor can provide image data only from the magnified region to the ISP. Providing image data only from the magnified region helps save bandwidth, as other regions may not be used to generate the picture. In some cases, the image data provided by the image sensor can be enhanced to provide variable resolution image data, allowing for increased AWB performance. It is noted that while discussed in the context of white balance and AWB, it is understood that the techniques discussed herein can be applicable to other image capture settings, such as exposure (e.g., automatic exposure control) and contrast (e.g., automatic contrast).

[0049] Figures 4A to 4C An example image produced by an image sensor with multiple image readouts in accordance with aspects of the present disclosure is illustrated. In some cases, when performing in-sensor zoom, the image sensor can transmit the image data of a first image 400 corresponding to a magnified region to the ISP at a relatively high resolution compared to a typical full field of view image captured by the image sensor, e.g., via a Mobile Industry Processor Interface (MIPI) bus. That is, the first image 400 can be captured at a higher effective resolution compared to a typical image captured by the sensor (e.g., by having more pixels (e.g., higher resolution density) when compared to a typical image of scaled dimensions). In some cases, the first image 400 can be captured and transmitted at the full resolution of the image sensor. In addition to the relatively high-resolution first image 400, the image sensor can also transmit Figure 4A of the first image 400 corresponding to the magnified region to the ISP. That is, the first image 400 can be captured at a higher effective resolution compared to a typical image captured by the sensor (e.g., by having more pixels (e.g., higher resolution density) when compared to a typical image of scaled dimensions). In some cases, the first image 400 can be captured and transmitted at the full resolution of the image sensor. In addition to the relatively high-resolution first image 400, the image sensor can also transmit Figure 4Bimage data of a second image 430 at a lower resolution (e.g., effective resolution), compared to the first image 400, the second image 430 at the lower resolution may have a wider field of view. Thus, compared to the first image 400, Figure 4B the second image 430 may have a lower effective resolution because the second image 430 may have a lower pixel density, resulting in an image with fewer pixels compared to the first image 400 when scaled to the same size. In some cases, the field of view of the second image 430 may coincide with the full (or nearly full) field of view of the image sensor. In some cases, the lower resolution second image 430 and the first image 400 may be captured in response to a single capture command. In some cases, the lower resolution second image 430 and the first image 400 may be generated based on a single readout pass of the photodiodes of the image sensor.

[0050] In some cases, this second image 430 may be used for AWB. For example, if desired, an AWB algorithm may be used to find a gray area of the second image 430 as a reference for determining white balance. The determined white balance may also be used as the white balance for the first image 400. For example, AWB may first be run on the first image 400. If there is no suitable gray area (e.g., a statistically likely gray area) for reference (e.g., if the AWB algorithm attempts to fallback to a default white balance), then AWB may be performed on the second image 430. In other cases, AWB may be performed on both the first image 400 and the second image 430, and if the white balance can be appropriately determined for the first image 400, then that white balance may be used. If the white balance cannot be appropriately determined for the first image 400, then the white balance for the second image 430 may be used. In some cases, compared to the first image 400, the second image 430 may have a larger field of view. In some cases, the second image has a field of view that coincides with the entire (or nearly entire) field of view available to the image sensor.

[0051] In some cases, the resolution of the second image 430 is less than (e.g., lower pixel density, lower effective resolution) another resolution that the image sensor would use for a full field of view image. For example, if the sensor typically generates a 12 megapixel image for a full field of view image (e.g., unzoomed or corresponding to a 1x magnification level), then the second image 430 may have a resolution less than 12 megapixels (e.g., one megapixel). In some cases, the resolution of the second image 430 may be significantly lower than that of the first image 400. For example, the second image 430 may have a resolution of 64x48 or lower. Since the second image 430 has a relatively low resolution, the total data transmission bandwidth for the image sensor to transmit the first image 400 and the second image 430 can be much lower than the total data transmission bandwidth for transmitting the entire field of view image at the same resolution as the first image 400. In some cases, multiple second images 430 may be transmitted. In some cases, the second image 430 may be generated by subsampling the image sensor outside the magnified region.

[0052] In some cases, as Figure 4C shown, rather than transmitting a second image 430 that has a larger field of view than the first image 400, the image sensor may send the high resolution first image 400 and relatively low resolution (e.g., lower pixel density, lower effective resolution) image data for one or more additional images 450 of the peripheral region around the high resolution region where the first image 400 is captured. In some cases, the peripheral region is adjacent to (e.g., immediately adjacent to) the high resolution region where the first image 400 is captured. In Figure 4CIn this case, four additional images 450 are sent, an image of the peripheral region 452A above the high-resolution region of the first image 400, an image of the peripheral region 452B to the left of the high-resolution region of the first image 400, an image of the peripheral region 452C to the right of the high-resolution region of the first image 400, and an image of the peripheral region 452D below the high-resolution region of the first image 400. In some cases, the additional images 450 may be captured / sent (e.g., sent to the ISP) at a pixel density lower than another pixel density that the image sensor would use for the full-field-of-view image (e.g., to produce a lower-resolution image). Compared to the magnified first image 400, the second image 430 and the additional images 450 provide a larger field of view that can be used for AWB. Once the white balance setting (e.g., the image capture setting) is determined based on the second image 430 and / or the additional images 450, the white balance setting can be applied to the first image 400 (e.g., by the ISP). In some examples, the white balance setting (or other image capture setting) can be applied in substantially the same manner as the white balance setting would be applied in the case where the white balance setting is determined based on the first image 400 using any technique for adjusting the white balance (or other image capture setting) of an image. In some cases, the additional images 450 and the first image 400 may also be captured in response to a single capture command. In some cases, the additional images 450 and the first image 400 may be generated based on a single read-through of the photodiodes of the image sensor.

[0053] Figure 5 is a block diagram of an example image sensor 500 with multiple image readouts according to aspects of the present disclosure. In some cases, the image sensor 500 may include an array of photodiodes 502 for sensing light, as well as a high-resolution image buffer 504 and a low-resolution image buffer 506. The image buffers may store image data read from the photodiode array 502 before being sent to, for example, an ISP (not shown). In some cases, when in-sensor zoom is being used, the high-resolution image buffer 504 may store image data captured from a zoomed-in region at a high resolution (e.g., at a higher image density). For example, Figure 4A the image data of the first image 400 may be stored in the high-resolution image buffer 504. In some cases, the low-resolution image buffer 506 may store lower-resolution image data, such as image data for a lower-resolution, full-field-of-view image. For example, Figure 4B the image data of the second image 430 or the peripheral region around the high-resolution zoomed-in region (such as, Figure 4CThe image data of the additional image 450) can be stored in the low-resolution image buffer 506. In some cases, multiple low-resolution image buffers 506 can be used for individual images of the peripheral regions around the high-resolution zoomed-in regions, such as for Figure 4C a separate buffer for the image of the additional image 450.

[0054] In some cases, the operation of an image sensor capable of performing multiple image readouts can be enhanced to help reduce power consumption and processing time. In some cases, an image sensor may require a certain amount of power, time, and / or bandwidth to sample the entire field of view of the photodiode array, especially at full resolution. Since Figure 4B the second image 430 or Figure 4C the additional image 450 can be performed at a lower resolution, it can be beneficial for the image sensor to apply subsampling in regions outside the high-resolution region used to generate the first image 400.

[0055] Figure 6A and Figure 6B illustrate additional enhancements for multiple image readouts according to aspects of the present disclosure. In some cases, when performing a zoom with multiple image readouts, the image sensor can skip sampling certain sampled pixels. In some cases, the skipped pixels can be located in regions that do not need to be sampled at a relatively high resolution, such as peripheral regions outside the high-resolution zoomed-in region. Figure 6A The image 600 represents a scanned portion of the photodiode array of the image sensor. In the image 600, the peripheral regions 602 above and below the high-resolution region 604 can be subsampled by skipping the sampling of certain rows (e.g., every other row, every two rows, or another such similar pattern). In the image 600, the peripheral regions to the right and left of the high-resolution region 604 may also be scanned at high resolution. In some cases, such as in Figure 6B the image 650, the peripheral regions 652 above, below, to the left, and to the right of the high-resolution region 654 can be subsampled by skipping the sampling of certain rows. In some cases, the image sensor can subsample the peripheral regions by skipping (e.g., merging) certain columns in addition to or instead of skipping rows.

[0056] In some cases, instead of sending the peripheral regions to the ISP, the image sensor can perform some white balance processing and send information about the white balance to the ISP. For example, the image sensor can analyze the peripheral regions as discussed above and transmit the white balance calibration (WBC) scaler and / or estimated color temperature via a separate metadata channel (e.g., a separate virtual channel from the image data, where the virtual channel can be separated by a virtual channel number, different ports, etc.).

[0057] In some cases, image data may be sent from an image sensor to an ISP over multiple virtual channels (e.g., MIPI channels). For example, image data for a high-resolution region may be sent via a first virtual channel, and image data for a peripheral region (or a relatively lower-resolution full field of view image) may be sent over one or more other virtual channels. In some cases, these virtual channels may operate in an interleaved fashion, where information is sent via the first virtual channel, then from the other virtual channels, then from the first virtual channel, and so on. In some cases, the image data for the high-resolution region may be sent before the image data for the peripheral region (or the relatively lower-resolution full field of view image). The image data from the peripheral region (or the relatively lower-resolution full field of view image) may be of a lower resolution and smaller than the image data from the high-resolution region. Thus, in some cases, the image data from the peripheral region may be stored in memory while the image data from the high-resolution region is sent to the ISP. For example, the image sensor may read image data from a photodiode array in a raster scan order from right to left and top to bottom. The image sensor may initially read image data (or image data for a relatively lower-resolution full field of view image) from photodiodes in the peripheral region at a lower resolution (optionally using subsampling). The image sensor may store the image data from the peripheral region. When the image sensor reaches the high-resolution region, the image sensor may read image data from the photodiodes at a higher resolution (e.g., at full resolution, reading image data from each photodiode individually without interpolation, binning, etc.). The image sensor may stream the image data from the high-resolution region to the ISP. When the image sensor reaches another peripheral region, the image sensor may switch back to reading and storing the image data at a lower resolution. This pattern may continue until the image data readout is complete. The stored image data from the peripheral region may then be transferred to the ISP.

[0058] Figure 7 is a flowchart illustrating process 700 for applying an image capture setting to an image sensor having multiple image readouts in accordance with aspects of the present disclosure. Process 700 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other types of computing devices. The operations of process 700 may be implemented as software components executed and run on one or more processors.

[0059] At block 702, a computing device (or its component) may obtain a first image captured using a first portion of an image sensor. In some cases, the first image is received separately from a second image. In some cases, the computing device (or its component) may send a capture command to the image sensor, and wherein the first image and the second image are received in response to the capture command. In some cases, the first image and the second image are received based on a single read of the image sensor.

[0060] At block 704, the computing device (or its component) may obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion. In some cases, these portions may be different because not all pixels sensed for the first image by the image sensor may be sensed for the second image. In some examples, the portions may partially overlap (e.g., some pixels of the image sensor may be sensed for both the first portion and the second portion, e.g., if the second image includes a portion of the first image, as partially overlapping in Figure 4B second image 430 of Figure 4A and first image 400 of Figure 4A or the portions may not overlap (e.g., as non - overlapping in additional image 450 of Figure 4A and first image 400). In some examples, the second image is captured at a lower pixel density. In some cases, the image capture settings include one of a white balance setting or an exposure setting. In some cases, the second image has a wider field of view than the first image. In some cases, the second image includes the full field of view of the image sensor. In some cases, the second portion is adjacent to the first portion. In some cases, the computing device (or its component) may determine that a white balance setting cannot be determined for the first image. In some cases, the computing device (or its component) may determine a white balance setting for the first image based on the second image, based on determining that a white balance setting cannot be determined for the first image. In some cases, to determine that a white balance setting cannot be determined for the first image, the computing device (or its component) may attempt to determine a white balance setting based on the first image; and determine that a reference region cannot be found in the first image. In some cases, the second portion of the image sensor is separate from the first portion of the image sensor. In some cases, the second image is subsampled from the second portion. In some cases, the second image is subsampled by skipping at least one of a row or a column of photodiodes of the image sensor.

[0061] At block 706, the computing device (or its component) may determine image capture settings based on the second image. At block 708, the computing device (or its component) may apply the image capture settings to the first image. In some cases, the computing device (or its component) may apply a white balance setting determined based on the second image to the first image.

[0062] Figure 8 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 8 illustrates an example of a computing system 800, which can be any computing device that constitutes, for example, an internal computing system, a remote computing system, a camera, or any component thereof, where the components of the system communicate with each other using connection 805. Connection 805 can be a physical connection using a bus or a direct connection to the processor 810, such as in a chipset architecture. Connection 805 can also be a virtual connection, a networking connection, or a logical connection.

[0063] In some embodiments, the computing system 800 is a distributed system, where the functions described in this disclosure can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent many such components that each perform some or all of the functions for which the component is described. In some embodiments, the components can be physical devices or virtual devices.

[0064] The example system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including a system memory 815 (such as a read-only memory (ROM) 820 and a random access memory (RAM) 825), to the processor 810. The computing system 800 can include a cache 812 that is directly connected to, in close proximity to, or integrated as part of the processor 810 for high-speed memory.

[0065] The processor 810 can include any general-purpose processor and hardware services or software services, such as services 832, 834, and 836 stored in a storage device 830, which are configured to control the processor 810 and a dedicated processor in which software instructions are incorporated into the actual processor design. The processor 810 can essentially be a completely independent computing system that includes multiple cores or processors, buses, memory controllers, caches, etc. The multi-core processor can be symmetric or asymmetric.

[0066] To enable user interaction, computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 800 may also include an output device 835 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with computing system 800. Computing system 800 may include a communication interface 840 that generally may govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using audio jack / plug, microphone jack / plug, universal serial bus (USB) port / plug, port / plug, Ethernet port / plug, fiber optic port / plug, dedicated wired port / plug, wireless signaling, low-power (BLE) wireless signaling, wireless signaling, radio frequency identification (RFID) wireless signaling, near field communication (NFC) wireless signaling, dedicated short range communication (DSRC) wireless signaling, 802.11 Wi-Fi wireless signaling, wireless local area network (WLAN) signaling, visible light communication (VLC), worldwide interoperability for microwave access (WiMAX), infrared (IR) communication wireless signaling, public switched telephone network (PSTN) signaling, integrated services digital network (ISDN) signaling, 3G / 4G / 5G / LTE cellular data network wireless signaling, ad hoc network signaling, radio wave signaling, microwave signaling, infrared signaling, visible light signaling, ultraviolet light signaling, wireless signaling along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of computing system 800 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operating on any particular hardware arrangement, and thus the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0067] The storage device 830 can be a non-volatile and / or non-transitory and / or computer-readable memory device, and can be a hard disk or other types of computer-readable media that can store data accessible by a computer, such as a tape cassette, a flash memory card, a solid-state memory device, a digital versatile disc, a cassette tape, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, a digital video disc (DVD) optical disc, a Blu-ray disc (BDD) optical disc, a holographic optical disc, another optical medium, a Secure Digital (SD) card, a micro Secure Digital (microSD) card, a card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge and / or a combination thereof.

[0068] The storage device 830 can include software services, servers, services, etc., and when the code defining such software is executed by the processor 810, the code causes the system to perform functions. In some embodiments, the hardware services for performing specific functions can include software components for performing the functions stored in a computer-readable medium connected to necessary hardware components such as the processor 810, the connection 805, the output device 835, etc.

[0069] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. A computer-readable medium may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. By passing and / or receiving information, data, arguments, parameters, or memory contents, a code segment can be coupled to another code segment or hardware circuit. Information, arguments, parameters, data, etc. can be transferred, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0070] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bitstreams, etc. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0071] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, one of ordinary skill in the art will understand that the embodiments may be practiced without these specific details. For clarity of illustration, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks containing devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown in block diagram form as components to avoid obscuring these embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without necessary detail to avoid obscuring the embodiments.

[0072] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process is terminated when the operations of the process are completed, but the process may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0073] The processes and methods according to the above examples can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store the instructions, the information used, and / or the information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0074] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may execute the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By additional example, such functionality may also be implemented on a circuit board among different chips or different processes executed on a single device.

[0075] Instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0076] In the foregoing description, aspects of the present application have been described with reference to specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be construed to cover such variations, unless limited by the prior art. The various features and aspects of the foregoing applications can be used singly or in combination. In addition, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications beyond those described herein. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive. For purposes of illustration, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods can be performed in a different order than that described.

[0077] One of ordinary skill in the art will understand that the less than (“<”) and greater than (“>”) symbols or terms used herein can be replaced with less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively, without departing from the scope of this specification.

[0078] In cases where a component is described as “configured to” perform certain operations, such a configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform the operations, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0079] The phrase “coupled to” means that any component is directly or indirectly physically connected to another component, and / or any component directly or indirectly communicates with another component (e.g., is connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0080] Claim language or other language reciting “at least one of” a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0081] The various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0082] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device such as a cellular phone, or an integrated circuit device having multiple uses, including applications in wireless communication devices such as cellular phones and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be at least partially implemented by a computer-readable data storage medium including program code, the program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be at least partially implemented by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0083] The program code can be executed by a processor, which can include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor; but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Thus, the term "processor" as used herein can refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0084] Exemplary aspects of the present disclosure include:

[0085] Aspect 1. An imaging device, the imaging device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion, and wherein the second image has a lower resolution than the first image; determine a white balance setting based on the second image; and apply the white balance setting.

[0086] Aspect 2. The imaging device according to aspect 1, wherein the second image has a wider field of view than the first image.

[0087] Aspect 3. The imaging device according to any one of aspects 1 or 2, wherein the second image includes a full field of view of the image sensor.

[0088] Aspect 4. The imaging device according to any one of aspects 1 to 3, wherein the second portion is adjacent to the first portion.

[0089] Aspect 5. The imaging device according to any one of aspects 1 to 4, wherein the at least one processor is further configured to determine that a white balance setting cannot be determined for the first image.

[0090] Aspect 6. The imaging device according to aspect 5, wherein the at least one processor is configured to determine the white balance setting based on the second image based on determining that the white balance setting cannot be determined for the first image.

[0091] Aspect 7. The imaging device according to any one of Aspects 5 or 6, wherein, in order to determine that the white balance setting cannot be determined for the first image, the at least one processor is further configured to: attempt to determine the white balance setting based on the first image; and determine that a reference region cannot be found in the first image.

[0092] Aspect 8. The imaging device according to any one of Aspects 1 to 7, wherein the second part of the image sensor is separate from the first part of the image sensor.

[0093] Aspect 9. The imaging device according to any one of Aspects 1 to 8, wherein the second image is subsampled from the second part.

[0094] Aspect 10. The imaging device according to any one of Aspects 1 to 9, wherein the second image is subsampled by skipping at least one of the rows or columns of the photodiodes of the image sensor.

[0095] Aspect 11. The imaging device according to any one of Aspects 1 to 10, wherein the first image and the second image are received separately.

[0096] Aspect 12. The imaging device according to any one of Aspects 1 to 11, wherein the at least one processor is further configured to send a capture command to the image sensor, and wherein the first image and the second image are received in response to the capture command.

[0097] Aspect 13. The imaging device according to any one of Aspects 1 to 12, wherein the first image and the second image are received based on a single read of the image sensor.

[0098] Aspect 14. The imaging device according to any one of Aspects 1 to 13, wherein the at least one processor is configured to apply the white balance setting determined based on the second image to the first image.

[0099] Aspect 15. A method for imaging, the method comprising: obtaining a first image captured using a first part of an image sensor; obtaining a second image captured using a second part of the image sensor, wherein the second part is different from the first part, and wherein the second image has a lower resolution than the first image; determining a white balance setting based on the second image; and applying the white balance setting.

[0100] Aspect 16. The method according to Aspect 15, wherein the second image has a wider field of view than the first image.

[0101] Aspect 17. The method according to any one of Aspects 15 or 16, wherein the second image includes the full field of view of the image sensor.

[0102] Aspect 18. The method according to any one of Aspects 15 to 17, wherein the second part is adjacent to the first part.

[0103] Aspect 19. The method according to any one of Aspects 15 to 18, the method further comprising determining that a white balance setting cannot be determined for the first image.

[0104] Aspect 20. The method according to Aspect 19, wherein determining the white balance setting based on the second image is based on determining that the white balance setting cannot be determined for the first image.

[0105] Aspect 21. The method according to any one of Aspects 19 or 20, wherein determining that a white balance setting cannot be determined for the first image includes: attempting to determine the white balance setting based on the first image; and determining that a reference region cannot be found in the first image.

[0106] Aspect 22. The method according to any one of Aspects 15 to 21, wherein the second part of the image sensor is separated from the first part of the image sensor.

[0107] Aspect 23. The method according to any one of Aspects 15 to 22, wherein the second image is subsampled from the second part.

[0108] Aspect 24. The method according to any one of Aspects 15 to 23, wherein the second image is subsampled by skipping at least one of the rows or columns of the photodiodes of the image sensor.

[0109] Aspect 25. The method according to any one of Aspects 15 to 24, wherein the first image and the second image are received separately.

[0110] Aspect 26. The method according to any one of Aspects 15 to 25, the method further comprising sending a capture command to the image sensor, and wherein the first image and the second image are received in response to the capture command.

[0111] Aspect 27. The method according to any one of Aspects 15 to 26, wherein the first image and the second image are received based on a single read of the image sensor.

[0112] Aspect 28. The method according to any one of Aspects 15 to 27, the method further comprising applying the white balance setting determined based on the second image to the first image.

[0113] Aspect 29. A non-transitory computer-readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion, and wherein the second image has a lower resolution than the first image; determine a white balance setting based on the second image; and apply the white balance setting.

[0114] Aspect 30. The non-transitory computer-readable medium according to aspect 29, wherein the second image has a wider field of view than the first image.

[0115] Aspect 31. The non-transitory computer-readable medium according to any one of aspects 29 or 30, wherein the second image includes the full field of view of the image sensor.

[0116] Aspect 32. The non-transitory computer-readable medium according to any one of aspects 29 to 31, wherein the second portion is adjacent to the first portion.

[0117] Aspect 33. The non-transitory computer-readable medium according to any one of aspects 29 to 32, wherein the instructions further cause the at least one processor to determine that a white balance setting cannot be determined for the first image.

[0118] Aspect 34. The non-transitory computer-readable medium according to aspect 33, wherein the instructions further cause the at least one processor to determine the white balance setting based on the second image based on determining that a white balance setting cannot be determined for the first image.

[0119] Aspect 35. The non-transitory computer-readable medium according to any one of aspects 33 or 34, wherein, in order to determine that a white balance setting cannot be determined for the first image, the instructions further cause the at least one processor to: attempt to determine the white balance setting based on the first image; and determine that a reference region cannot be found in the first image.

[0120] Aspect 36. The non-transitory computer-readable medium according to any one of aspects 29 to 35, wherein the second portion of the image sensor is separated from the first portion of the image sensor.

[0121] Aspect 37. The non-transitory computer-readable medium according to any one of aspects 29 to 36, wherein the second image is subsampled from the second portion.

[0122] Aspect 38. The non-transitory computer-readable medium according to any one of aspects 29 to 37, wherein the second image is subsampled by skipping at least one of the rows or columns of the photodiodes of the image sensor.

[0123] Aspect 39. The non-transitory computer-readable medium according to any one of aspects 29 to 38, wherein the first image and the second image are received separately.

[0124] Aspect 40. The non-transitory computer-readable medium according to any one of aspects 29 to 39, wherein the instructions further cause the at least one processor to send a capture command to the image sensor, and wherein the first image and the second image are received in response to the capture command.

[0125] Aspect 41. The non-transitory computer-readable medium according to any one of aspects 29 to 40, wherein the first image and the second image are received based on a single read of the image sensor.

[0126] Aspect 42. The non-transitory computer-readable medium according to any one of aspects 29 to 41, wherein the at least one processor is configured to apply the white balance setting determined based on the second image to the first image.

[0127] Aspect 43. An imaging device, the imaging device comprising components for obtaining a first image captured using a first portion of an image sensor; components for obtaining a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion, and wherein the second image has a lower resolution than the first image; components for determining a white balance setting based on the second image; and components for applying the white balance setting.

[0128] Aspect 44: The device according to aspect 43, the device further comprising one or more components for performing any of the operations according to aspects 16 to 28.

[0129] Aspect 45. An imaging device, the imaging device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determine an image capture setting based on the second image; and apply the image capture setting to the first image.

[0130] Aspect 46. The imaging device according to aspect 45, wherein the second image is captured at a lower pixel density.

[0131] Aspect 47. The imaging device according to any one of aspects 45 to 46, wherein the image capture setting includes one of a white balance setting or an exposure setting.

[0132] Aspect 48. The imaging device according to any one of aspects 45 to 47, wherein the second image has a wider field of view than the first image.

[0133] Aspect 49. The imaging device according to any one of aspects 45 to 48, wherein the second image includes the entire field of view of the image sensor.

[0134] Aspect 50. The imaging device according to any one of aspects 45 to 49, wherein the second part is adjacent to the first part.

[0135] Aspect 51. The imaging device according to any one of aspects 45 to 50, wherein the image capture setting includes a white balance setting, and wherein the at least one processor is further configured to: determine that the image capture setting cannot be appropriately determined for the first image; and based on the determination that the image capture setting cannot be appropriately determined for the first image, determine the image capture setting for the first image based on the second image.

[0136] Aspect 52. The imaging device according to aspect 51, wherein, in order to determine that the white balance setting cannot be appropriately determined for the first image, the at least one processor is further configured to: attempt to determine the white balance setting based on the first image; and determine that a reference region cannot be found in the first image.

[0137] Aspect 53. The imaging device according to any one of aspects 45 to 52, wherein the second image is subsampled from the second part by skipping at least one of the rows or columns of photodiodes of the image sensor.

[0138] Aspect 54. The imaging device according to any one of aspects 45 to 53, wherein the first image and the second image are received separately.

[0139] Aspect 55. The imaging device according to any one of aspects 45 to 54, wherein the at least one processor is further configured to send a capture command to the image sensor, wherein the first image and the second image are received in response to the capture command, and wherein the first image and the second image are received based on a single read of the image sensor.

[0140] Aspect 56. A method for imaging, the method comprising: obtaining a first image captured using a first portion of an image sensor; obtaining a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determining an image capture setting based on the second image; and applying the image capture setting to the first image.

[0141] Aspect 57. The method according to aspect 56, wherein the second image is captured at a lower pixel density.

[0142] Aspect 58. The method according to any one of aspects 56 to 57, wherein the image capture setting includes one of a white balance setting or an exposure setting.

[0143] Aspect 59. The method according to any one of aspects 56 to 58, wherein the second image has a wider field of view than the first image.

[0144] Aspect 60. The method according to any one of aspects 56 to 59, wherein the second image includes the full field of view of the image sensor.

[0145] Aspect 61. The method according to any one of aspects 56 to 60, wherein the second portion is adjacent to the first portion.

[0146] Aspect 62. The method according to any one of aspects 56 to 61, wherein the image capture setting includes a white balance setting, and the method further comprises: determining that the white balance setting cannot be appropriately determined for the first image; and determining the white balance setting for the first image based on the second image based on the determination that the white balance setting cannot be appropriately determined for the first image.

[0147] Aspect 63. The method according to aspect 62, wherein determining that the white balance setting cannot be appropriately determined for the first image includes: attempting to determine the white balance setting based on the first image; and determining that a reference region cannot be found in the first image.

[0148] Aspect 64. The method according to any one of aspects 56 to 63, wherein the second image is subsampled from the second portion by skipping at least one of the rows or columns of photodiodes of the image sensor.

[0149] Aspect 65. The method according to any one of aspects 56 to 64, wherein the first image and the second image are received separately.

[0150] Aspect 66. The method according to any one of aspects 56 to 65, the method further comprising sending a capture command to the image sensor, wherein the first image and the second image are received in response to the capture command, and wherein the first image and the second image are received based on a single read of the image sensor.

[0151] Aspect 67. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determine an image capture setting based on the second image; and apply the image capture setting to the first image.

[0152] Aspect 68. The non-transitory computer-readable medium according to aspect 67, wherein the second image is captured at a lower pixel density.

[0153] Aspect 69. The non-transitory computer-readable medium according to any one of aspects 67 to 68, wherein the image capture setting includes one of a white balance setting or an exposure setting.

[0154] Aspect 70. The non-transitory computer-readable medium according to any one of aspects 67 to 69, wherein the second image has a wider field of view than the first image.

[0155] Aspect 71. The non-transitory computer-readable medium according to any one of aspects 67 to 70, wherein the second image includes the full field of view of the image sensor.

[0156] Aspect 72. The non-transitory computer-readable medium according to any one of aspects 67 to 71, wherein the second portion is adjacent to the first portion.

[0157] Aspect 73. The non-transitory computer-readable medium according to any one of aspects 67 to 72, wherein the image capture setting includes a white balance setting, and wherein the instructions further cause the at least one processor to: determine that the white balance setting cannot be appropriately determined for the first image; and based on the determination that the white balance setting cannot be appropriately determined for the first image, determine the white balance setting for the first image based on the second image.

[0158] Aspect 74. The non-transitory computer-readable medium according to aspect 73, wherein, in order to determine that the white balance setting cannot be appropriately determined for the first image, the instructions further cause the at least one processor to: attempt to determine the white balance setting based on the first image; and determine that a reference region cannot be found in the first image.

[0159] Aspect 75. The non-transitory computer-readable medium according to any one of aspects 67 to 74, the non-transitory computer-readable medium further comprising instructions for causing the processor to perform any of the operations according to aspects 56 to 66.

[0160] Aspect 76. An imaging device, the imaging device comprising means for obtaining a first image captured using a first portion of an image sensor; means for obtaining a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; means for determining an image capture setting based on the second image; and means for applying the image capture setting to the first image.

[0161] Aspect 77: The apparatus according to aspect 43, the apparatus comprising one or more means for performing any of the operations according to aspects 56 to 66.

Claims

1. An imaging device, the imaging device comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: obtain a first image captured using a first portion of an image sensor; obtain a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determine an image capture setting based on the second image; and apply the image capture setting to the first image.

2. The imaging device according to claim 1, wherein the second image is captured at a lower pixel density.

3. The imaging device according to claim 1, wherein the image capture setting includes one of a white balance setting or an exposure setting.

4. The imaging device according to claim 1, wherein the second image has a wider field of view than the first image.

5. The imaging device according to claim 1, wherein the second image includes a full field of view of the image sensor.

6. The imaging device according to claim 1, wherein the second portion is adjacent to the first portion.

7. The imaging device according to claim 1, wherein the image capture setting includes a white balance setting, and wherein the at least one processor is further configured to: determine that the white balance setting cannot be appropriately determined for the first image; and based on determining that the white balance setting cannot be appropriately determined for the first image, determine the white balance setting for the first image based on the second image.

8. The imaging device according to claim 7, wherein, To determine that the white balance setting cannot be appropriately determined for the first image, the at least one processor is further configured to: attempt to determine the white balance setting based on the first image; and determine that a reference region cannot be found in the first image.

9. The imaging device according to claim 1, wherein the second image is subsampled from the second portion by skipping at least one of rows or columns of photodiodes of the image sensor.

10. The imaging device according to claim 1, wherein the first image and the second image are received separately.

11. The imaging device according to claim 1, wherein the at least one processor is further configured to send a capture command to the image sensor, wherein the first image and the second image are received in response to the capture command, and wherein the first image and the second image are received based on a single read of the image sensor.

12. A method for imaging, the method comprising: obtaining a first image captured using a first portion of an image sensor; obtaining a second image captured using a second portion of the image sensor, wherein the second portion is different from the first portion; determining an image capture setting based on the second image; and applying the image capture setting to the first image.

13. The method according to claim 12, wherein the second image is captured at a lower pixel density.

14. The method according to claim 12, wherein the image capture setting includes one of a white balance setting or an exposure setting.

15. The method according to claim 12, wherein the second image has a wider field of view than the first image.

16. The method according to claim 12, wherein the second image includes the full field of view of the image sensor.

17. The method according to claim 12, wherein the second part is adjacent to the first part.

18. The method according to claim 12, wherein the image capture setting includes a white balance setting, and the method further comprises: determining that the white balance setting cannot be appropriately determined for the first image; and based on the determination that the white balance setting cannot be appropriately determined for the first image, determining the white balance setting for the first image based on the second image.

19. The method according to claim 18, wherein determining that the white balance setting cannot be appropriately determined for the first image includes: attempting to determine the white balance setting based on the first image; and determining that a reference region cannot be found in the first image.

20. The method according to claim 12, wherein the second image is subsampled from the second part by skipping at least one of the rows or columns of the photodiodes of the image sensor.

21. The method according to claim 12, wherein the first image and the second image are received separately.

22. The method according to claim 12, the method further comprises sending a capture command to the image sensor, wherein the first image and the second image are received in response to the capture command, and wherein the first image and the second image are received based on a single read of the image sensor.

23. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain a first image captured using a first part of an image sensor; obtain a second image captured using a second part of the image sensor, wherein the second part is different from the first part; determine an image capture setting based on the second image; and apply the image capture setting to the first image.

24. The non-transitory computer-readable medium according to claim 23, wherein the second image is captured at a lower pixel density.

25. The non-transitory computer-readable medium according to claim 23, wherein the image capture setting includes one of a white balance setting or an exposure setting.

26. The non-transitory computer-readable medium according to claim 23, wherein the second image has a wider field of view than the first image.

27. The non-transitory computer-readable medium according to claim 23, wherein the second image includes the full field of view of the image sensor.

28. The non-transitory computer-readable medium according to claim 23, wherein the second part is adjacent to the first part.

29. The non-transitory computer-readable medium according to claim 23, wherein the image capture settings include white balance settings, and wherein the instructions further cause the at least one processor to: Determine that the white balance setting cannot be appropriately determined for the first image; and Based on the determination that the white balance setting cannot be appropriately determined for the first image, determine the white balance setting for the first image based on the second image.

30. The non-transitory computer-readable medium according to claim 29, wherein, To determine that the white balance setting cannot be appropriately determined for the first image, the instructions further cause the at least one processor to: Attempt to determine the white balance setting based on the first image; and Determine that a reference region cannot be found in the first image.

Images